Nutrition measurement and diet recommendation platform and method based on multi-modal big language model

Abstract

The invention provides a nutrition measurement and diet recommendation platform and method based on a multi-modal large language model, and the platform comprises an input information obtaining module which is used for obtaining word information, statement information and / or picture information inputted by a user in a plurality of target input regions, and is used for analyzing the word information, statement information and / or picture information, and sending the analyzed word information, statement information and / or picture information to a user terminal; determining at least one target food type and food quantity; the nutrition analysis module is used for determining the nutrient element quantity and the dynamic evaluation result of the user according to the at least one target food type and the food quantity, and is used for determining the health risk index of the user according to the nutrient element quantity and the dynamic evaluation result of the user; the health propaganda and education module is used for providing nutrition and health knowledge content; and the diet recommendation module is used for generating highly personalized diet suggestions according to the nutrient element quantity and the dynamic evaluation result of the user, the personal physiological parameters and the health targets. And intelligence, individuation and convenience of nutrition and health management are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the cross field of artificial intelligence, nutrition, computer vision and natural language processing, in particular, to a nutrition measurement and meal recommendation based on a multi-modal large language model and a method. BACKGROUND

[0002] With the general improvement of global health awareness and the increasing demand for personalized health management, the inherent limitations of traditional nutrition analysis and meal recommendation methods have become increasingly prominent. Existing solutions often rely on tedious manual recording, static database queries and simple rule-based calculation models, which make it difficult to meet the expectations of modern users in terms of convenience, accuracy, real-time and personalization. For example, users have difficulty in efficiently recording their daily diet, and lack the ability to identify food pictures; nutrition assessment lacks depth and dynamics, and cannot fully consider individual differences and the balance of multiple nutrients; meal recommendation strategies are single, and do not fully combine user health goals, dietary preferences, and even multiple factors such as traditional Chinese medicine constitution; in addition, the lack of immediate and professional nutrition consultation channels makes it difficult for users to obtain timely and effective guidance when they have questions.

[0003] In recent years, the breakthroughs of artificial intelligence technology, especially deep learning in the fields of computer vision and natural language processing, have brought unprecedented opportunities for intelligent nutrition and health management. However, some existing intelligent nutrition applications still have the following shortcomings: first, the multi-modal input processing capability is limited, making it difficult to achieve seamless integration and deep understanding of image and text information; second, the nutrition knowledge base and data model are relatively independent, and cannot effectively utilize massive professional knowledge for retrieval enhancement generation, resulting in limited intelligent level of question answering and recommendation; third, there is a lack of deep integration of traditional Chinese medicine wisdom of "medical food homology", making it difficult to provide personalized guidance in line with the concept of Oriental health preservation; fourth, there are still challenges in data security and privacy protection, and the hardware performance requirements are high, limiting its popularity on low-end devices. Therefore, there is an urgent need to develop a more intelligent, flexible, personalized, and data security and inclusive nutrition quantitative evaluation and meal recommendation platform. SUMMARY

[0004] In view of the above, the purpose of the present application is to provide a nutrition measurement and meal recommendation platform and method based on a multi-modal large language model, providing an intelligent, personalized, authoritative and easy-to-use nutrition analysis and meal recommendation tool to help users better understand their nutritional status, develop healthy eating habits, achieve their health goals, and promote the advancement of national health management.

[0005] In a first aspect, the embodiments of the present application provide a nutrition measurement and meal recommendation platform based on a multi-modal large language model, characterized in that the platform comprises:

[0006] an input information acquisition module configured to acquire word information, sentence information, and / or picture information input by a user in a plurality of target input areas, and to parse the word information, the sentence information, and / or the picture information to determine at least one target food type and at least one food quantity;

[0007] a nutrition analysis module configured to determine, based on a multi-modal large language model, a retrieval-augmented generation (RAG) technique, and an authoritative nutrition database, a quantity of nutritional elements and a dynamic assessment result of the user according to the at least one target food type and the at least one food quantity, and to determine a health risk index of the user according to the quantity of nutritional elements and the dynamic assessment result of the user;

[0008] a health education module configured to provide nutritional health knowledge content;

[0009] a meal recommendation module configured to generate highly personalized meal suggestions according to the quantity of nutritional elements and the dynamic assessment result of the user, personal physiological parameters, and health goals, and to support comparison and selection of a plurality of meal plans;

[0010] an AI question and answer module configured to provide real-time intelligent answers, meal planning, and chronic disease management suggestions based on a multi-modal large language model, a RAG technique, and a professional knowledge base in response to nutritional-related questions of the user.

[0011] Optionally, the platform further comprises:

[0012] a user input module configured to determine, based on natural language processing technology, a diet description, food names, quantities, and unit information of the user, and health goals, preferences, and restrictions of the user in response to text input of the user in at least one target input area.

[0013] Optionally, the user input module is further configured to display a lifestyle questionnaire determined by an expert, and to determine a lifestyle of the user in response to input of the user in the lifestyle questionnaire.

[0014] Optionally, the user input module is further configured to identify food types and portion sizes in a picture through image recognition technology in response to picture input of the user in at least one target input area, and to determine diet data of the user by combining a multi-modal large language model to deeply understand and extract information from the picture content.

[0015] Optionally, the platform further comprises:

[0016] a nutrition analysis module, the nutrition analysis module comprising a nutrition database, an AI analysis unit, and a nutrition assessment unit;

[0017] The nutrition database includes an ingredient database and a recipe database, which contain food nutrient data such as calories, protein, fat, carbohydrates, vitamins, and minerals, as well as a dynamic nutrition knowledge graph.

[0018] The AI ​​analysis unit is used to perform structured processing on the dietary information and lifestyle questionnaire data input by the user using a multimodal large language model and RAG technology, extracting food names, quantities and units, and to determine the user's input energy and intake of various nutrients based on the food nutrient composition data in the nutrition database.

[0019] The nutrition assessment unit is used to conduct a multi-dimensional assessment of the user's nutritional intake based on the user's individual physiological parameters, health goals, dietary reference intake, and lifestyle questionnaire data determined by experts. It is also used to determine the user's nutrient input based on the results of the multi-dimensional assessment of the user's nutritional intake and generate a health risk index based on dietary data.

[0020] Optionally, the AI ​​analysis unit includes:

[0021] An image recognition subunit is used to identify the types of food in an image;

[0022] The portion estimation subunit is used to estimate the portion size of food in the image;

[0023] The multimodal fusion subunit combines image recognition results with the user's text input to generate the user's dietary information.

[0024] Optionally, the dietary recommendation module is also used for comparing and selecting multiple dietary plans, and for generating personalized weekly menus based on user input.

[0025] Optionally, it also includes a user interface module for displaying nutritional analysis results, health education content, dietary recommendations, and AI Q&A content, and provides a user input interface and operation controls, as well as an interactive interface for an expert-defined lifestyle questionnaire.

[0026] Optionally, the user interface module is also used to visually display the user's nutritional intake, specifically by displaying the intake ratio of energy, macronutrients and micronutrients through charts, as well as a real-time comparison with the recommended intake.

[0027] Secondly, this application also provides a method for nutritional measurement and dietary recommendation based on a multimodal large language model, the method comprising:

[0028] The input information acquisition module controls the acquisition of word information, sentence information and / or image information entered by the user in multiple target input areas, and controls the acquisition of the input information to parse the word information, sentence information and / or image information to determine at least one target food type and at least one food quantity;

[0029] The control nutrition analysis module, based on multimodal large language model, retrieval enhancement generation technology and authoritative nutrition database, determines the user's nutritional element content and dynamic assessment results based on the absence of one target food type and at least one food quantity, and determines the user's health risk index based on the user's nutritional element content and dynamic assessment results.

[0030] The health education module provides nutritional and health knowledge content.

[0031] The dietary recommendation module generates highly personalized dietary suggestions based on the user's nutrient levels, dynamic assessment results, personal physiological parameters, and health goals, and supports the comparison and selection of multiple dietary plans;

[0032] The AI-powered question-and-answer module responds to users' nutrition-related questions, providing real-time intelligent answers, dietary planning, and chronic disease management advice based on a multimodal large language model, RAG technology, and a professional knowledge base.

[0033] This application provides a method for nutritional measurement and dietary recommendations based on a multimodal large language model. It offers an intelligent, personalized, authoritative, and easy-to-use nutritional analysis and dietary advice tool to help users better understand their nutritional status, develop healthy eating habits, achieve their health goals, and promote the shift of the focus of national health management forward.

[0034] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A schematic diagram of the structure of a nutrition measurement and dietary recommendation platform based on a multimodal large language model provided in an embodiment of this application;

[0037] Figure 2 This is a schematic diagram of the nutrient analysis module provided in an embodiment of this application;

[0038] Figure 3 This is a schematic diagram of the health education module provided in an embodiment of this application;

[0039] Figure 4 A schematic diagram of the AI ​​question-answering module provided in an embodiment of this application;

[0040] Figure 5 This is a flowchart illustrating the nutritional measurement and dietary recommendation method based on a multimodal large language model provided in an embodiment of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0042] First, the applicable scenarios for this application will be introduced. This application can be applied to the interdisciplinary fields of artificial intelligence, nutrition, computer vision, and natural language processing.

[0043] With the increasing global awareness of health and the growing demand for personalized health management, the inherent limitations of traditional nutritional analysis and dietary recommendation methods are becoming increasingly apparent. Existing solutions often rely on cumbersome manual recording, static database queries, and simple rule-based computational models, making it difficult to meet the expectations of modern users in terms of convenience, accuracy, real-time performance, and personalization. For example, users struggle to efficiently record their daily diets and lack the ability to recognize food images; nutritional assessments lack depth and dynamism, failing to comprehensively consider individual differences and the balance of various nutrients; dietary recommendation strategies are simplistic, failing to fully integrate diverse factors such as user health goals, dietary preferences, and even traditional Chinese medicine constitution; furthermore, the lack of immediate and professional nutritional consultation channels means users cannot obtain timely and effective guidance when they encounter questions.

[0044] In recent years, breakthroughs in artificial intelligence technology, particularly deep learning in computer vision and natural language processing, have brought unprecedented opportunities for intelligent nutrition and health management. However, some existing intelligent nutrition applications still have the following shortcomings: First, their multimodal input processing capabilities are limited, making it difficult to achieve seamless integration and deep understanding of image and text information; second, the nutritional knowledge base and data model are relatively independent, failing to effectively utilize massive amounts of professional knowledge for retrieval and enhancement generation, thus limiting the level of intelligence in question answering and recommendation; third, they lack deep integration with the wisdom of "medicine and food sharing the same origin" in traditional Chinese medicine, making it difficult to provide personalized guidance that aligns with Eastern health concepts; fourth, they still face challenges in data security and privacy protection, and their high hardware performance requirements limit their widespread adoption on low-end devices. Therefore, there is an urgent need to develop a more intelligent, flexible, and personalized nutritional quantitative assessment and dietary recommendation platform that balances data security and accessibility.

[0045] Based on this, the embodiments of this application provide a nutrition measurement and dietary recommendation platform and method based on a multimodal large language model, providing an intelligent, personalized, authoritative and easy-to-use nutrition analysis and dietary advice tool to help users better understand their nutritional status, develop healthy eating habits, achieve their health goals, and promote the forward shift of the health management focus for the entire population.

[0046] Firstly, please refer to Figure 1 , Figure 1 This is a schematic diagram of a nutrition measurement and dietary recommendation platform based on a multimodal large language model, provided as an embodiment of this application. Figure 1 As shown in the figure, the nutrition measurement and dietary recommendation platform based on a multimodal large language model provided in this application embodiment includes: an input information acquisition module 100, a nutrition analysis module 200, a health education module 300, a dietary recommendation module 400, and an AI question answering module 500.

[0047] The input information acquisition module is used to acquire word information, sentence information and / or image information entered by the user in multiple target input areas, and to parse the word information, sentence information and / or image information to determine at least one target food type and at least one food quantity;

[0048] The nutrition analysis module is used to determine the user's nutrient element content and dynamic assessment results based on multimodal large language model, retrieval enhanced generation (RAG) technology and authoritative nutrition database, according to at least one target food type and at least one food quantity; and to determine the user's health risk index based on the user's nutrient element content and dynamic assessment results.

[0049] The health education module is used to provide nutritional and health knowledge content;

[0050] The dietary recommendation module generates highly personalized dietary suggestions based on the user's nutrient levels, dynamic assessment results, personal physiological parameters, and health goals, and supports the comparison and selection of multiple dietary plans.

[0051] The AI ​​question-answering module is used to respond to users' nutrition-related questions, providing real-time intelligent answers, dietary planning, and chronic disease management suggestions based on multimodal large language models, RAG technology, and a professional knowledge base.

[0052] Here, the nutrition measurement and dietary recommendation platform provides nutritional dietary early warning services. By analyzing users' meal data and combining it with a healthy dietary model, the platform conducts in-depth analysis of energy intake, protein, fat, carbohydrates (the three macroelements), and micronutrient intake, assesses food diversity, determines whether health needs are met, and provides intelligent warnings for any excesses. It also offers online nutrition and health education services, compiling dietary lecture videos from industry nutrition experts to provide in-depth popularization of healthy dietary knowledge; simultaneously, it synchronizes and updates the latest recommendations of the "Chinese Dietary Guidelines" in real time, ensuring users receive authoritative and scientific nutritional guidance. Furthermore, it offers intelligent suggestion and Q&A services. The platform's core innovation lies in the deep integration of multimodal large language models, retrieval-enhanced generation (RAG) technology, and a professional nutrition knowledge base and self-built database. This enables intelligent parsing of user multimodal input (including text and images), accurate nutritional intake analysis, and highly personalized dietary recommendations. Through RAG technology, the platform effectively utilizes knowledge base and database resources to provide users with intelligent nutrition Q&A and personalized dietary guidance, offering high practicality and flexibility.

[0053] The proposed nutrition measurement and dietary recommendation platform based on a multimodal large language model provides an efficient and stable user experience. This application utilizes federated learning technology to ensure data security and edge computing to enable smooth operation on low-end mobile phones, ensuring efficient operation of AI functions across various devices. As a cross-platform application built on the Uni-App framework, its front-end design emphasizes responsiveness and user experience, ensuring smooth interface switching and rapid operation response, maintaining good interactivity even under poor network conditions. Through optimized API interfaces and WebSocket communication, it achieves rapid data exchange with backend services and AI models, ensuring real-time nutrition assessment and dietary recommendations. Although core data is stored on the backend, the front-end application possesses powerful data processing and visualization capabilities, enabling rapid localization and display of user-input dietary data, providing immediate feedback.

[0054] As an example, the minimum hardware requirements for the nutrition measurement and dietary recommendation platform based on the multimodal large language model in this application are: CPU: 2.3HZ or higher, memory: 1G or higher, and storage space: 32GB.

[0055] Specifically, the nutrition analysis module is the core of achieving precise quantitative nutrition assessment. It provides users with comprehensive and scientific nutrition intake analysis by integrating authoritative nutrition databases, advanced AI analysis units, and dynamic evaluation systems.

[0056] The nutrition database includes: an ingredient database, which is an authoritative ingredient database built on the 6th edition of the "Chinese Food Composition Table Standard Edition". This database contains detailed data on at least 41 key nutrients, including energy, protein, fat, carbohydrates, vitamins (such as vitamins A, C, D, E, and B complex), and minerals (such as calcium, iron, zinc, and selenium), for thousands of common ingredients. The authority of the data is the cornerstone of ensuring the accuracy of nutritional analysis; a dish database, which, in addition to the ingredient database, is self-built and continuously expanded. This database contains nutritional data for a large number of common home-cooked dishes, local specialties, and pre-cooked dishes. The dish data not only includes the overall nutritional composition but also traces the nutritional contributions of the constituent ingredients. This database has self-learning and expansion capabilities and can be updated and optimized based on user-added dish data; and a dynamic nutrition knowledge graph, which, based on the above databases, further integrates the "Chinese Dietary Guidelines", over 200,000 clinical nutrition cases, and the latest nutritional research results to construct a dynamically updated nutrition knowledge graph. This graph provides AI analysis units with in-depth nutritional background knowledge through semantic association and knowledge reasoning, thereby improving the level of intelligence in the analysis.

[0057] Specifically, the AI ​​analysis unit includes a structured information extraction function, capable of receiving multimodal fusion-based structured dietary information (including food names, quantities, and units) provided by the user input module. The AI ​​analysis unit utilizes large language models and RAG technology to further refine and standardize this information, ensuring it matches the data format in the nutrition database.

[0058] The implementation principles and applications of RAG include: vectorized index construction, which divides all food and dish entries in the nutrition database, as well as knowledge points in the dynamic nutrition knowledge graph, into fine-grained chunks, and uses a pre-trained large language model to transform each chunk into a high-dimensional vector (embedding). These vectors are efficiently stored in a dedicated vector database, forming a rapidly searchable knowledge index; intelligent retrieval, where the AI ​​analysis unit transforms user input of dietary information into query vectors. In the vector database, a similarity matching algorithm (such as cosine similarity) is used to quickly retrieve the K most relevant nutrition database chunks and knowledge graph chunks related to the user's dietary content; and enhanced generation and computation, where the retrieved K relevant chunks, along with the user's original input information, are used as input to the large language model. At this stage, LLM not only performs semantic understanding, but more importantly, it can use this retrieved authoritative data to perform accurate nutritional component calculations. For example, it transforms "one apple" into "apple, 1, 150 grams," and calculates the energy, vitamins, etc., contained in 150 grams of apple based on the nutritional data of the apple in the database. This RAG mechanism ensures the accuracy and traceability of the calculation results; for nutrient intake calculation, based on structured food information and RAG-enhanced nutrient data, the AI ​​analysis unit accurately calculates the total amount of energy (calories / joules), protein, fat, carbohydrates, dietary fiber, vitamins, minerals and other at least 41 nutrients consumed by the user during a specific meal period (such as breakfast, lunch and dinner) or throughout the day.

[0059] The nutrition assessment unit employs a dynamic assessment system, utilizing a patented technology. This system provides real-time, dynamic evaluation of a user's nutritional intake based on individual physiological parameters (age, gender, height, weight, activity level), health goals (such as weight loss, muscle gain, and chronic disease management), and the Dietary Reference Intakes (DRIs) recommended in the Chinese Dietary Guidelines. It also utilizes multi-dimensional assessment indicators, including not only whether the intake of individual nutrients meets standards but also the ratio of macronutrient energy, the synergistic effect of micronutrients, food diversity, and the rationality of dietary patterns. Furthermore, it employs visual reports and alerts, presenting assessment results to users in an intuitive and visual manner. For example, it uses charts (such as radar charts and bar charts) to display the intake ratios of energy, macronutrients, and micronutrients, as well as real-time comparisons with recommended intakes, and uses color coding (e.g., green indicates compliance, yellow indicates deviation, and red indicates severe excess / deficiency) for alerts. Simultaneously, it generates a health risk index based on 30 days of dietary data to help users comprehensively understand their nutritional status.

[0060] Specifically, the health education module (the core of the "Take a Look" function) aims to provide users with authoritative, systematic, and easy-to-understand nutritional and health knowledge, empowering users to manage their health independently. It integrates authoritative knowledge content, bringing together dietary lecture videos from renowned nutrition experts, doctors, and health managers in the industry. These videos cover basic dietary nutrition knowledge, dietary guidance for specific groups (such as pregnant women, children, the elderly, people with hypertension, hyperlipidemia, and hyperglycemia, and fitness enthusiasts), dietary management of common chronic diseases, food pairing techniques, and healthy cooking methods, providing in-depth popularization of healthy dietary knowledge in a vivid and intuitive format. It also synchronizes with the "Dietary Guidelines for Chinese Residents" in real time, updating the latest recommendations from the National Health Commission. Users can access officially recommended dietary patterns, food selection principles, and physical activity suggestions at any time, ensuring access to the most authoritative and scientific nutritional guidance. The platform also constructs a comprehensive health knowledge base, including diverse content such as nutrition encyclopedia, food efficacy, the relationship between disease and diet, and healthy lifestyles, which users can search and learn according to their needs. It also includes personalized knowledge delivery and a recipe engine, including health education based on user profiles. Based on user health profiles (including age, gender, physiological parameters, health goals, TCM constitution identification results, and lifestyle questionnaire data), the platform can intelligently push targeted health education content. For example, it can recommend pregnancy nutrition recipes and precautions for pregnant women and provide dietary advice on controlling salt and oil intake for people with high blood pressure, high cholesterol, and high blood sugar. The recipe engine and nutritional gap recommendation system: the platform has a powerful built-in recipe engine that not only provides a wide selection of dishes but also intelligently recommends dishes that can fill these gaps based on the user's nutritional analysis results (identified nutritional deficiencies or excesses). For example, if a user is deficient in Vitamin C, the system will recommend dishes rich in Vitamin C and their recipes, achieving a seamless integration of knowledge and practice.

[0061] Specifically, the dietary recommendation module aims to provide users with highly personalized, scientific dietary advice tailored to their health needs. Its core lies in integrating Traditional Chinese Medicine (TCM) constitution identification, lifestyle data, and intelligent recommendation algorithms. This includes a TCM constitution AI identification system that deeply integrates classic TCM health preservation theories such as the *Huangdi Neijing* (Yellow Emperor's Inner Classic) with modern medical physiological indicators to construct a TCM constitution identification model. Furthermore, through machine learning algorithms, it comprehensively analyzes user-input health questionnaires and physiological data (such as body temperature, pulse, tongue coating, and complexion, accessed through user self-reporting or smart devices) to accurately identify nine TCM constitutions (such as balanced constitution, Qi deficiency constitution, Yang deficiency constitution, Yin deficiency constitution, phlegm-dampness constitution, damp-heat constitution, blood stasis constitution, Qi stagnation constitution, and special constitution). Based on the user's TCM constitution identification results, it can also generate personalized dietary dos and don'ts tags for the user. For example, it recommends "warming and nourishing foods for those with Yang deficiency" and "avoiding spicy and greasy foods for those with damp-heat constitution." It also features refined dish labels; each dish in the database is pre-labeled with rich nutritional tags (such as "low sugar," "low fat," "high protein," "rich in dietary fiber," "rich in vitamin C," etc.) and TCM health tags (such as "warming," "cooling," "strengthening the spleen," "removing dampness," etc.). Furthermore, it executes a multi-dimensional matching strategy. The dietary recommendation module comprehensively considers the user's nutritional analysis results (nutritional deficiencies), health goals, dietary preferences, allergy information, TCM constitution identification results, and lifestyle questionnaire data verified by experts. Through intelligent matching algorithms, it selects dishes that simultaneously meet multiple conditions and effectively fill nutritional gaps. For example, for a user with "Yin deficiency constitution" who is trying to lose weight and has insufficient protein intake, the system will prioritize recommending dishes that "nourish Yin and moisten dryness, are high in protein and low in fat." It also provides personalized dietary advice and plan selection, specifically including a recipe engine: The platform has a powerful built-in recipe engine that can intelligently generate personalized dietary suggestions based on the above matching results. These suggestions include not only specific dish recommendations but also detailed recipes, cooking methods, and ingredient pairing suggestions; multiple options for comparison and customization: The platform supports the generation of various dietary plans, allowing users to intuitively compare the nutritional composition, calorie distribution, ingredient lists, and cooking difficulty of different plans. Users can choose and adjust according to their own needs, such as selecting a "weekly weight loss diet" or a "diabetic patient-specific diet," and can replace ingredients in the diet according to personal preferences. The system updates nutritional analysis in real time; intelligent weekly diet generation: Based on the user's long-term dietary data, health goals, and physical characteristics, the AI ​​diet planner can automatically generate personalized weekly diets and provide shopping lists, greatly facilitating the user's dietary management.

[0062] Specifically, the AI ​​Q&A module ("Ask a Question") aims to provide users with 24-hour online, professional, and intelligent nutrition consultation services, acting as users' exclusive nutritionist and health manager.

[0063] The AI ​​question-answering module has built a professional knowledge base covering a wide range of professional nutrition knowledge. The content comes from authoritative domestic and international nutrition monographs, international / national dietary guidelines (such as the *Chinese Dietary Guidelines*), the latest clinical nutrition research reports, and over 200,000 clinical nutrition cases. It has also been graphed, with the knowledge base content structured and constructed into a highly interconnected knowledge network. This facilitates efficient and accurate retrieval using RAG technology. Furthermore, it can identify user needs through natural language understanding and intent recognition. For example, when users ask nutrition-related questions in natural language (such as "What fruits can diabetic patients eat?" or "How to scientifically build muscle?"), the AI ​​question-answering module uses a large language model for NLU processing to accurately identify the user's question intent and keywords.

[0064] Specifically, the deep application of RAG technology includes: question vectorization and retrieval, transforming user questions into query vectors and efficiently retrieving them from the vector database of the professional knowledge base to identify the K most relevant knowledge chunks; enhanced answer generation, inputting the retrieved relevant knowledge chunks along with the user's original question into the large language model. Based on this, LLM performs information integration, reasoning, and language generation, outputting accurate, professional, and easily understandable natural language answers. This mechanism avoids the "illusion" problem that may occur in traditional question-answering systems, ensuring the authority and reliability of the answers; multi-turn dialogue and personalized suggestions, the AI ​​question-answering module supports multi-turn dialogue, understands the context of the dialogue, and provides in-depth answers based on the user's follow-up questions. Combining the user's personal health data, nutritional analysis results, and lifestyle questionnaire data verified by experts, the AI ​​nutritionist can also provide highly personalized nutritional guidance and suggestions; chronic disease management assistant, for patients with common chronic diseases such as diabetes, hypertension, and hyperlipidemia, the AI ​​question-answering module provides a dedicated dietary warning system and management suggestions. For example, when a user inquires about a certain food, the system will provide professional guidance on whether it is suitable and the recommended intake based on their chronic disease type.

[0065] Optionally, the platform further includes: a user input module, used to determine, in response to text input by the user in at least one target input area, the user's dietary description, food name, quantity and unit information, as well as the user's health goals, preferences and restrictions, based on natural language processing technology.

[0066] By employing Natural Language Understanding (NLU) technology, users can describe their meals using everyday, conversational natural language, such as "I ate two eggs, a glass of milk, and a slice of bread for breakfast today." The platform utilizes an advanced Large Language Model (LLM) for NLU processing to accurately identify and extract key information from the text, including food names, quantities, units (such as "pieces," "cups," and "slices"), and meal times.

[0067] The user input module possesses contextual understanding capabilities, remembering and associating dietary information and health goals mentioned by the user in multiple rounds of dialogue, thereby providing a more coherent and user-friendly input experience. For example, a user can first describe breakfast and then add lunch, which the system can automatically integrate. Users can also input their health goals (such as weight loss, muscle gain, and blood sugar control) as well as dietary preferences and restrictions (such as vegetarianism, food allergies, and dietary restrictions) via text. This information will serve as an important basis for subsequent nutritional analysis and dietary recommendations.

[0068] Specifically, the user input module is also used to display a lifestyle questionnaire determined by experts, and to determine the user's lifestyle in response to the user's input in the lifestyle questionnaire.

[0069] Specifically, the user input module is also used to respond to the user's image input in at least one target input area, identify the types and quantities of food in the image through image recognition technology, and perform deep understanding and information extraction of the image content by combining a multimodal large language model to determine the user's dietary data.

[0070] For example, please refer to Figure 2 , Figure 2 This is a schematic diagram of the nutrition analysis module provided in an embodiment of this application. The nutrition analysis module corresponds to the "Take a Test" function, allowing users to submit dietary information and view the nutrition analysis results by selecting image recognition or text input.

[0071] The "Take a Test" section allows for intelligent dietary recording and personalized nutritional assessment. By inputting text or taking a photo of food, it automatically identifies 93 types of ingredients, generates an accurate nutritional report, and provides a comprehensive interpretation of 41 nutrients.

[0072] When users first enter the mini-program, they are directly redirected to the "Take a Test" page without logging in. They can identify dishes by text or images and receive nutritional information and a simplified diet report. Clicking "Save" will take them to the login page.

[0073] After logging in, users can save dishes to their food history. Then, a quiz will pop up asking if they need to enter personal information for more accurate service. The information to be entered includes: gender, date of birth, height, weight, daily activity level, and usage goals (maintaining health, weight loss, fitness, prenatal care, saving money, etc.).

[0074] For those who are not satisfied with simply entering the dish name, a more professional input mode is provided. After clicking the professional mode button, a professional mode button will be added after the input box, which will take you to the search page. The search interface allows you to search for and record the food you eat on a specific day. You can search for a certain ingredient and add it to your shopping cart. Clicking on a single food item allows you to adjust the weight in grams. After clicking submit, the total value will be calculated and recorded.

[0075] Regarding the number of food servings, the weight in grams of one serving is determined by the recipe weight, but it is displayed as the nutritional value per 100g on the food details page. The actual amount added is 1 serving by default. You can slide left and right to different units, such as changing "small bowl" to "grams".

[0076] The nutrition analysis report provides your daily nutrient score, including intake values ​​and recommended values ​​for six components: energy, carbohydrates, fat, protein, dietary fiber, and sodium. It also provides health advice based on your actual intake and personal information.

[0077] For example, please refer to Figure 3 , Figure 3 This is a schematic diagram of the health education module provided in an embodiment of this application. The health education module corresponds to the "Take a Look" function, which provides nutritious recipe recommendations and health education. Based on the user's nutritional deficiencies, it intelligently recommends over 3,000 international recipes, covering the special nutritional needs of 90% of the population. It also provides health education content for nine special groups, including pregnant women, postpartum women, and people with hypertension, hyperlipidemia, and hyperglycemia.

[0078] It provides health education videos for 13 groups, including obese people, college teachers, college students, children and adolescents, the elderly, people with high blood pressure, high cholesterol, high blood sugar, and high cholesterol, people who consume additives, pregnant women, fitness enthusiasts, working professionals, people who read "Millennial Dietary Records: Health Wisdom in Ancient Scrolls", babies with allergies, and people suffering from nutritional hunger. It also provides recipes with labels such as low sugar, containing added sugar, moderate sodium content, low fat, low cholesterol, low sodium, high sodium, or for people with diabetes, high blood pressure, or pregnant women.

[0079] For example, please refer to Figure 4 , Figure 4 This is a schematic diagram of the AI ​​question-and-answer module provided in an embodiment of this application. The AI ​​question-and-answer module corresponds to the "Ask a Question" function. In the "Ask a Question" section, users can enter nutrition-related questions in the chat interface and obtain real-time answers from an AI nutritionist. The 24 / 7 health management assistant is online, simulating the logic of nutrition experts from top-tier hospitals, providing nutritional guidance and health advice.

[0080] The core innovation of this module lies in the deep integration of large language models, Retrieval Enhanced Generation (RAG) technology, and a professional nutrition knowledge base and self-built database. This enables intelligent parsing of user multimodal input (including text and images), accurate nutritional intake analysis, and highly personalized dietary recommendations. The platform has built a food ingredient database based on the Chinese Food Composition Table (Standard Version 6) and has also built a continuously expanding self-built recipe database, ensuring the authority and comprehensiveness of the data. Through RAG technology, the platform can effectively utilize knowledge base and database resources to provide users with intelligent nutrition Q&A and personalized dietary guidance, demonstrating high practicality and flexibility.

[0081] Optionally, the platform further includes:

[0082] The nutrition analysis module includes a nutrition database, an AI analysis unit, and a nutrition assessment unit.

[0083] The nutrition database includes an ingredient database and a recipe database, which contain food nutrient data such as calories, protein, fat, carbohydrates, vitamins, and minerals, as well as a dynamic nutrition knowledge graph.

[0084] The AI ​​analysis unit is used to perform structured processing on the dietary information and lifestyle questionnaire data input by the user using a multimodal large language model and RAG technology, extracting food names, quantities and units, and to determine the user's input energy and intake of various nutrients based on the food nutrient composition data in the nutrition database.

[0085] The nutrition assessment unit is used to conduct a multi-dimensional assessment of the user's nutritional intake based on the user's individual physiological parameters, health goals, dietary reference intake, and lifestyle questionnaire data determined by experts. It is also used to determine the user's nutrient input based on the results of the multi-dimensional assessment of the user's nutritional intake and generate a health risk index based on dietary data.

[0086] Specifically, the AI ​​analysis unit includes:

[0087] An image recognition subunit is used to identify the types of food in an image;

[0088] The portion estimation subunit is used to estimate the portion size of food in the image;

[0089] The multimodal fusion subunit combines image recognition results with the user's text input to generate the user's dietary information.

[0090] The dietary recommendation module is also used for comparing and selecting multiple dietary plans, as well as for generating personalized weekly menus based on user input.

[0091] Among them, food image recognition allows users to directly upload food images. The platform utilizes a multimodal large language model integrated with advanced computer vision technology to perform high-precision analysis on the images and automatically identify the types of food in them. This invention supports the accurate identification of at least 92 common food categories, covering most scenarios in daily diets.

[0092] Among these features, Portion Estimation: While identifying the type of food, this submodule can also intelligently estimate the approximate portion size of food in an image by combining image features and deep learning algorithms. This greatly simplifies the tedious process of manually entering portions for users and improves recording efficiency.

[0093] Multimodal information fusion is a key component. Image recognition results (food type, estimated portion size) are deeply integrated with supplementary text descriptions provided by the user (e.g., "This is braised pork I made, about 200 grams"). A multimodal large language model plays a crucial role here, generating structured, high-precision dietary information through cross-validation and semantic understanding, ensuring the integrity and accuracy of the data. For example, if the image recognizes "rice" and the user adds "one bowl," the system can intelligently infer the standard portion size.

[0094] Optionally, it also includes a user interface module for displaying nutritional analysis results, health education content, dietary recommendations, and AI Q&A content, and provides a user input interface and operation controls, as well as an interactive interface for an expert-defined lifestyle questionnaire.

[0095] Among them, the expert-designed questionnaire: The platform includes a lifestyle questionnaire carefully designed by a team of nutrition and medical experts. This questionnaire covers multiple dimensions such as the user's daily routine, exercise habits, sleep quality, emotional state, past medical history, and family history, aiming to comprehensively understand the user's health background and lifestyle habits.

[0096] The questionnaire employs intelligent interaction and data collection: it dynamically adjusts subsequent questions based on user responses to ensure targeted and efficient data collection. After completion, the system structures the questionnaire data and uses it as a crucial personalized parameter for nutritional analysis and dietary recommendations. This questionnaire data, along with dietary data and physiological parameters, constructs a complete health profile for the user, providing richer data support for subsequent accurate assessments and personalized services.

[0097] The user interface module is also used to visually display the user's nutritional intake, specifically by showing the intake ratio of energy, macronutrients and micronutrients through charts, as well as a real-time comparison with the recommended intake.

[0098] The platform's data accuracy is supported by the following core elements: National-level nutrition database support: Integrating the "Chinese Dietary Guidelines" and over 200,000 clinical nutrition cases to construct a dynamic nutrition knowledge graph, ensuring the authority and comprehensiveness of the nutrition data; Intelligent nutrition assessment model: Employing a nationally patented dynamic assessment system, it can quantitatively analyze the intake levels of 41 nutrients, providing accurate nutrition assessments; Multimodal large-scale model and RAG technology: Combining advanced multimodal large-scale language models and Retrieval Enhanced Generation (RAG) technology, it intelligently parses user input and retrieves information from professional knowledge bases and databases, ensuring the accuracy and professionalism of question answering and recommendations.

[0099] This application proposes a nutrition measurement and dietary recommendation platform based on a multimodal large language model, which supports users to input dietary information and health data through various methods such as natural language text descriptions, food image uploads, and expert-assessed lifestyle questionnaires. Utilizing an advanced multimodal large language model, the platform can not only accurately identify food types and estimate portions, but also deeply understand the contextual semantics of images and text. Combined with questionnaire data, it achieves seamless information fusion and structured extraction, greatly improving the convenience of dietary records and the comprehensiveness of user profiles. By using a multimodal large language model and RAG technology to deeply process structured dietary information and lifestyle questionnaire data, it can calculate and dynamically assess the user's energy intake and at least 41 key nutrients (including macronutrients, vitamins, and minerals) in real time. More importantly, this invention adopts a machine learning-based dynamic assessment system, combined with the user's individual physiological parameters and health goals, to provide highly personalized, multi-dimensional integrated nutrition assessment reports and health risk warnings based on 30 days of dietary data. A two-layer nutrition database is constructed: one is an authoritative food ingredient database based on the "Chinese Food Composition Table (Standard Edition 6)"; the other is a self-built and continuously expanding dish database. Building upon this foundation, the platform integrates the "Dietary Guidelines for Chinese Residents," over 200,000 clinical nutrition cases, and the latest nutritional research findings to create a dynamically updated nutritional knowledge graph. Combined with Retrieval Enhanced Generation (RAG) technology, it ensures the authority, comprehensiveness, and real-time nature of nutritional analysis and Q&A. Furthermore, the platform offers online nutrition and health education services, compiling dietary lecture videos from industry nutrition experts and updating them in real-time with the latest recommendations from the "Dietary Guidelines for Chinese Residents," providing users with authoritative and scientific health knowledge. Based on accurate nutritional assessment results, individual TCM constitution identification, health goals, and lifestyle questionnaire data, the platform intelligently generates personalized dietary recommendations through food tag matching and nutritional gap filling algorithms, supporting the comparison and selection of multiple dietary plans, including customized weekly menus. This recommendation strategy not only considers nutritional balance but also incorporates traditional TCM health wisdom, making dietary recommendations more targeted and effective. A built-in AI Q&A module acts as a 24 / 7 intelligent nutritionist and AI dietary planner for users. Leveraging multimodal large language models and RAG technology, combined with a professional knowledge base, it can analyze users' nutrition-related questions in real time, providing accurate and professional answers and suggestions. It supports multi-turn dialogues and offers a dedicated dietary early warning system for chronic diseases such as diabetes and hypertension, achieving comprehensive nutrition and health management services. Federated learning technology ensures user data security and privacy, while edge computing optimizes model deployment, enabling smooth use on low-end mobile phones. It truly brings cutting-edge technology to the general public, reflecting the social value of science and technology benefiting people's livelihoods.

[0100] Based on the same inventive concept, this application also provides a method for implementing the aforementioned multimodal large language model-based nutrition measurement and dietary recommendation platform. The solution provided by this method is similar to the implementation scheme described in the aforementioned platform. Therefore, the specific limitations in the one or more method embodiments of the multimodal large language model-based nutrition measurement and dietary recommendation platform provided below can be found in the above-described limitations of the multimodal large language model-based nutrition measurement and dietary recommendation platform, and will not be repeated here.

[0101] Please refer to Figure 5 In one exemplary embodiment, a method for nutritional measurement and dietary recommendation based on a multimodal large language model is provided, including:

[0102] S101, the input information acquisition module acquires word information, sentence information and / or image information input by the user in multiple target input areas, and controls the input information acquisition module to parse the word information, sentence information and / or image information to determine at least one target food type and at least one food quantity;

[0103] S102, the control nutrition analysis module, based on multimodal large language model, retrieval enhancement generation technology and authoritative nutrition database, determines the user's nutritional element content and dynamic assessment results based on one less target food type and at least one food quantity, and determines the user's health risk index based on the user's nutritional element content and dynamic assessment results.

[0104] S103, the health education module provides nutritional and health knowledge content;

[0105] S104, the dietary recommendation module generates highly personalized dietary suggestions based on the user's nutrient element levels and dynamic assessment results, personal physiological parameters and health goals, and supports the comparison and selection of multiple dietary plans;

[0106] S105 controls the AI ​​question-and-answer module to respond to users' nutrition-related questions, providing real-time intelligent answers, dietary planning, and chronic disease management suggestions based on multimodal large language models, RAG technology, and a professional knowledge base.

[0107] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0108] The nutrition measurement and dietary recommendation platform and method based on a multimodal large language model provided in this application provides an intelligent, personalized, authoritative and easy-to-use nutrition analysis and dietary advice tool to help users better understand their nutritional status, develop healthy eating habits, achieve their health goals, and promote the forward shift of the national health management focus.

[0109] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0110] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A nutrition measurement and dietary recommendation platform based on a multimodal large language model, characterized in that, The platform includes: The input information acquisition module is used to acquire word information, sentence information and / or image information entered by the user in multiple target input areas, and to parse the word information, sentence information and / or image information to determine at least one target food type and at least one food quantity; The nutrition analysis module is used to determine the user's nutrient element content and dynamic assessment results based on multimodal large language model, retrieval enhanced generation (RAG) technology and authoritative nutrition database, according to at least one target food type and at least one food quantity; and to determine the user's health risk index based on the user's nutrient element content and dynamic assessment results. The health education module is used to provide nutritional and health knowledge content; The dietary recommendation module generates highly personalized dietary suggestions based on the user's nutrient levels, dynamic assessment results, personal physiological parameters, and health goals, and supports the comparison and selection of multiple dietary plans. The AI ​​question-answering module is used to respond to users' nutrition-related questions, providing real-time intelligent answers, dietary planning, and chronic disease management suggestions based on multimodal large language models, RAG technology, and a professional knowledge base.

2. The nutrition measurement and dietary recommendation platform according to claim 1, characterized in that, The platform also includes: The user input module is used to respond to text input by the user in at least one target input area and, based on natural language processing technology, determine the user's dietary description, food names, quantities and units, as well as the user's health goals, preferences and restrictions.

3. The nutrition measurement and dietary recommendation platform according to claim 2, characterized in that, The user input module is also used to display an expert-assessed lifestyle questionnaire and, in response to the user's input in the lifestyle questionnaire, to determine the user's lifestyle.

4. The nutrition measurement and dietary recommendation platform according to claim 2, characterized in that, The user input module is also used to respond to the user's image input in at least one target input area, identify the type and quantity of food in the image through image recognition technology, and perform deep understanding and information extraction of the image content by combining a multimodal large language model to determine the user's dietary data.

5. The nutrition measurement and dietary recommendation platform according to claim 1, characterized in that, The platform also includes: The nutrition analysis module includes a nutrition database, an AI analysis unit, and a nutrition assessment unit. The nutrition database includes an ingredient database and a recipe database, which contain food nutrient data such as calories, protein, fat, carbohydrates, vitamins, and minerals, as well as a dynamic nutrition knowledge graph. The AI ​​analysis unit is used to perform structured processing on the dietary information and lifestyle questionnaire data input by the user using a multimodal large language model and RAG technology, extracting food names, quantities and units, and to determine the user's input energy and intake of various nutrients based on the food nutrient composition data in the nutrition database. The nutrition assessment unit is used to conduct a multi-dimensional assessment of the user's nutritional intake based on the user's individual physiological parameters, health goals, dietary reference intake, and lifestyle questionnaire data determined by experts. It is also used to determine the user's nutrient input based on the results of the multi-dimensional assessment of the user's nutritional intake and generate a health risk index based on dietary data.

6. The nutrition measurement and dietary recommendation platform according to claim 1, characterized in that, The AI ​​analysis unit includes: An image recognition subunit is used to identify the types of food in an image; The portion estimation subunit is used to estimate the portion size of food in the image; The multimodal fusion subunit combines image recognition results with the user's text input to generate the user's dietary information.

7. The nutrition measurement and dietary recommendation platform according to claim 1, characterized in that, The dietary recommendation module is also used for comparing and selecting multiple dietary plans, as well as for generating personalized weekly menus based on user input.

8. The nutrition measurement and dietary recommendation platform according to claim 1, characterized in that, It also includes a user interface module for displaying nutritional analysis results, health education content, dietary recommendations, and AI Q&A content, and provides a user input interface and operation controls, as well as an interactive interface for an expert-defined lifestyle questionnaire.

9. The nutrition measurement and dietary recommendation platform according to claim 1, characterized in that, The user interface module is also used to visually display the user's nutritional intake, specifically by showing the intake ratio of energy, macronutrients and micronutrients through charts, as well as a real-time comparison with the recommended intake.

10. A method for nutritional measurement and dietary recommendation based on a multimodal large language model, characterized in that, The method includes: The input information acquisition module controls the acquisition of word information, sentence information and / or image information entered by the user in multiple target input areas, and controls the acquisition of the input information to parse the word information, sentence information and / or image information to determine at least one target food type and at least one food quantity; The control nutrition analysis module, based on multimodal large language model, retrieval enhancement generation technology and authoritative nutrition database, determines the user's nutritional element content and dynamic assessment results based on the absence of one target food type and at least one food quantity, and determines the user's health risk index based on the user's nutritional element content and dynamic assessment results. The health education module provides nutritional and health knowledge content. The dietary recommendation module generates highly personalized dietary suggestions based on the user's nutrient levels, dynamic assessment results, personal physiological parameters, and health goals, and supports the comparison and selection of multiple dietary plans; The AI-powered question-and-answer module responds to users' nutrition-related questions, providing real-time intelligent answers, dietary planning, and chronic disease management advice based on a multimodal large language model, RAG technology, and a professional knowledge base.

Images