Human-in-loop diet monitoring robot based on body intelligence

Abstract

The invention discloses a human-in-the-loop diet monitoring robot based on body intelligence. The human-in-the-loop diet monitoring robot at least comprises a food information acquisition module, a food identification module, a human-computer interaction module, an identification model incremental learning module, a diet recording module and a diet analysis module. The system obtains food information, and performs real-time identification and recording in combination with a machine learning algorithm. Besides, the man-machine interaction module provides an intelligent diet accompanying function by utilizing cue word engineering and improves the diet experience of the user, and the diet analysis module carries out nutritional ingredient analysis on diet records through an image-text interaction deep fusion technology based on a multi-modal large model and generates personalized diet suggestions according to the diet condition of the user. The system is combined with a healthy diet recording and analyzing function to ensure data synchronization and analysis. According to the method, the adaptability of new food types is remarkably improved, and reliable technical support is provided for intelligent diet monitoring, diet accompanying and healthy diet analysis.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction, and mainly relates to a human-in-the-loop diet monitoring robot based on embodied intelligence. By introducing a human-in-the-loop interaction mechanism, the robot dynamically optimizes its own food recognition model in combination with user feedback, continuously improving the recognition accuracy and adaptability. At the same time, the robot has the functions of diet monitoring and recording. By capturing food images, identifying food categories, and analyzing the nutritional components and providing personalized diet suggestions for the user's diet status, it realizes the recording and management of the user's diet behavior. On this basis, the robot also incorporates a moderate diet companionship function, providing simple chat interactions such as diet suggestions through natural interaction with the user, further enhancing the practicality of the technology. Background Art

[0002] In recent years, China's economy has achieved rapid development, the living standards of the people have been significantly improved, and the diet structure has gradually shifted from the initial problem of food and clothing to the pursuit of a more healthy, nutritious and delicious lifestyle. With economic growth, the diet culture has become increasingly diversified, and the eating habits of combining Chinese and Western elements have been widely spread. Chinese fast food, exotic flavors and international foods have gradually become common choices on modern dinner tables. At the same time, with the enhancement of health awareness, people's attention to dietary balance and nutritional value has also increased day by day, and more people tend to choose healthy diet ways such as organic foods, vegetarian diets, low sugar and low salt. The new diet concept has gradually been accepted by the public and widely spread.

[0003] However, the fast-paced life and work pressure in modern society have caused many people to face the problem of irregular diet. Especially single people, the elderly or those working away from home. Due to their busy work and lack of time, they often rely on takeaways or fast food, and it is difficult to ensure a balanced diet. The lack of scientific diet recording and management has also led to many people having bad eating habits in their diet, such as excessive intake of oil, salt or sugar, and lack of necessary nutrients. Such diet behaviors not only affect health, but may also lead to health problems such as obesity and chronic diseases. Therefore, how to establish a scientific and personalized diet recording and feedback system to help people optimize their diet structure has become an important topic in modern health management.

[0004] Currently, there are various diet monitoring systems based on human-computer interaction in the field of health management. These systems can collect users' diet data and provide corresponding health advice. However, existing systems have many limitations. Traditional health management systems mainly rely on users to manually record diet data. Such passive recording methods have multiple problems. For example, users are prone to omitting records due to forgetfulness or time constraints, the recorded content may be distorted due to memory bias, and users need to interrupt the dining experience to complete the recording during the meal, reducing the user experience. While existing automated systems can record in real time, they still face challenges in food recognition. The same dish may show significant differences due to factors such as cooking methods and presentation styles, resulting in insufficient model recognition accuracy; in addition, the same dish in different regions and restaurants may have different production characteristics, further increasing the difficulty of automatic recognition.

[0005] Therefore, how to build an intelligent diet monitoring system that can accurately identify diverse foods, reduce the user's recording burden, and maintain the naturalness of interaction is an important direction in current human-computer interaction research. Based on this, the embodied intelligent human-in-the-loop diet monitoring system emerges as an innovative solution. This system can not only actively capture and identify users' diet behaviors, but also ensure the accuracy of data by introducing a human-in-the-loop interaction mechanism, and use embodied intelligence and intelligent interaction to achieve a natural and friendly user interaction experience, realize diet companionship, and enhance the diet experience. Summary of the Invention

[0006] Technical Problem: The present invention aims at the problems in the prior art and provides a human-in-the-loop diet monitoring robot based on embodied intelligence. This robot supports food category recognition: when the user is dining, the robot can obtain the position information of the food the user eats, and then conduct category recognition on the food to obtain the food type. This robot supports active interaction to confirm the recognition result: according to the recognition result, the robot actively interacts with the user to confirm whether the recognized food type is correct. When the misrecognized food is corrected through interaction with the user, the robot will record the relevant information of the food and use this record to improve its recognition ability. This robot supports food data analysis: it can conduct certain nutritional component analysis and personalized diet advice on food records to help users adjust their diet structure and achieve health management goals.

[0007] To achieve the above object, the technical solution adopted by the present invention is as follows: A human-in-the-loop diet monitoring robot based on embodied intelligence includes at least a food information acquisition module, a food recognition module, a human-computer interaction module, an identification model incremental learning module, a diet record module, and a diet analysis module.

[0008] The food information collection module: configured to capture relevant data of the food consumed by the user, record the data in real time, and upload it to the food recognition module and the recognition model incremental learning module.

[0009] The food recognition module: used to identify the food category consumed by the user, output the corresponding category information and its confidence level, and provide data support for the subsequent human-computer interaction module.

[0010] The human-computer interaction module: responsible for managing the interaction between the user and the robot, having a real-time conversation with the user by combining the recognition result and confidence level output by the food recognition module, receiving and parsing the user input to further confirm the food type information, and feeding it back to the recognition model incremental learning module. This module can also have a conversation with the user to achieve diet companionship.

[0011] The recognition model incremental learning module: based on the human-machine collaborative incremental learning mechanism, combines the food category information feedback by the user with the collected food data to optimize the food category recognition model, and continuously improves the model recognition accuracy and the adaptability to new categories.

[0012] The diet record module: configured to merge and store the collected food data, the food category information confirmed by the user, and the food nutrition components and personalized diet suggestions feedback by the diet analysis module.

[0013] The diet analysis module: uses the trained multi-modal large model to extract features from the food data, generates food nutrition components while providing personalized diet suggestions, and feeds them back to the diet record module.

[0014] As an improvement of the present invention, the food information collection method includes one or a combination of mobile device automatic capture, user upload, mobile application recording, or other methods supporting image data collection.

[0015] As an improvement of the present invention, the food recognition module is based on a machine learning model supporting image detection, and generates food category information and corresponding confidence information by extracting features and analyzing the collected food images.

[0016] As an improvement of the present invention, the human-computer interaction module supports multi-modal interaction, where the input includes but is not limited to text input and voice input, etc., and the output includes but is not limited to text output, voice output, and robot embodied actions, etc. The human-computer interaction module can interact in a combination of one or more of the above input and output methods, and has the functions of actively initiating conversations and responding to user requirements and instructions, so as to achieve intelligent human-computer interaction;

[0017] The human-computer interaction module supports human-in-the-loop food recognition, communicates with the user in real time based on the food category information and the corresponding confidence level output by the food recognition module, and obtains the user's feedback information to confirm the food category information consumed by the user. In addition, the human-computer interaction module communicates with the user for diet companionship through a certain interaction strategy - Prompt Engineering, provides diet suggestions, supports the sharing of health knowledge or interactive chatting, etc., to enhance the user's dining experience.

[0018] As an improvement of the present invention, the recognition model incremental learning module dynamically optimizes the basic recognition model through the food category information confirmed after interacting with the user's feedback, and realizes the rapid adaptation of the model to new categories in an incremental learning manner;

[0019] The recognition model incremental learning module combines the collected food image data with the food category information obtained from the user's feedback, expands the training data set, and performs hybrid sample incremental learning based on the food category recognition basic model to generate an optimized recognition model, thereby realizing the collaborative learning between humans and machines and the continuous improvement of the model.

[0020] As an improvement of the present invention, the diet record module can merge and store and manage the user's diet data and the data of diet analysis feedback. The storage methods include but are not limited to local storage, cloud storage, database storage, etc. The stored content includes but is not limited to food images, food categories, time, location, and information such as nutritional components and diet suggestions obtained from subsequent feedback, providing a traceable diet record for the user.

[0021] As an improvement of the present invention, the diet analysis module is based on a multi-modal large model trained by images and texts, combines the deep feature extraction of food images with scalable task-specific regression prediction, analyzes food images through image embedding and multi-modal interaction attention mechanisms, generates a list of ingredients in the food images, and on this basis, uses a linear hierarchical regression algorithm to predict the indicators of calories, proteins, fats, and carbohydrates of each ingredient, and at the same time generates personalized diet suggestions based on the text large model.

[0022] For the human-in-the-loop diet monitoring robot system based on embodied intelligence, the usage method includes the following steps:

[0023] S1: The robot captures and records food pictures,

[0024] S2: Use the pre-trained food category recognition model to recognize the food pictures, obtain the food category information and its corresponding confidence level, and transmit the information to the human-computer interaction module.

[0025] S3: The robot generates corresponding interaction texts based on the food category information and confidence levels and initiates interactions to confirm the food categories. The robot supports diet-related dialogue interactions to achieve companion communication, and provides diet suggestions and health knowledge based on a preset diet knowledge base.

[0026] S4: Integrate the user feedback food category information into the incremental learning dataset, and optimize the basic recognition model through the incremental learning mechanism to improve the recognition performance for new food categories.

[0027] S5: Associate and store the food pictures with the confirmed food category information to achieve diet monitoring records, support user access. At the same time, through the configured multi-modal large model, conduct nutritional component analysis on the user's daily diet data and generate personalized diet suggestions for the user.

[0028] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0029] (1) The present invention combines artificial intelligence and human-computer interaction technology, and realizes the accurate monitoring of diet information and the dynamic optimization of the food recognition model based on the idea of human-in-the-loop. The present invention provides diverse and scientific support in health behavior tracking, can help users more effectively achieve diet recording and analysis, and assist in promoting changes in health behaviors.

[0030] (2) The diet monitoring robot of the present invention realizes the automatic collection of food information, and actively captures food images based on embodied intelligence. Compared with the traditional method that relies on users to manually record, it greatly reduces the user burden.

[0031] (3) The present invention conducts food recognition based on a machine learning model, continuously optimizes the recognition model through incremental learning, and improves the adaptability of the system to new food categories.

[0032] (4) The human-computer interaction module of the present invention can both actively initiate conversations and respond to user needs. This interaction module can generate personalized communication content in combination with the user's diet monitoring results, providing an intelligent interaction experience.

[0033] (5) The present invention integrates user feedback data into the incremental learning process through the "human-in-the-loop" mechanism to form a closed-loop optimization, continuously improving the recognition accuracy and robustness of the system.

[0034] (6) The present invention conducts diet analysis based on a multi-modal large model, and promotes users to develop scientific and reasonable eating habits through intelligent analysis and real-time feedback, thereby improving health management and quality of life in the long term.

[0035] (7) The application scenarios of the present invention are extensive and applicable to various scenarios such as family dining, single-person dining, and restaurant service. In addition, the user coverage of the present invention is wide, meeting the personalized needs of ordinary users and special groups (such as the elderly and patients with chronic diseases). BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0037] Figure 1 is the system working flowchart;

[0038] Figure 2 is the working flowchart of the system on the mobile robot; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] The present invention will be described in detail below in conjunction with the drawings and specific embodiments. Note that the following description of the embodiments is only a substantial example, and the present invention does not intend to limit the objects or uses to which it is applied, and the present invention is not limited to the following embodiments.

[0040] For the convenience of understanding, the terms involved in the embodiments are first explained.

[0041] Incremental Learning: A machine learning method whose core lies in that the model can dynamically update through small-scale newly added data or tasks without losing the existing knowledge, and there is no need to access the complete historical data or perform a comprehensive retraining. Compared with the traditional batch learning method, incremental learning can retain the original model capabilities while continuously updating knowledge, avoiding the problem of forgetting old knowledge due to the introduction of new knowledge, that is, "catastrophic forgetting". This feature is particularly important in data scenarios that require real-time updates or processing of dynamic environments. For example, in the target monitoring of the real-time monitoring system involved in the present invention, when the camera captures a new type of object, the model can quickly learn to recognize the new object without affecting the recognition ability of the old object. Its importance is reflected in the following three aspects: efficiency, adaptability, and scalability. These characteristics make incremental learning one of the key technologies in large-scale data stream processing, online learning, and real-time intelligent systems.

[0042] Human-in-the-loop: Human-in-the-loop is a branch of artificial intelligence that utilizes human and machine intelligence to create machine learning models. It integrates personal judgment into the AI system process, forming a closed loop for the process to run, and supports repeated operation with relatively fixed steps and states.

[0043] Prompt Engineering: In the field of Natural Language Processing (NLP), it is the process of designing and optimizing the input text, i.e., the prompt, to effectively guide the AI model to generate the expected output. With the wide application of large language models, prompt engineering has become increasingly important. Its core goal is to influence the response quality and accuracy of the model by designing accurate, concise, and guiding prompts. Prompt engineering is becoming increasingly important for leveraging existing large-scale pre-trained models because these models do not require retraining, but their performance depends to a large extent on whether the input prompts are clear and accurate. By optimizing the prompts, the effectiveness of the model can be improved and a large amount of time and resources can be saved.

[0044] Example:

[0045] Such as Figure 2 Describes the work process of the system applied to a mobile robot. The core includes a food collection module, a food recognition module, a human-machine interaction module, an incremental learning module for the recognition model, and a diet record module. The work process is as follows:

[0046] 1. Automatic food information collection

[0047] Based on a multi-modal perception system, the robot combines visual perception and ultrasonic detection functions to efficiently locate the specific position of the food consumed by the user, and uses a mobile camera to capture images of the food target in real time. For example, when the robot detects food in front of the user, it will immediately stop moving and activate the camera to collect food images. The whole process incorporates a real-time obstacle avoidance strategy, enabling the robot to flexibly respond to the surrounding environment while efficiently completing the collection task.

[0048] 2. Food type recognition

[0049] The food images collected by the robot will be sent to the food recognition module for processing. In the example of the present invention, the system uses the YOLO target detection model. The model is fast, good at small target detection, and adapts to food classification needs. The model is based on the ChineseFoodNet dataset and was trained for about 30 hours with the support of the high-performance graphics card NVIDIA RTX 4090. After the training, the performance of the model on the test set is as follows: Precision (precision) reached 0.81, Recall (recall rate) reached 0.78, and mAP50 (average precision, IoU = 0.50) was 0.86. The overall training results performed well. No pre-trained weights were used during the training process, and independent training was started from scratch, and the training time was about 30 hours. The trained system can quickly identify targets and frame them in the form of bounding boxes.

[0050] 3. Human-computer interaction

[0051] The identified food categories will serve as the basis for human-computer interaction. The robot communicates with the user through voice to verify the actual type of food. A multi-level prompt word strategy is designed for this interaction. For example, when the recognition confidence is high, the system will use an affirmative tone and describe the appearance characteristics of the food to attract users to participate in the conversation; when the confidence is low, the system will directly ask the user for more information. In order to make the conversation more natural, the tone of the system has been specially designed to avoid overly mechanical expressions and emphasize a relaxed style that is close to human communication habits. During the conversation, the system can adjust its strategy based on user feedback. If the user objects to the recognition results, the system will frankly admit the mistake and guide the user to provide further information. This flexible conversation capability not only makes users feel a more comfortable user experience, but also effectively improves the verification efficiency of the system. Specifically, the prompt words are designed as follows:

[0052] Interaction design principles and conversation style:

[0053]

[0054] Task execution steps and dialogue strategies:

[0055]

[0056] In order to improve the user's auditory experience, the system uses the TTS model for sound output to ensure the accuracy of the output voice. In addition, the system can provide users with a variety of voice options, and users can choose a more pleasing voice according to their personal preferences to obtain better human-computer interaction effects.

[0057] 4. Incremental learning and system optimization

[0058] The food category information feedback by users will be automatically recorded by the system and added as new data to the mixed dataset for incremental training of the model. In the training strategy of the present invention, the bottom-layer parameters are frozen and the top-layer parameters are unfrozen, enabling the model to not forget the old categories when learning new categories. This dynamic learning mechanism allows the system to maintain stable recognition performance when facing more diverse food scenarios.

[0059] 5. Diet Record and Analysis

[0060] To facilitate users' query and management of diet records, the system stores the food categories and related images recognized each time in a web database. In the diet analysis function, the present invention performs feature analysis on the food data in the web database based on the trained multi-modal large model. The system uses the Qwen-VL multi-modal large model, combines the open-source datasets VIREO Food-172 and Nutrition5K for multi-task joint training, and uses Lora for fine-tuning. After training and fine-tuning, the performance on the test set is excellent, with the Precision reaching 0.78. The model can accurately output a list of ingredients containing nutritional information such as calories, protein, fat, and carbohydrates based on the input food images and labels, and provide personalized diet suggestions through the text generation module. All analysis results and suggestions can be displayed in real-time in the web database for easy access and management by users.

[0061] In summary, the diet monitoring robot described in this example realizes a complete set of intelligent processes from food recognition to dynamic verification and then to health management through the combination of visual perception, machine learning models, natural language interaction technology, and multi-modal large models. It is not only applicable to home scenarios but also provides important support for solitary individuals and the health care field. In the future, with the continuous improvement of the system functions, the present invention can bring more intelligent and personalized diet and health solutions to users.

[0062] It should be noted that the above content only illustrates the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. For those of ordinary skill in the art of this technology, without departing from the principle of the present invention, several improvements and refinements can still be made, and these improvements and refinements all fall within the protection scope of the claims of the present invention.

Claims

1. A human-in-the-loop diet monitoring robot based on embodied intelligence, characterized in that: It at least includes a food information collection module, a food recognition module, a human-computer interaction module, a recognition model incremental learning module, a diet record module and a diet analysis module. The food information collection module is configured to capture the relevant data of the food consumed by the user, record the data in real time, and upload it to the food recognition module and the recognition model incremental learning module. The food recognition module is used to identify the food category consumed by the user, output the corresponding category information and its confidence, and provide data support for the subsequent human-computer interaction module. The human-computer interaction module is responsible for managing the interaction between the user and the robot, and conducts real-time dialogue with the user based on the recognition results and confidence output by the food recognition module, receives and analyzes user input to further confirm food type information, and feeds back to the recognition model incremental learning module. This module can also communicate with the user to provide dietary companionship. The recognition model incremental learning module: Based on the human-machine collaborative incremental learning mechanism, the food category information fed back by users is combined with the collected food data to optimize the food category recognition model, continuously improve the model recognition accuracy and adaptability to new categories, The diet record module is configured to associate and store the collected food data with the food category information confirmed by the user, generate complete diet record data, and can be connected to an external analysis module. The diet analysis module is configured to perform nutritional analysis on the diet record data, generate nutritional data of the user's intake, and provide personalized diet suggestions, which are fed back to the diet record module.

2. A human-in-the-loop diet monitoring robot based on embodied intelligence as claimed in claim 1, characterized in that: The food information collection method includes automatic capture by mobile devices, user upload, mobile phone application recording or other methods that support image data collection, and the collection methods are used alone or in combination of multiple methods; The food recognition module is based on a machine learning model that supports image detection. It extracts and analyzes features of collected food images to generate food category information and corresponding confidence information.

3. A human-in-the-loop diet monitoring robot based on embodied intelligence as claimed in claim 1, characterized in that: The human-computer interaction module supports multimodal interaction, wherein input includes text input and voice input, and output includes text output, voice output and robot embodied action. The human-computer interaction module can interact in a combination of one or more of the above input and output methods, and has the function of actively initiating dialogues and responding to user needs and instructions, thereby realizing intelligent human-computer interaction.

4. A human-in-the-loop diet monitoring robot based on embodied intelligence as claimed in claim 3, characterized in that: The human-computer interaction module supports human-in-the-loop food recognition, communicates with the user in real time based on the food category information and corresponding confidence output by the food recognition module, and obtains the user's feedback information to confirm the food category information consumed by the user. In addition, the human-computer interaction module communicates with the user about diet through a certain interaction strategy - prompt engineering, provides dietary advice, supports health knowledge sharing or interactive chat, etc., to enhance the user's dining experience.

5. The human-in-the-loop diet monitoring robot based on embodied intelligence as claimed in claim 1, characterized in that: The recognition model incremental learning module dynamically optimizes the basic recognition model through food category information confirmed after interactive feedback from users, and realizes rapid adaptation of the model to new categories in an incremental learning manner.

6. The human-in-the-loop diet monitoring robot based on embodied intelligence according to claim 5, characterized in that: By combining the collected food image data with the food category information obtained from user feedback, the training data set is expanded, and mixed sample incremental learning is performed based on the basic model of food category recognition to generate an optimized recognition model, thereby achieving human-computer collaborative learning and continuous improvement of the model.

7. The human-in-the-loop diet monitoring robot based on embodied intelligence according to claim 1, characterized in that: The diet record module can merge and store the user's diet data and related data of diet analysis feedback. The storage methods include local storage, cloud storage and database storage. The storage content includes food images, food categories, time, location information, and nutritional components and diet recommendation information obtained from subsequent feedback.

8. The human-in-the-loop diet monitoring robot based on embodied intelligence according to claim 1, characterized in that: The diet analysis module can analyze the nutritional components and dietary structure of the user's diet data. The analysis methods include external nutrition database, deep learning model, and multimodal large model. This module integrates the deep representation and feature fusion capabilities of the multimodal large model, generates a list of nutritional components by analyzing the food data of the diet record module, and generates personalized diet recommendations, which are fed back to the diet record module.

9. The human-in-the-loop diet monitoring robot based on embodied intelligence according to claim 1, characterized in that: The multimodal big model combines deep feature extraction of food images with scalable task-specific regression prediction. It parses food images through image embedding and multimodal interactive attention mechanism, generates a list of ingredients in the food images, and uses a hierarchical regression algorithm to predict the calorie, protein, fat and carbohydrate indicators of each ingredient. At the same time, it generates personalized dietary recommendations based on the text big model.

10. The human-in-the-loop diet monitoring robot based on embodied intelligence according to claim 1, characterized in that: The specific implementation includes the following steps: S1: The robot captures and records food pictures, S2: Use the pre-trained food category recognition model to recognize the food image, obtain food category information and its corresponding confidence, and transmit the information to the human-computer interaction module. S3: The robot generates corresponding interactive text based on food category information and confidence and initiates interaction to confirm food categories. The robot supports diet-related dialogue interaction, enables companion communication, and provides diet recommendations and health knowledge based on a preset diet knowledge base. S4: Integrate user feedback on food categories into the incremental learning dataset, optimize the basic recognition model through the incremental learning mechanism, and improve the recognition performance of new food categories. S5: Store food images in association with confirmed food category information to implement dietary monitoring records and support user access. At the same time, through the configured multimodal large model, analyze the nutritional composition of users' daily dietary data and generate personalized dietary recommendations for users.

Images