Intelligent food identification and diet analysis system based on multi-modal interaction

Abstract

The invention relates to the field of health management, in particular to an intelligent food identification and diet analysis system based on multi-modal interaction, which comprises an intelligent food scale hardware module, a multi-modal identification module, a food database module, a diet analysis module and a mobile phone end synchronization module. The intelligent food scale hardware module comprises a weighing sensor, a camera, a voice interaction unit and a communication unit; the multi-mode recognition module obtains food information through image recognition or voice recognition and matches the food information with the food database module. The diet analysis module calculates the actual intake according to the weighing data and the matched food information; and the mobile phone end synchronization module supports real-time transmission of data to a mobile phone end APP, and generates visual diet analysis reports and health suggestions. Through an image and voice bimodal recognition mechanism, the food recognition accuracy is remarkably improved, the intelligent weighing and data synchronization functions are combined, the user operation process is simplified, and accurate diet management and health guidance are achieved.

Description

Technical Field

[0001] The present invention relates to the field of health management, and in particular to an intelligent food recognition and dietary analysis system based on multimodal interaction. Background Art

[0003] Dietary survey is a key link in weight management. The commonly used method is the 24-hour dietary recall method, but this method has certain limitations: there may be recall bias and food quantification problems for the respondents, which greatly limits the accuracy of the survey. In recent years, dietary survey tools include: (1) Chinese Patent CN118430746A constructs a virtual food model through VR wearable devices, and further calculates the food weight through the volume of the virtual food. Although the survey process is simplified, it cannot solve the subjective influence of user recall bias and food quantification; (2) Chinese Patent CN117936030A uses the watershed method to segment images, determines the image boundary based on edge detection, and then calculates the image contour to determine the type and quantity of food. Then, combined with the size of the reference item, the actual intake of food is estimated. Although the survey time is shortened, it cannot accurately segment mixed dishes and the quantification problem of various foods in mixed dishes; (3) Chinese Patent CN117976144A calculates the food component content by comparing the food weight before and after meals and distinguishing the food types through food recognition technology. This method still does not solve the quantification problem of various foods in mixed dishes. Therefore, there is a certain error in the measured value, and the application scenario is relatively limited; (4) The survey methods of some patents are mostly limited to using traditional image recognition technology for food recognition, ignoring the key problem of one-to-one correspondence between food and its weight. Therefore, large errors will occur in the component measurement results, and accurate dietary management and health guidance cannot be achieved. Summary of the Invention

[0004] Aiming at the problems that the existing dietary recognition technology cannot accurately quantify mixed ingredients and relies on manual records with low efficiency, the present invention provides an intelligent food recognition and dietary analysis system based on multimodal interaction. Through the image and voice dual-modal interaction recognition mechanism, the food recognition accuracy is significantly improved. Combining the intelligent weighing and data synchronization functions, the user operation process is simplified, and accurate dietary management and health guidance are realized.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] An intelligent food recognition and dietary analysis system based on multimodal interaction, characterized by comprising: an intelligent food scale hardware module, a multimodal recognition module, a food database module, a dietary analysis module, and a mobile phone synchronization module.

[0007] The intelligent food scale hardware module includes a weighing sensor, a camera, a voice interaction unit, and a communication unit;

[0008] The multi-modal recognition module is used for various foods before cooking. It collects food images through the camera, extracts image features, matches them with the food database module to obtain food information. If the matching confidence is lower than the preset threshold of 75%, the voice interaction unit is activated to receive voice input, and through voice recognition conversion, it is secondarily matched with the food database module to obtain food information;

[0009] The food database module is used to store food names, image features, voice tag libraries, unit weight nutrient composition data, and associated codes, and supports dynamic expansion. Users can upload images, voice descriptions, barcodes of packaged foods, and their corresponding nutrition label information of custom foods through the mobile APP. After being reviewed by the system administrator, the cloud database is updated.

[0010] The dietary analysis module is used to receive the matching results (including associated codes) from the multi-modal recognition module and the gram weight data measured by the weighing sensor. By associating the unit weight nutrient composition data in the food database module, it calculates the carbohydrate, fat, protein, and trace element content recorded in the Sixth Edition of the Chinese Food Composition Table of the actually ingested food;

[0011] The mobile phone synchronization module transmits the data generated by the dietary analysis module to the mobile APP in real time through the communication unit to generate a visual dietary analysis report and health suggestions.

[0012] Further, the multi-modal recognition module includes an image recognition sub-module and a voice interaction sub-module,

[0013] The image recognition sub-module adopts an improved ResNet-50 model of convolutional neural network. By adding an SE attention module before the last fully connected layer, it improves the recognition accuracy of foods. The voice interaction sub-module supports the recognition of Shandong dialect, Cantonese, and dialects and colloquial expressions in Yunnan, Guizhou, and Sichuan provinces, and performs semantic analysis based on the BERT model fine-tuned with food domain corpus to optimize the matching results.

[0014] Further, the food database module also includes a special dietary library sub-module and a packaged food library sub-module,

[0015] The special dietary library sub-module stores data on infant formula foods, special medical foods, meal replacement foods, nutrient supplements, GI foods, and health foods;

[0016] The packaged food library sub-module stores the nutrition label information associated with the scanned GS1 standard commodity barcodes;

[0017] The special dietary library sub-module (31) and the packaged food library sub-module (32) also support dynamic expansion. Users can upload images, voice descriptions, barcodes of packaged foods, and their corresponding nutritional label information of custom foods through the mobile APP. After being reviewed by the system administrator, the cloud database is updated.

[0018] Furthermore, the mobile synchronization module further includes a nutritional balance display sub-module and a food recommendation sub-module.

[0019] The nutritional balance display sub-module supports users to independently input age, gender, weight, and health goals such as fat loss, muscle gain, blood sugar control, or salt control. Based on the "Dietary Reference Intakes for Chinese Residents" (2023 edition) standard, the daily recommended intakes are generated and compared with the actual intake data obtained by calculating through the dietary analysis module (4). It also supports historical data backtracking and time period analysis, and visualizes the nutritional balance in the form of charts.

[0020] The food recommendation sub-module makes reasonable dietary recommendations for nutritional components that are insufficient or exceed the recommended intakes according to the results of the nutritional balance display sub-module.

[0021] Furthermore, the communication unit supports Bluetooth and Wi-Fi, ensuring real-time data synchronization and being compatible with mobile terminals of iOS 12 and above and Android 8.0 and above systems.

[0022] The beneficial effects of the present invention are as follows:

[0023] The present invention meets the daily needs of users. Before cooking, the weight data of the washed ingredients weighed by the weighing sensor in the intelligent food scale hardware module is used, and the food information in the food database is matched through image or voice recognition, so as to perform dietary analysis. It not only eliminates the influence of users' recall bias and subjective quantification of foods, enables every kind of food to be avoided from being omitted and accurate weight data to be obtained, but also reduces the cumbersome operations of users frequently using mobile phones to record or take pictures.

[0024] Under the action of the nutritional balance display sub-module in the mobile synchronization module, the system can match the recommended intakes according to gender, age, weight, and health goals such as fat loss, muscle gain, blood sugar control, or salt control. Users can also customize the historical data backtracking time period, and visualize the comparison results with the actual intakes in the form of charts, strengthening users' awareness of dietary balance.

[0025] Under the action of the food recommendation sub-module in the mobile phone synchronization module, according to the analysis results of the customized historical data backtracking period, when the intake of a certain nutrient is insufficient or exceeds the recommended intake, the food recommendation sub-module will sort according to the content of this nutrient in different foods, and at the same time give food type and intake recommendations, and gradually guide users to reasonably plan their diets and maintain a healthy weight by pushing the corresponding food cooking methods. Brief Description of the Drawings

[0026] Figure 1 is the structural framework diagram of the intelligent food recognition and diet analysis system based on multi-modal interaction of the present invention.

[0027] Figure 2 is the overall flowchart of the intelligent food recognition and diet analysis system based on multi-modal interaction of the present invention.

[0028] In the figure, 1 - intelligent food scale hardware module, 11 - weighing sensor, 12 - camera, 13 - voice interaction unit, 14 - communication unit, 2 - multi-modal recognition module, 21 - image recognition sub-module, 22 - voice interaction sub-module, 3 - food database module, 31 - special diet library sub-module, 32 - packaged food library sub-module, 4 - diet analysis module, 5 - mobile phone synchronization module, 51 - nutritional balance display sub-module, 52 - food recommendation sub-module. Detailed Embodiments

[0029] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0030] Please refer to Figure 1 and Figure 2 , an intelligent food recognition and diet analysis system based on multi-modal interaction in a preferred embodiment of the present invention, includes an intelligent food scale hardware module 1, a multi-modal recognition module 2, a food database module 3, a diet analysis module 4, and a mobile phone synchronization module 5.

[0031] The intelligent food scale hardware module 1 includes a weighing sensor 11, a camera 12, a voice interaction unit 13, and a communication unit 14.

[0032] The multimodal recognition module 2 is used for various types of food before cooking. It collects food images through the camera 12, extracts image features, matches them with the food database module 3 to obtain food information. If the matching confidence is lower than the preset threshold of 75%, the voice interaction unit 13 is activated to receive voice input, and through voice recognition conversion, it is secondarily matched with the food database module 3 to obtain food information. Various types of food before cooking include two forms: before and after cutting. When the food matching fails, the system will prompt "Picture recognition failed. Please tell me the name of the food", and the user can use voice interaction to match with the food database 3 again to improve the accuracy of food recognition.

[0033] The food database module 3 is used to store food names, image features, voice tag libraries, nutrient content data per unit weight, and associated codes, and supports dynamic expansion. Users can upload images, voice descriptions, barcodes of packaged foods, and their corresponding nutrition label information of custom foods through the mobile APP. After being reviewed by the system administrator, the cloud database is updated. The food names and nutrient content data per unit weight come from the Sixth Edition of the Chinese Food Composition Table.

[0034] The dietary analysis module 4 is used to receive the matching results (including associated codes) from the multimodal recognition module 2 and the gram weight data measured by the weighing sensor 11. By associating with the nutrient content data per unit weight in the food database module 3, it calculates the carbohydrate, fat, protein, and trace element content recorded in the Sixth Edition of the Chinese Food Composition Table of the actually ingested food.

[0035] The mobile synchronization module 5 transmits the data generated by the dietary analysis module 4 to the mobile APP in real time through the communication unit 14 to generate a visual dietary analysis report and health suggestions.

[0036] The multimodal recognition module 2 includes an image recognition sub-module 21 and a voice interaction sub-module 22.

[0037] The image recognition sub-module 21 adopts an improved ResNet-50 model of convolutional neural network. By adding an SE attention module before the last fully connected layer, it improves the recognition accuracy of food.

[0038] The voice interaction sub-module 22 supports the recognition of Shandong dialect, Cantonese, and dialects and colloquial expressions in Yunnan, Guizhou, and Sichuan. Based on the BERT model fine-tuned with food domain corpus, it performs semantic analysis to optimize the matching results.

[0039] The food database module 3 also includes a special dietary library sub-module 31 and a packaged food library sub-module 32.

[0040] The special dietary library sub-module 31 stores data on infant formula foods, foods for special medical purposes, meal replacement foods, nutrient supplements, GI foods, and health foods; the packaged food library sub-module 32 stores the nutritional label information associated with the scanned GS1 standard product barcodes. At the same time, the food database module supports dynamic expansion. Users can upload images, voice descriptions, barcodes of packaged foods, and their corresponding nutritional label information of custom foods through the mobile APP. After being reviewed by the system administrator, the cloud database is updated.

[0041] The mobile synchronization module 5 also includes a nutritional balance display sub-module 51 and a food recommendation sub-module 52.

[0042] The nutritional balance display sub-module 51 supports users to independently input age, gender, weight, and health goals such as fat loss, muscle gain, blood sugar control, or salt control. Based on the "Dietary Reference Intakes for Chinese Residents" (2023 Edition) standard, it generates the daily recommended intake and compares it with the actual intake data obtained by the dietary analysis module 4. At the same time, it supports historical data backtracking and time period analysis, and visualizes the nutritional balance in the form of charts.

[0043] The food recommendation sub-module 52 makes reasonable dietary recommendations for the nutritional components that are insufficient or exceed the recommended intake according to the results of the nutritional balance display sub-module 51.

[0044] The nutritional balance display sub-module 51 of this embodiment can calculate the body mass index based on the user's height and weight, and by default, display all the information of relevant foods in the "Sixth Edition of the Chinese Food Composition Table" through the mobile APP. It can also customize the display of food-related information according to user needs. For example, if the user only focuses on calories, carbohydrates, proteins, and fats, the user can select the corresponding options in the nutritional balance display sub-module 51. In addition, the food recommendation sub-module 52 will also give corresponding dietary recommendations according to the results of the customized display of the nutritional balance display sub-module 51.

Claims

1. An intelligent food recognition and dietary analysis system based on multimodal interaction, characterized in that: include: Smart food scale hardware module (1), multimodal recognition module (2), food database module (3), dietary analysis module (4) and mobile phone synchronization module (5), The smart food scale hardware module (1) comprises a weighing sensor (11), a camera (12), a voice interaction unit (13) and a communication unit (14); The multimodal recognition module (2) is used for various types of food before cooking, and collects food images and extracts image features through the camera (12), and matches them with the food database module (3) to obtain food information; If the matching confidence is lower than a preset threshold value of 75%, the voice interaction unit (13) is activated to receive the voice input, and the voice is converted through voice recognition to perform a secondary match with the food database module (3) to obtain food information; The food database module (3) is used to store food names, image features, voice tag libraries, unit weight nutrient content data and associated codes, and supports dynamic expansion. Users can upload customized food images, voice descriptions, packaged food barcodes and their corresponding nutritional label information through a mobile phone APP, and the cloud database will be updated after review by a system administrator.

2. The dietary analysis module (4) is used to receive the matching result (including the associated code) from the multimodal recognition module (2) and the gram weight data measured by the weighing sensor (11), and calculate the carbohydrate, fat, protein and trace element content recorded in the "Chinese Food Composition Table Sixth Edition" of the actual ingested food by associating the unit weight nutrient component data in the food database module (3); The mobile phone synchronization module (5) transmits the data generated by the dietary analysis module (4) to the mobile phone APP in real time via the communication unit (14), thereby generating a visual dietary analysis report and health advice.

3. The intelligent food recognition and dietary analysis system based on multimodal interaction according to claim 1, characterized in that: The multimodal recognition module (2) comprises an image recognition submodule (21) and a voice interaction submodule (22). The image recognition submodule (21) adopts the improved ResNet-50 model of the convolutional neural network, and improves the accuracy of food recognition by adding an SE attention module before the last layer of full connection. The voice interaction submodule (22) supports the recognition of Shandong, Cantonese, Yunnan, Guizhou and Sichuan dialects and colloquial expressions, and performs semantic analysis based on the BERT model fine-tuned with food field corpus to optimize the matching results.

4. The intelligent food recognition and dietary analysis system based on multimodal interaction according to claim 1, characterized in that: The food database module (3) further comprises a special diet database submodule (31) and a packaged food database submodule (32). The special dietary library submodule (31) stores data on infant formula food, special medical food, meal replacement food, nutrient supplements, GI food and health food; The packaged food library submodule (32) stores nutrition label information associated with scanned GS1 standard commodity barcodes; The special diet library submodule (31) and the packaged food library submodule (32) also support dynamic expansion. Users can upload images, voice descriptions, packaged food barcodes and their corresponding nutrition label information of customized foods through the mobile phone APP, and update the cloud database after review by the system administrator.

5. The intelligent food recognition and dietary analysis system based on multimodal interaction according to claim 1, characterized in that: The mobile phone synchronization module (5) further includes a nutritional balance display submodule (51) and a food suggestion submodule (52). The nutritional balance display submodule (51) supports the user to independently input age, gender, weight and health goals of fat loss, muscle gain, sugar control or salt control, generates daily recommended intake based on the "Dietary Nutrient Reference Intake for Chinese Residents" (2023 Edition) standard, and compares it with the actual intake data calculated and obtained by the dietary analysis module (4), while supporting historical data backtracking and time period analysis, and visualizing nutritional balance in the form of charts; The food suggestion submodule (52) makes reasonable dietary recommendations for nutrients that are insufficient or exceed the recommended intake according to the results of the nutritional balance display submodule (51).

6. The intelligent food recognition and dietary analysis system based on multimodal interaction according to claim 1, characterized in that: The communication unit (14) supports Bluetooth and Wi-Fi, ensuring real-time data synchronization and being compatible with mobile terminals running iOS 12 and above, and Android 8.0 and above.

Images