Refrigerator food material management and health intervention system and method thereof

Abstract

The invention discloses a refrigerator food material management and health intervention system and method. The system comprises an identity authentication module, a multi-mode acquisition module, a control module, a health management engine and an interaction module. The identity authentication module outputs a user identity signal; the multi-modal acquisition module comprises a visual acquisition unit, a quality sensing unit and a voice input unit; the control module is in communication connection with other modules to dynamically construct a food material database; the health management engine generates an intervention instruction; the interaction module responds to the intervention instruction and outputs health suggestions; triple biological characteristic collaborative verification is achieved through the identity authentication module, closed-loop linkage of a dynamic food material database and a health management engine is formed in combination with a spatio-temporal data binding mechanism of the multi-modal acquisition module and the control module, and full-life-cycle management of food materials from storage to taking out is achieved. The automation degree of inventory updating and health intervention is improved, and the efficiency of household food material management and the health risk prevention and control capability are remarkably improved.

Description

Technical Field

[0001] This invention relates to smart home appliances, and more particularly to a refrigerator food management and health intervention system and method. Background Technology

[0002] With the rapid development of smart home technology, refrigerators, as core devices for household food storage, are receiving increasing attention for their management functions. Current refrigerator systems typically use a single sensor or rely on manual user input to track food. Some devices record food increases and decreases through basic weight detection or use cameras to scan packaging information, but lack a mechanism for collaborative collection and binding of multimodal data. Furthermore, user authentication is often limited to passwords or simple biometrics, making it susceptible to environmental interference and misidentification. These issues lead to fragmented food data collection: for example, weight changes are not spatially or temporally linked to image information; the shelf life of packaged food cannot be automatically correlated; manual descriptions of unpackaged food are prone to errors; and the system struggles to intervene in real-time based on the user's health status. These problems result in untimely updates to food inventory, leading to near-expiration waste or the risk of expiration, and a lack of health management, increasing the burden of chronic disease management, and reducing user experience and system reliability. Summary of the Invention

[0003] Purpose of the invention: The purpose of this invention is to provide a food management and health intervention system that integrates multimodal data acquisition, dynamic database construction, and hierarchical health intervention. Another purpose of this invention is to provide a food management and health intervention method using this system.

[0004] Technical Solution: The refrigerator food management and health intervention system of the present invention includes an identity authentication module, a multimodal acquisition module, a control module, a health management engine, and an interaction module. The identity authentication module is used to collect and verify the user's biometrics when the refrigerator door is opened and output the user's identity signal. The multimodal acquisition module includes a visual acquisition unit, a quality sensing unit, and a voice input unit, used to generate food image signals, quality change signals, and voice description signals. The control module is communicatively connected to the identity authentication module and the multimodal acquisition module to dynamically build a food database. The health management engine calls the corresponding health profile data based on the user's identity signal and generates intervention instructions in combination with the food database. The interaction module responds to the intervention instructions and outputs health suggestions. The output signal of the quality sensing unit triggers the start of the visual acquisition unit, and the data collected by both is spatiotemporally bound to the same food ID.

[0005] Preferably, the visual acquisition unit includes a main visual subunit facing the refrigerator door storage area and an auxiliary visual subunit facing the internal storage layer, and its field of view coverage area is associated with the partition detection position of the quality sensing unit; the quality sensing unit is partitioned and numbered according to the spatial position of the storage layer, and each partition independently outputs the quality change value and position code.

[0006] Preferably, the control module is configured to: extract text information from the image signal of the visual acquisition unit for packaged ingredients; and associate the voice description signal of the voice input unit with the image signal of the auxiliary visual subunit to establish an ingredient profile for unpackaged ingredients.

[0007] Preferably, the identity authentication module includes a fingerprint sensor integrated inside the refrigerator door's concealed switch handle and a facial recognition camera inside the refrigerator, used to collaboratively verify the user's identity.

[0008] Preferably, the identity authentication module further includes a voiceprint recognition unit, which together with the fingerprint sensor and the face recognition camera constitutes triple biometric verification; the refrigerator interior also includes a light.

[0009] Preferably, the text extraction logic of the visual acquisition unit includes: locating the packaging text area through an image recognition algorithm, extracting the product name, shelf life, and production date information, and associating it with the ingredient ID; when unpackaged ingredients are stored, the voice input unit receives the user's voice description and combines it with the image signal from the auxiliary visual subunit to generate a temporary file containing the ingredient image, name, storage time, and estimated weight; the control module calibrates the temporary file data after the quality sensing unit detects the actual quality.

[0010] Preferably, the health management engine executes: an early warning strategy for near-expiration based on food shelf-life data; a filtering strategy for contraindicated foods based on user medical history; and a planning strategy for generating personalized recipes by combining inventory data.

[0011] Preferably, it also includes a verification module. When the visual acquisition unit is started and no quality change signal is received, the interaction module is triggered to output a verification instruction to remind the user to put in the ingredients or cancel the entry.

[0012] Preferably, the main vision subunit is a main vision collector installed on the inside of the refrigerator door for scanning food packaging information, and the auxiliary vision subunit is an auxiliary vision collector on the inner wall of the refrigerator body; the mass sensing unit is a layer-by-layer numbered array of weight sensors embedded in the bottom of the internal partition and the bottom of the door storage compartment; the auxiliary vision collector is spatially associated with the weight sensors; and the voice input unit is a built-in microphone and speaker that are communicatively connected to the control module.

[0013] Preferably, the concealed switch handle has a groove structure, with the fingerprint sensor embedded in the bottom of the groove, so that the user's finger naturally contacts the sensor when opening the door.

[0014] Preferably, the health management engine connects to a cloud-based nutrition database and filters out prohibited ingredients based on the user's medical history linked to their identity information.

[0015] Preferably, each module is positioned based on the refrigerator body and the refrigerator side door.

[0016] The food ingredient management and health intervention methods based on the above system include the following steps:

[0017] S1: Generate user identity signal through biometric verification;

[0018] S2: When storing ingredients, if the ingredients are packaged, visual data acquisition is triggered and text information is parsed; if the ingredients are unpackaged, voice input is initiated and associated with visual images to create a file.

[0019] S3: Detects quality change signals; if no change is detected, outputs a verification command.

[0020] S4: When taking out ingredients, the ingredient ID is deduced from the weight reduction value and location code, and an intervention instruction is generated by combining the user's health profile.

[0021] S5: Based on the food database and health profile data, implement a tiered health strategy and output feedback.

[0022] Preferably, the graded health strategy in step S5 includes: a basic layer, which triggers an expiration warning when the difference between the shelf life of the ingredients and the current time is less than or equal to a threshold; a suggestion layer, which generates alternative ingredient recommendations when ingredients are retrieved and matched with the user's list of contraindications; and a planning layer, which generates weekly menus based on the combination of inventory ingredients and synchronizes them to the terminal.

[0023] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: By realizing triple biometric collaborative verification through the identity authentication module, and combining the spatiotemporal data binding mechanism of the multimodal acquisition module and the control module, a closed-loop linkage between the dynamic food database and the health management engine is formed, realizing precise management of the entire life cycle of food from storage to retrieval. This not only improves the automation level of inventory updates and health interventions, but also solves the problems of misidentification and delayed intervention caused by data fragmentation and lack of health guidance in the existing system, significantly improving the efficiency of family food management and the ability to prevent and control health risks. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the system control of the present invention.

[0025] Figure 2 This is a schematic diagram of the food visual recognition process of the present invention.

[0026] Figure 3 This is a schematic diagram of the health management logic of the present invention.

[0027] Figure 4-6 This is a schematic diagram of the system hardware location of the present invention. Detailed Implementation

[0028] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0029] like Figure 1-6 As shown in this embodiment, a refrigerator food management and health intervention system and method are disclosed. The core of the system includes an identity authentication module, a multimodal acquisition module, a control module, a health management engine, and an interaction module. The identity authentication module is used to collect and verify the user's biometric features when the refrigerator door is opened, and outputs a user identity signal. Specifically, this module integrates a fingerprint sensor 1 located in the recess of the refrigerator door handle and a face recognition camera 2 inside the refrigerator. The user needs to pre-register their fingerprint and facial information to a cloud database. When the refrigerator door is opened, fingerprint comparison initially confirms the identity, and the face recognition camera 2 performs secondary collaborative verification. To enhance reliability, the system also includes a voiceprint recognition unit. The user can pre-register their voiceprint information, which is matched with fingerprint and facial information to form a triple biometric verification. After successful verification, the system outputs a user identity signal, activates the lighting 7, and retrieves relevant user data.

[0030] After the refrigerator door is opened, the multimodal acquisition module begins operation. This module includes a visual acquisition unit, a mass sensing unit, and a voice input unit, used to generate food image signals, mass change signals, and voice description signals. The visual acquisition unit is specifically divided into: a main visual acquisition unit 3, located inside the refrigerator door, for scanning food packaging information (i.e., the main visual subunit); and an auxiliary visual acquisition unit 4, located on the inner wall of the refrigerator and spatially associated with the weight sensor 5 (i.e., the auxiliary visual subunit). The mass sensing unit is an array of weight sensors 5, numbered by layer, located at the bottom of the internal shelves of the refrigerator. The voice input unit is embedded inside the refrigerator door and includes a microphone and speaker 6 that communicate with the control module. The key linkage mechanism is that the mass change output signal detected by the mass sensing unit triggers the activation of the visual acquisition unit, and the data acquired by both are spatially and temporally bound to the same food ID, ensuring data correlation.

[0031] The control module communicates with the authentication module and the multimodal acquisition module, and is responsible for dynamically building and updating the food database. The food storage process is automated by the multimodal acquisition module and the control module working together. For packaged food, the user brings the packaging information close to the main vision acquisition unit 3, which uses the YOLOv5 algorithm to locate the text area on the packaging and uses the Tesseract engine to extract text information such as product name, shelf life, and production date. At the same time, the corresponding storage layer partition weight sensor 5 detects an increase in mass, and its signal triggers vision acquisition and ensures data binding. Users can also enter information about packaged food through the voice input unit. For unpackaged food, the user describes the food information (name, storage time, and estimated weight) through the microphone to generate a temporary file, which assists the vision acquisition unit 4 in simultaneously capturing and archiving images. The control module associates the voice description with the image signal to establish an "image-name-storage time" file. If the weight sensor 5 does not detect an increase in mass, the verification module triggers the interaction module to remind the user to put in the food or cancel the entry through the speaker. All food ingredient data (including location code, quality value, image, name, storage time, shelf life, etc.) is dynamically updated to the food ingredient database by the control module, forming associated data. After storage, the system automatically matches the food ingredient information, finds near-expiry foods or frozen foods that have been stored for too long, and reminds the user to remove them.

[0032] When food is removed, weight sensor 5 is triggered, detecting a change in weight. The system compares and retrieves information from the database, using the image and the changed weight value to determine the name of the removed food, updates the database information, and triggers health management. The mass sensing unit detects the weight reduction value and location code; the control module uses this to deduce the food ID, matches it to the database entry, and triggers the health management engine. The health management engine, based on the user's identity signal, calls the corresponding health profile data (such as medical history, BMI, dietary preferences, allergy information, etc.), combines it with the food database, executes a tiered health strategy, and generates intervention instructions.

[0033] The tiered health strategy mainly includes: a basic layer strategy that provides voice warnings when food is nearing its expiration date; a suggestion layer strategy that recommends alternatives when a user removes prohibited ingredients; and a planning layer strategy that generates personalized weekly recipes based on inventory and pushes them to the mobile app. The basic layer primarily focuses on expiration date reminders. After recognizing packaged food, it automatically associates the production date, expiration date, and food ID, reminding users of near-expiration items three days in advance. For unpackaged frozen foods, it reminds users after five days to prevent prolonged freezing from affecting food quality. Food storage time can also be set via voice. For short-term refrigerated storage, such as 2 hours for raw marinated food, the system will announce the time via speakerphone. For leftovers and other special foods, a time limit will be set, and if a user puts the food in the refrigerator multiple times the next day, the system will remind the user via voice. The suggestion layer recommends alternative ingredients based on the user's health status. After the refrigerator identifies the user, it will recommend specific foods based on the user's health condition. For example, if a person with high blood pressure mistakenly takes cheese, it will recommend replacing it with low-fat cheese. Based on the user's BMI or specific health status, if their daily calorie intake reaches a predetermined value, the system will remind them via voice to reduce their intake of high-calorie foods. The planning layer can generate recipes based on the food inventory and sync them to the mobile app. Users can input their predetermined recipe types, such as muscle gain or fat loss, as well as their favorite foods and allergies. The system can plan weekly recipes based on the food inventory, and users can also retrieve the day's recipe by voice input when opening the refrigerator. When retrieving ingredients, the refrigerator will alert the user to their location. Users can input their past medical history or dietary plans via the app or voice input, such as planning a fitness diet. The system can generate personalized health recipes based on the food inventory and the user's health profile. The interaction module responds to intervention commands from the health management engine and outputs health suggestions, primarily through voice broadcast. If the user re-stores uneaten food, the system updates the remaining weight data by recognizing it, and the voice input unit announces the remaining weight.

[0034] In addition, the system handles special situations: if the refrigerator door is not closed tightly or has been left open for an extended period, and the camera does not recognize the user's information, it will announce through the speaker that the refrigerator door is not closed properly to remind the user. Users can also set up dietary monitoring via voice or an app, and the system will output a reminder when the refrigerator is opened at a specified time, thus monitoring the user's health. For example, when preparing food or recipes for the next day, the refrigerator will recognize the user's information and match it with relevant memo information to remind the relevant user (if there is no memo information, this step is skipped). For example, if a user enters a memo saying they want to eat bread and drink milk before a business trip in the morning, the refrigerator will remind them of the corresponding location of the food in the refrigerator, and may light up the corresponding area to remind the user if necessary. Users can also enter memos to remind other users, for example, in a family, they can enter memo information to remind other users to eat fruit or to remind them that there is food left in the refrigerator. The system will compare the information in the memos after other users close the refrigerator, and when the user opens the refrigerator again, it will play this information, such as feedback that other users have eaten fruit on time. The health management engine connects to a cloud-based nutrition database, filtering prohibited foods based on the user's medical history, achieving full life-cycle management with zero operational burden.

[0035] The refrigerator's food management and health intervention methods include viewing user-entered memos and any dietary monitoring reminders. The system automatically queries the refrigerator's food database, comparing information to identify near-expiration items and issuing health alerts. Users can input information about packaged foods through multiple methods. A visual acquisition unit inside the refrigerator door scans text on food packaging, automatically extracting information such as expiration dates and production dates. When food is placed in the refrigerator, weight sensors on each shelf are triggered, automatically recording the food's weight. If no weight change is detected for an extended period, a voice input unit prompts the user to continue placing the food or cancel the placement. The system creates an information table with the placement time and stores it in the database. Users can also input information via voice input, such as expiration dates and estimated storage time. The system creates an information table and stores it in the database after detecting weight changes. Finally, the refrigerator's internal visual acquisition unit automatically captures images of all food placed inside, storing them along with the information table. When a user removes food from the refrigerator, it triggers the health management engine's suggestion layer. The system autonomously identifies the user's information, reviews pre-registered personal health information, identifies the removed food, and if any factors are found that could negatively impact the user's health, prompts the user to replace their food. When the user removes food, the system autonomously compares the reduced weight with an image of the weight reduction layer in the database, determines the food ID, and updates the database accordingly. If the user puts unused food back into the refrigerator, the system can also update the database. The system autonomously accesses the health management engine's planning layer using the user's voiceprint information. Based on the user's input requirements and the food types in the food database, it can plan recipes for the user, such as recipes for people with high blood pressure or weight-loss recipes. The system can detect the refrigerator door's closing status. If the door is left open for an extended period or not closed properly, it will announce this to the user via speaker. Users can also customize their voice; they can download voice packs through the app or record their own, which will then be used in subsequent refrigerator voice announcements.

Claims

1. A refrigerator food management and health intervention system, characterized in that, It includes an identity authentication module, a multimodal acquisition module, a control module, a health management engine, and an interaction module. The identity authentication module is used to collect and verify the user's biometric features when the refrigerator door is opened, and output the user's identity signal. The multimodal acquisition module includes a visual acquisition unit, a quality sensing unit, and a voice input unit, used to generate food image signals, quality change signals, and voice description signals. The control module is communicatively connected to the identity authentication module and the multimodal acquisition module to dynamically build a food database. The health management engine calls the corresponding health profile data based on the user's identity signal and generates intervention instructions by combining them with the food database; the interaction module responds to the intervention instructions and outputs health suggestions; wherein, the output signal of the quality sensing unit triggers the visual acquisition unit to start, and the data collected by the two are spatiotemporally bound to the same food ID.

2. The system according to claim 1, characterized in that: The visual acquisition unit includes a main visual subunit facing the refrigerator door storage area and an auxiliary visual subunit facing the internal storage layer. Its field of view coverage area is associated with the partition detection position of the quality sensing unit. The quality sensing unit is partitioned and numbered according to the spatial position of the storage layer, and each partition independently outputs the quality change value and position code.

3. The system according to claim 1, characterized in that: The control module is configured to: extract text information from the image signal of the visual acquisition unit for packaged ingredients; and associate the voice description signal of the voice input unit with the image signal of the auxiliary visual subunit to establish an ingredient profile for unpackaged ingredients.

4. The system according to claim 1, characterized in that: The identity authentication module includes a fingerprint sensor (1) integrated into the refrigerator door handle and a face recognition camera (2) inside the refrigerator, used to collaboratively verify the user's identity; the main vision subunit is specifically a main vision collector (3) set on the inside of the refrigerator door for scanning food packaging information, and the auxiliary vision subunit is specifically an auxiliary vision collector (4) placed in the cabinet partition; the mass sensing unit is specifically an array of weight sensors (5) embedded in the bottom of the internal partition and the bottom of the door storage compartment of the refrigerator, numbered according to the layer; the auxiliary vision collector is spatially associated with the weight sensor; the voice input unit is specifically a built-in microphone and speaker (6) that communicate with the control module.

5. The system according to claim 4, characterized in that: The identity authentication module also includes a voiceprint recognition unit, which together with the fingerprint sensor (1) and the face recognition camera (2) constitutes a triple biometric verification; the refrigerator also includes a lighting lamp (7).

6. The system according to claim 3, characterized in that: The text extraction logic of the visual acquisition unit includes: locating the text area of ​​the packaging through an image recognition algorithm, and extracting the product name, shelf life, and production date information, which are then associated with the ingredient ID; when unpackaged ingredients are stored, the voice input unit receives the user's voice description and generates a temporary file containing the ingredient name, storage time, and estimated weight; the control module calibrates the temporary file data after the quality sensing unit detects the actual quality.

7. The system according to claim 1, characterized in that: The health management engine executes: an early warning strategy for near-expiration based on food shelf-life data; and a filtering strategy for contraindicated foods based on the user's medical history. Planning strategies for generating personalized recipes by combining inventory.

8. The system according to claim 1, characterized in that: It also includes a verification module, which triggers the interaction module to output a verification command when no quality change signal is received after the vision acquisition unit is started.

9. A method for food ingredient management and health intervention based on the system described in claims 1-8, characterized in that, Includes the following steps: S1: Generate user identity signal through biometric verification; S2: When storing ingredients, if the ingredients are packaged, visual data acquisition is triggered and text information is parsed; if the ingredients are unpackaged, voice input is initiated and associated with visual images to create a file. S3: Detects quality change signals; if no change is detected, outputs a verification command. S4: When taking out ingredients, the ingredient ID is deduced from the weight reduction value and location code, and an intervention instruction is generated by combining the user's health profile. S5: Based on the food database and health profile data, implement a tiered health strategy and output feedback.

10. The method according to claim 9, characterized in that: The graded health strategy in step S5 includes: a basic layer, which triggers an expiration warning when the difference between the shelf life of an ingredient and the current time is less than or equal to a threshold; a suggestion layer, which generates alternative ingredient recommendations when ingredients are retrieved and matched with the user's list of contraindications; and a planning layer, which generates weekly menus based on combinations of in-stock ingredients and synchronizes them to the terminal.

Images