Cooking assistance program using artificial intelligence

Abstract

This cooking support system analyzes video, audio, and motion data during cooking in real time and provides appropriate advice to the user, enabling even beginners and children to cook safely and accurately. It also provides a cooking support program that connects the act of cooking with the rest of life, creating a comprehensive support environment. [Solution] The cooking support program uses a device equipped with a camera, microphone, various sensors, and AI processing functions to analyze video, audio, and motion data during cooking in real time, provide appropriate advice to the user, and provide multimodal feedback such as voice notifications, screen displays, and vibrations according to the user's visual information and cooking actions. It also provides individualized optimization according to the user's cooking skill level and preferences, and integrates with IoT refrigerators and online shopping sites for food management, utilization of special sale information, and recipe suggestion functions, thereby realizing comprehensive support that links the act of cooking with the whole life.

Description

Technical Field

[0001] The present invention relates to a cooking support program utilizing artificial intelligence (AI) technology. In particular, it relates to a cooking support system that uses a device equipped with a camera, a microphone, various sensors, and an AI processing function to analyze the progress of cooking in real time and provide appropriate advice to the cook. Further, the present invention is applicable not only to cooking at home but also to cooking guidance and educational support in cooking schools and restaurants, and even to safe cooking support for children and the elderly. It not only supports cooking actions but also contributes to improving the convenience of daily life as a whole.

Background Art

[0002] Conventionally, when novice cooks or children learn cooking, it has been common to refer to cookbooks, printed recipes, recipe websites on the Internet, recipe videos using video distribution services, etc. These means are useful in presenting the ingredients and procedures required for cooking statically, but in the situation where the user is actually proceeding with the steps while moving their hands, the content of the information presented is limited, so it is often insufficient to determine in real time whether the current cooking state is appropriate. In addition, auxiliary support devices such as voice assistants and cooking appliances with timers are also becoming more widespread, but these also basically return stereotypical responses to predetermined operations and have not yet reached the point of giving appropriate advice according to the specific situation during cooking and the user's proficiency.

[0003] For example, typical recipes use phrases like "sauté the onions until golden brown" or "simmer over medium heat," but while these instructions are intuitively understandable to experienced cooks, they can be difficult for beginners and children to judge. In reality, mistakes such as overcooking and burning, or undercooking and not reducing the sauce enough, happen frequently. Furthermore, basic cooking techniques such as how to cut and mix ingredients are difficult to reproduce without sufficient knowledge and experience, which can affect the appearance, texture, and cooking time of the food. In addition, unexpected problems often arise during cooking, such as the pot size being inappropriate for the amount of ingredients, difficulty in sautéing due to more moisture than expected, or smoke and burning due to excessive oil temperature, which are not mentioned in the recipe. However, conventional cooking assistance tools do not have the function to immediately respond to these specific problems and provide appropriate solutions.

[0004] In recent years, technologies that utilize visual and auditory information to analyze cooking actions have been researched, but most of these are primarily aimed at generating cooking records and analyzing them later, and are not sufficient for real-time support during cooking. Furthermore, interactive interfaces utilizing the user's gaze, movements, and voice, recipe suggestions tailored to cooking skills, and interactive support from characters are being considered in some areas, but these are treated as separate elemental technologies, and a system for integrating them into the overall cooking support system has not yet been established. A comprehensive system that provides real-time support for the entire cooking process, closely linked to the user's actual environment and actions, including integration with IoT refrigerators, ingredient selection support utilizing sale information, and learning functions based on the user's cooking history, is still in the process of being put into practical use. Against this backdrop, there is a need for new cooking support technologies that can acquire and analyze the cooking situation in real time and provide appropriate and immediate feedback according to the user's situation and attributes. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2017-199296 [Patent Document 2] Japanese Patent Publication No. 2023-170229

[0006] In contrast, as an example of prior art, a system has been disclosed that records the cooking procedures performed by a cook and automatically generates a cooking recipe based on that information (see, for example, Patent Document 1). In this prior art, in order to record the cooking process, the cook wears glasses-type wearable terminals to acquire visual information, and also acquires hand and arm movement data through a wristband-type terminal worn on the wrist. This information is transmitted to an information processing device, where the cooking process is analyzed, and a recipe is generated based on the individual actions of the cook. In other words, this technology aims to record and formalize an individual's cooking actions and accumulate them as reproducible know-how, with a focus on process documentation.

[0007] However, the conventional technology described above does not provide real-time advice or guidance functions that allow users to receive assistance as they progress through the cooking process. The cooking procedure is only analyzed and recorded afterward, and immediate advice such as "The heat is too high right now," "Be careful not to overcook," or "Your knife movements are unstable" is not available. In particular, for users who need continuous decision-making support during cooking, such as cooking beginners or children, the lack of real-time advice may cause anxiety in operation or lead to repeated cooking mistakes. Furthermore, even if hesitation arises during the process, the system does not understand the user's intentions and does not proactively provide support. In contrast, the present invention differs significantly from conventional technology in that it analyzes the user's actions at each step using video, audio, motion, and environmental data during cooking, and provides real-time advice according to the progress. In other words, by realizing a real-time and interactive support environment in which cooks can receive assistance without stopping, it is possible to simultaneously improve the quality and safety of cooking.

[0008] Another conventional technology is a system that determines the safety of the object being heated in a cooking appliance and prevents the heating of inappropriate objects (see, for example, Patent Document 2). This technology uses sensors and cameras to analyze the shape and material of the object being heated, and if an object unsuitable for heating, such as metal, is placed on the appliance, it automatically stops heating or changes the heating method. This makes it possible to prevent misuse that could cause fires or accidents, especially in microwave ovens. It is also useful in that it can accurately identify objects that should not be heated by distinguishing between food and non-food items, thereby ensuring safety.

[0009] However, the technology described in Patent Document 2 is not a system that provides support for the entire cooking process, but is limited to a single function: ensuring the safety of the heating process itself. For example, it does not have functions to make judgments or provide support based on the quality and progress of cooking, such as whether the stir-frying is done properly, how far along the simmering is, or whether there is room for improvement in the cutting or mixing methods. Furthermore, it does not envision advanced support functions such as adaptive advice that adjusts the guidance content considering the user's skill level and learning history, or integration with a character-type interface that is empathetic to emotional responses. Moreover, it does not include elements such as recipe suggestion functions linked to the storage status of ingredients and special sale information, or sharing of results after cooking and community interaction with others, so the scope of cooking support is very limited. The system according to the present invention is based on a fundamentally different technological concept from conventional technology in that it monitors all processes from the initial stages to the finishing touches of the cooking process in real time, and provides support that incorporates user-specific information, environmental factors, and even character elements. [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] The problem that this invention aims to solve is that when cooking beginners or children refer to a recipe while cooking, it is difficult for them to judge whether the current procedure is correct, whether the heat level is appropriate, or whether the cooking time is sufficient. In particular, beginners find it difficult to understand and implement vague expressions such as "sauté over medium heat" or "sauté until golden brown," and mistakes such as burning ingredients or proceeding to the next step before the food is cooked sufficiently often occur due to misunderstandings or misjudgments. In addition, a certain level of skill is required for knife skills and how to cut ingredients, and incorrect handling often negatively affects the appearance and texture of the food. Furthermore, conventional cooking support means mainly consist of presenting recipes and recording the process, and have not provided a system that allows users to receive advice in real time while cooking. As a result, the cause of cooking failures cannot be identified on the spot, and this does not lead to improvement in the future. Against this backdrop, this invention aims to provide a support system with a real-time feedback function so that cooks can correctly understand the current situation and proceed with cooking safely and reliably. [Means for solving the problem]

[0011] To solve the above problems, the present invention provides a cooking advice system that analyzes the cook's actions and cooking environment in real time and provides appropriate feedback using an AI processing unit equipped with multiple input devices such as a camera, microphone, and temperature, humidity, and motion sensors. The AI ​​utilizes a machine learning-based model and pre-registered pattern information to comprehensively evaluate the progress of the cooking process, the heat level, the processing status of ingredients, etc., and presents instructions and advice to the user according to the analysis results. The user interface combines a display, voice notification, and haptic feedback via vibration to achieve optimal information transmission according to the situation, even when the cook cannot see the screen or when the voice is difficult to hear. Furthermore, the system is configured to understand the user's attribute information (age, experience level, language setting, left-handed / right-handed, etc.) and provide optimized advice. The cooking advice system of the present invention has the flexibility to accommodate not only beginners and children who are unfamiliar with cooking, but also general users who value time efficiency and accuracy. [Effects of the Invention]

[0012] According to this invention, users can check whether their actions are correct while cooking and receive advice at the appropriate time as needed, thus preventing cooking errors and improving their confidence. For example, real-time feedback such as "The heat is too high," "The onions are still translucent," or "You've mixed it too much" allows users to follow recipe instructions more accurately and produce high-quality dishes. Furthermore, even if an error occurs in part of the process, the AI ​​can detect it early and suggest corrections, allowing for recovery in the next step and ultimately avoiding cooking failure.

[0013] Furthermore, the advice and level of detail are automatically adjusted according to the user's cooking skill level, allowing for flexible responses such as providing beginners with easy-to-understand explanations and advanced users with only the bare minimum of instructions. This reduces the annoyance of excessive instructions and, conversely, the anxiety caused by insufficient information, creating a comfortable operating environment for each user. In addition, providing positive feedback on the user's actions and results, such as "You're doing great!" or "Perfectly cooked!", boosts the user's confidence and increases their motivation to cook again.

[0014] This continuous and appropriate feedback allows users to feel their skills improving with each cooking session, freeing them from anxiety and stress related to failure. Especially for those who require assistance, such as children and the elderly, the step-by-step guidance while prioritizing safety makes it a valuable tool for home-based food education and supporting independent living. Furthermore, by integrating with peripheral functions such as character-based encouragement, visualization of progress records, sharing of cooking results, and support for ingredient purchase management, it can provide learning support that goes beyond mere task assistance, incorporating enjoyment. This invention, as a next-generation cooking support platform combining real-time capabilities, individual adaptability, and multi-sensory feedback, possesses both high practicality and educational value. [Modes for carrying out the invention]

[0015] In one embodiment of the present invention, the cooking advice system is configured using an information terminal such as a smartphone, tablet, or wearable device with a dedicated application installed. This application has the function of consistently supporting the user's cooking process from start to finish. The user launches the app before cooking and selects a recipe for the desired dish. Recipes can be searched and selected based on conditions such as cooking time, difficulty level, and necessary equipment and ingredients, and may also be automatically suggested according to the user's past cooking history and skill level.

[0016] Once cooking begins, the system uses a camera mounted on the device to photograph the cooking area, such as the countertop, pots, pans, and cutting boards, capturing video in real time. Simultaneously, it uses a microphone to collect audio data during cooking (e.g., the sound of ingredients frying, oil sizzling, boiling, the sound of the knife hitting the cutting board, etc.). The acquired video and audio are immediately transmitted to an AI processing unit built into the device or in the cloud, where the analysis of the cooking process begins. Based on this information, the AI ​​determines the progress of the cooking process, the appropriateness of the heat level, and the degree of heating, and provides specific advice to the user through means such as screen displays, audio notifications, and vibration notifications, such as "The heat is too high" or "The onions have turned golden brown."

[0017] Furthermore, this cooking support system is equipped with analytical functions that support not only the heating process but also "cooking tasks on a cutting board," such as preparing ingredients and shaping them. Specifically, it targets tasks such as cutting with a knife, shaping dumplings and hamburgers, precision tasks such as deboning and peeling fish, and tenderizing and pounding meat. Based on data acquired from cameras and various sensors (temperature sensors, humidity sensors, acceleration sensors, gyroscopes, gaze detection sensors, etc.) mounted on terminals and wearable devices, it analyzes the user's movement patterns.

[0018] For example, regarding knife movements, the system judges the blade angle, speed, and rhythm to determine if the cutting technique is appropriate. If the ingredients are cut at an angle or crushed due to excessive force, advice such as "Try raising the knife angle a little" or "Try cutting with less force" will be provided. In the shaping process, the system evaluates in real time whether the dumpling wrappers are evenly rolled out, whether the filling is spilling out, and whether the hamburger patties are of uniform thickness, providing feedback as needed.

[0019] In addition to the accuracy of the operation, analysis has been carried out with consideration for safety. If it is determined that the movement of the hands is unstable and there is a risk of injury, a warning such as "Your hands are in a dangerous position. Please be careful" will be issued. With such support, the cook can receive objective and immediate guidance by AI while concentrating on the work with both hands, enabling early correction of mistakes and improvement of skills.

[0020] Furthermore, these analysis results are recorded and accumulated as the user's operation history, so it is also possible to cooperate with the proficiency determination, individual recipe proposal, character-linked evaluation function, etc. described later. That is, this embodiment also has a configuration as a guidance-type learning support system that goes beyond mere cooking assistance.

Brief Explanation of Drawings

[0021] FIG. 1 is a block diagram showing a configuration example of a cooking advice system according to the present invention. FIG. 2 is a configuration diagram showing various hardware devices (smartphones, tablets, smartwatches, wearable devices, etc.) used in the cooking advice system of the present invention. FIG. 3 is a flowchart showing an example of providing advice to the user during cooking. FIG. 4 is a diagram showing an outline of a process for adjusting the content of advice based on user attributes. Detailed Description of Embodiment

[0022] As shown in FIG. 1, the cooking advice system 100 of the present invention includes a camera 110, a microphone 120, a sensor group 130, an AI processing device 140, a user interface 150, and a recipe database 160.

[0023] The camera 110 and the microphone 120 function as input means for acquiring the user's cooking operations and acoustic information during cooking, and acquire the video and audio during cooking in real time. The sensor group 130 is composed of a temperature sensor, a humidity sensor, an acceleration sensor, a gyro sensor, a line-of-sight detection sensor, etc., and comprehensively detects the cooking environment, the state of the cooking appliance, the user's actions, etc.

[0024] After these input information are acquired in real time from the camera, the microphone, and the sensors, they are transmitted to the AI processing device 140. Based on the mounted machine learning algorithms, the AI processing device 140 analyzes these information from various angles, and comprehensively determines the progress of cooking, the operation pattern of the cook, the speech content and voice tone, the ambient environmental sound, etc. As a result, the system can accurately grasp the current state of the user, and can automatically generate and present optimal advice at the appropriate timing and content.

[0025] Furthermore, the AI processing device 140 is connected to the Internet as needed, and by cooperating with the AI models and external databases on the cloud, more advanced and flexible analysis processing is realized. For example, by incorporating the action patterns of other users with similar cooking histories, the latest recipe data, external knowledge regarding the cooking characteristics of ingredients, etc., the quality of the advice is continuously improved. Also, since software updates and the download of new learning data are automatically performed via the Internet, the AI processing device 140 is configured to be able to respond to individual optimizations for each user while always maintaining the latest state.

[0026] The user interface 150 includes a display, a speaker, a vibrator, etc., and presents the generated advice to the user by screen display, voice notification, or tactile feedback.

[0027] In this embodiment, the cooking advice system is designed to be compatible with a variety of hardware, including smartphones, smartwatches, and wearable devices (such as Google Glass and AR devices) (see Figure 2). Specific examples for each hardware type will be described in detail below.

[0028] In the smartphone-based implementation, the system employs a configuration where the user utilizes their smartphone to receive cooking assistance. By making maximum use of existing hardware such as the camera, microphone, display, speaker, and vibration function that are standard on smartphones, this cooking assistance system can be easily and practically implemented without the need for additional dedicated equipment. By placing the smartphone near or on top of the cooking counter, the user can record the entire cooking process with video and audio. The recorded information is transmitted in real time to an AI processing unit, which immediately analyzes the cooking status. Based on the analysis results, feedback is provided in various ways, such as screen display, audio, and vibration.

[0029] Furthermore, the camera function allows the AI ​​to recognize and analyze the movement of ingredients and cooking utensils in detail. For example, it analyzes the degree of sautéing of onions (transparent, translucent, golden brown, etc.), the degree of browning on the surface of meat, the boiling status of liquids, as well as the movement and grip of knives and the usage of cooking utensils (angle of tongs and spatulas). This allows users to numerically and logically evaluate the cooking state based on visual information, rather than relying on their intuition. Meanwhile, the microphone function captures sounds during cooking (splattering oil, boiling pots, sizzling food, etc.) and analyzes them using an acoustic pattern recognition algorithm. Based on changes in sound intensity and frequency spectrum, it detects conditions such as the heat being too high, the mixture being overcooked, or moisture being lost, and prompts the user for timely warnings. The accuracy of these analyses is further improved through cross-checking of video and audio, and the risk of false positives and over-detections is reduced.

[0030] Feedback is primarily provided through smartphone screen displays. Large fonts and color coding clearly show the current progress, heat level, and next steps for easy user visibility. The displayed messages change according to the user's cooking skill level, offering step-by-step advice such as, "The heat level is just right," or "The onions are cooked through. Now add the pork." This ensures users always know what to do next without confusion, reducing the risk of mistakes. Voice notifications are also used in conjunction with the system, anticipating situations where the user cannot check their smartphone screen while cooking. For example, even with both hands occupied, voice guidance such as, "The pot is hot. Reduce the heat," will automatically play, allowing users to obtain information without relying on visual cues.

[0031] Furthermore, in high-priority situations, the smartphone's vibration function activates, providing immediate auditory and tactile warnings. For example, if signs of burning or overheating are detected, the vibration will attract the user's attention and prompt a quick response. This multimodal notification system, integrating visual, auditory, and tactile senses, allows users to receive timely feedback in any situation, significantly improving cooking safety and accuracy. In addition, the system actively supports users in households cooking multiple dishes simultaneously or when young children are cooking for the first time, greatly reducing the risk of failures and accidents and allowing them to cook with peace of mind.

[0032] The system of this invention supports not only smartphones but also smartwatches, further enhancing the accuracy and immediacy of cooking assistance through wearable devices worn by the user. Because smartwatches are worn on the user's wrist, they are the devices that can capture the cooking process most closely. Taking advantage of this characteristic, this cooking assistance system concisely displays the progress of the process and the next instructions on the smartwatch's display, such as "Please add the pork shortly" or "Simmering for 2 minutes remaining." Furthermore, touch operation on the display allows for control such as "Repeat the current process" or "Stop voice guidance," enabling flexible operation according to the user's cooking environment. This creates an environment where the latest information can always be accessed from the wrist, even when the smartphone screen cannot be frequently looked at.

[0033] Furthermore, the vibration function built into the smartwatch is extremely effective as a means of haptic feedback. Notifications related to the progress of cooking, such as warnings like "It's time to turn off the heat" or "Be careful not to overcook," can be conveyed to the user by short vibrations, ensuring that attention is reliably given even in noisy environments or situations where it is difficult to notice voices. In particular, when cooking multiple dishes simultaneously, it is possible to use different vibration patterns according to the state of each dish, which greatly contributes to avoiding cooking mistakes and optimizing the process. In addition, if the AI ​​processing determines that a dangerous action has occurred, such as an unstable grip on the knife or an inadvertent approach of the face or hands to a heating pot, the wearable device (smartwatch, etc.) will instantly vibrate to warn the user, preventing accidents. This ensures a safe cooking environment, especially for children and beginners. Moreover, the intensity and frequency of vibration notifications can be customized according to each user's sensitivity and preferences, and are automatically adjusted in the system settings, enabling personalized support that takes into account physical and sensory characteristics.

[0034] This cooking support system further utilizes biometric sensors such as accelerometers and heart rate sensors installed in wearable devices like smartwatches to grasp the user's psychological and physical state in real time. In particular, this system has the function to continuously analyze heart rate information acquired from the wearable device using artificial intelligence and estimate the user's tension level with high accuracy based on the trend of increase or decrease in heart rate and the fluctuation pattern. For example, if the heart rate rises sharply compared to normal during cooking, it is determined that the user may be mentally stressed or feeling anxious or confused. In such cases, this system displays considerate messages to alleviate the user's psychological burden, such as "Let's take a short break" or "It's okay to proceed without rushing," to promote stress relief. Furthermore, the results of this tension level estimation are not limited to mere message display, but are also actively used to determine the progress of the cooking work. For example, if it is detected that the user's tension has increased significantly during a particular process, the system will determine that that process is likely to be burdensome for the user, and will be able to provide flexible support tailored to the individual user's psychological state, such as adjusting the timing of the automatic transition to the next process, or providing additional explanatory displays or voice navigation. Furthermore, if the accelerometer does not detect hand movement for a certain period of time, it will assume that the user has interrupted their work and a gentle prompt such as "Shall we proceed to the next step?" will be displayed. Through the combined analysis of motion and biometric data by AI, interactive support that takes into account both the user's state and the cooking process is realized, providing a comfortable user experience that combines visual, auditory, and voice input.

[0035] Next, in this embodiment of wearable devices, the system configuration is adopted in which the user wears glasses-type or head-mounted wearable devices (e.g., Google Glass or equivalent AR devices) and receives hands-free, real-time advice by directly displaying cooking assistance information in their field of vision. The wearable glasses are equipped with a front camera, microphone, display, speaker, etc., and by utilizing this hardware, an AI processing unit collects and analyzes the user's visual, auditory, and motion information while cooking, providing immediate feedback. As a result, the user can receive appropriate instructions while concentrating on the task without taking their eyes off the table or stopping what they are doing while cooking.

[0036] One specific feature is the "information display in the field of view." The wearable device's transparent display shows the current cooking process, the next steps to take, and any points to note. For example, during the stir-frying process, it might display "Add the meat when the onions become translucent," or during ingredient preparation, it might display "Raise the angle of the knife a little more," providing visual guidance that is naturally superimposed on the user's field of view. This eliminates the need to constantly refer to traditional recipe books or smartphone screens, providing an intuitive and stress-free learning experience, especially for cooking beginners.

[0037] In an example using an AR device, the user wears an AR-enabled headset and can receive intuitive and immersive cooking advice by overlaying virtual objects onto the cooking space. The AR device utilizes spatial recognition technology to grasp the position of the cooking surface and utensils in real time, and projects virtual arrows and text onto the cooking surface based on that information. For example, it may display "Cut here" where ingredients should be cut, or "Put ingredients in this position" on the frying pan, allowing the user to proceed with cooking without hesitation. In addition, the correct use of cooking utensils and the ideal state of ingredients (e.g., browning and cutting method) are also presented as 3D models, promoting visual understanding.

[0038] Furthermore, the AR device is equipped with a camera and an AI processing unit, which capture and analyze the user's cooking actions in real time and provide immediate feedback. For example, the degree of doneness of the meat can be analyzed in 3D from the camera image, and advice such as "cook it a little more" can be displayed directly in the virtual space. Users can also use interactive functions that can be operated by voice and gestures, such as "show the next step" or intuitively controlling the virtual UI by waving their hand. This allows users to receive cooking assistance without interrupting their work even if their hands are full, realizing an immersive and hands-free support environment unique to AR. The AI ​​processing unit can be located outside the AR device, such as in the cloud, and connected wirelessly or via the internet, which may allow for a simpler and smaller AR device.

[0039] Furthermore, thanks to its voice recognition function, users can obtain necessary information by simply speaking commands such as "What's the next step?" or "How's the heat?" while cooking. The voice interface analyzes the user's speech using natural language processing and provides responses tailored to the current cooking status and recipe information. In addition, the system adjusts the content and expression of its responses based on the user's experience level and past cooking history, enabling personalized responses such as detailed explanations for beginners and concise instructions for advanced users.

[0040] In addition, the wearable device's front-facing camera enables gesture recognition, tracking the user's hand movements and posture in real time. For example, if the stirring motion is excessively fast, advice such as "Please stir slowly" is immediately displayed, providing timely feedback through both sight and sound. Furthermore, knife movements, ingredient placement, and the use of cooking utensils are also analyzed, contributing to the accuracy and safety of the actions.

[0041] The real-time feedback function is one of the key features of this invention, dynamically displaying information in accordance with the user's actions at each stage of cooking. This function allows the user to proceed with cooking in a natural flow without having to check the procedure step by step or stop what they are doing to keep track of the time. Furthermore, by combining this with voice output from a wearable device and vibration notifications linked to specific actions, multimodal information presentation is realized, making it possible to support the user's cooking actions from multiple angles.

[0042] Furthermore, the wearable device is equipped with an eye-tracking sensor that can analyze the user's gaze. This allows it to detect which parts of the cooking process the user is paying attention to or where they are unsure, providing more detailed support. For example, if the user is unsure how to hold the knife, it will provide detailed instructions such as, "Move the handle of the knife slightly further back." This eye-tracking advice provides support that is tailored to the user's thoughts and uncertainties, much like a human cooking instructor.

[0043] Thus, a cooking advice system using wearable devices integrates a multi-layered interface, including direct display in the user's field of vision, voice and gesture recognition, eye tracking, and character integration, allowing users to focus on the cooking experience with a sense of immersion. In particular, its hands-free operation and real-time feedback are strengths not found in other devices, giving it extremely high practicality and a wide range of applications as a cooking support system.

[0044] Furthermore, this cooking support system (common to all implementations using smartphones, smartwatches, and wearable devices; the same applies hereinafter) also features a function that utilizes the user's past cooking history data to provide more personalized advice. Specifically, video, audio, sensor data, and score evaluations related to the user's previous cooking are recorded in a cloud-based database, and by comparing this with the current cooking behavior, the system analyzes the user's progress and areas for improvement. For example, if the evaluation from the previous cooking was "uneven cutting," and the cutting has improved this time, positive feedback such as "You've cut more evenly than last time, you're improving!" will be displayed. On the other hand, if the same mistake as last time is repeated, specific improvement suggestions will be made, such as "The heat was too high last time. Let's try adjusting it to medium heat this time."

[0045] Such advice not only continuously supports the accumulation and improvement of users' cooking skills, but also enhances the interactivity between the AI ​​and the user, contributing to long-term motivation maintenance. By combining it with eye tracking and motion analysis, it is possible to provide real-time warnings to prevent past mistakes, such as "You burned it here last time. Let's be careful with the heat this time," providing contextual guidance. Furthermore, by linking with the character function, it is possible to provide emotional reactions according to past progress, and comments such as "This is the best result you've had this week!" are displayed, giving the AI ​​a presence that goes beyond mere assistance to become a partner for the user. As a result, a continuous and gradual learning environment is created, rather than just one-off assistance, making it possible to visualize cooking experience and provide a sense of growth (see Figure 3).

[0046] This cooking support system features a function that allows users to select a virtual character that suits them, elevating the cooking experience from a mere task to a nurturing learning experience involving dialogue and emotion. Users can select their preferred character when the system starts up or from the settings screen, and that character will be constantly displayed on the screen or within their field of view during cooking, providing feedback while changing their voice and facial expressions according to the progress. The character watches over the user's progress and guides them carefully, much like a personal cooking coach.

[0047] The selectable characters include a variety of types with diverse appearances and personalities, such as a chef-like figure, a cute animal, a fantasy character, and an intelligent robot. Each character has their own unique way of speaking, facial expressions, and reactions, and even when giving the same instructions, their tone and vocabulary will differ, allowing users to choose the support style that suits them best. For example, an energetic character might brightly praise you with "That's great! Keep it up!", while a calm character might gently encourage you with "You're making good progress. Let's move on to the next step."

[0048] This character selection goes beyond mere preference; it's optimized in conjunction with the user's attributes and usage history. For example, the system can suggest characters based on the user's age, cooking experience, and past usage patterns. Children will be shown gentle and approachable characters, while adult beginners will be shown characters that provide detailed explanations, naturally providing a support style suited to the user. Furthermore, dynamic suggestions such as "Would you like to try a more energetic character?" are also made depending on usage. (See Figure 4)

[0049] Furthermore, this cooking support system is equipped with a voice emotion synchronization function that analyzes the tone of the user's voice and the content of their speech in real time, and changes the character's response accordingly. For example, if the user expresses discouragement by saying, "It's not going well," the system detects signs of emotion from the tone of voice and the length of the sentence ends, and the character responds gently, "It's okay. Let's take our time." On the other hand, if the user's voice is cheerful, the character shows empathy by saying, "You sound happy! You seem to be in good spirits today!" As a result, the interaction with the character has evolved from a one-way street to a natural communication that is attentive to the user's feelings.

[0050] The character's expressions are linked to scoring based on AI-generated cooking evaluations and change dynamically according to the user's actions. When a high score is achieved, the character praises the user with a big smile, saying, "That's fantastic!", and even when mistakes are made, it gives positive feedback such as, "You were so close to perfect!" In addition, to enhance the sense of accomplishment, the character may display animations such as clapping or dancing, providing the user with a fulfilling learning experience using all five senses: sight, sound, and emotion.

[0051] Furthermore, the character can offer personalized feedback and suggestions for improvement based on past cooking history. For example, comments such as, "Your knife skills are more stable than last time," or "You managed to avoid burning it this time," are displayed, allowing users to feel a sense of continuous improvement. This kind of personalized feedback gives users a sense of being watched and a feeling of growth, naturally motivating them to cook again. In this way, the character functions not merely as an animation, but as a continuous supporter that understands the user's learning history.

[0052] Furthermore, the characters interact with other system functions, engaging with external features such as network sharing, recipe suggestions, and sale information. For example, they might encourage users to post by saying, "Take a picture of this dish and show it to everyone!" or support ingredient selection by suggesting, "Tomatoes are cheap at the nearby supermarket right now, so how about this recipe?" These character statements are customized based on the user's preferences, history, and surrounding environment, functioning as a guide that accompanies the entire daily cooking experience. Future plans include customization of character voices and costumes, as well as the introduction of user-created characters, further enhancing personalization and fostering a deeper sense of attachment.

[0053] Furthermore, the characters are closely integrated with other functions within the system, naturally guiding user behavior through cooking suggestions, sale information, and sharing features. For example, they might suggest, "Today's recommendation is a variation of a stir-fry dish you've been making recently," or "Chicken thighs are on sale at your local supermarket. Shall I suggest a recipe?" They also encourage users to share their cooking results with others when posting to social media or communities, motivating them to share the joy of cooking with the outside world. In this way, the characters dynamically interact with the user's emotions, actions, history, and external environment, acting as more than just guides—they are co-stars in the cooking experience, enhancing the overall appeal of the system.

[0054] In addition, this cooking support system includes a network sharing function that allows users to share their cooking results with others, encouraging them to improve their cooking skills and continue using the system. After completing a dish, users can take photos or videos of their cooking and upload them within the app or to a linked web platform. Meta information such as the name of the recipe, cooking time, ingredients used, and rating score can be added to the post, allowing other users to view, compare, and comment on it. The system also features an AI function that automatically adds comments such as "Beautifully plated" or "Perfectly cooked," further supporting users in posting. This allows users to visibly record their cooking achievements and naturally gain opportunities to connect with others.

[0055] Uploaded creations receive various reactions from other users, such as "likes," 5-star ratings, and comments, allowing posters to receive objective feedback on their cooking. This not only allows users to feel their own improvement but also provides social feedback in the form of recognition and encouragement from others, significantly boosting their motivation. For beginner users in particular, having their creations recognized by others increases their self-esteem and becomes a source of motivation to continue cooking. In addition, there is a feature that highlights highly-rated images as "featured posts" within the system, providing an incentive for users to share their achievements.

[0056] This cooking support system also incorporates community features to stimulate interaction among users, serving as a platform for sharing knowledge and experience. Users can freely post and answer questions, tips, variations, and instructions on using specific cooking utensils in a forum-style bulletin board. An automatic theme-based thread classification function allows for quick access to desired information. A real-time chat function is also available, enabling immediate exchanges such as discussions on specific recipes or beginners seeking advice from more experienced users. An AI moderator is also implemented to automatically detect and suppress inappropriate comments, ensuring a safe and secure environment for interaction.

[0057] Through these network sharing and community features, users can gain a sense of social connection, knowing that their cooking activities are shared with others. This creates a cycle of mutual recognition, empathy, and advice that cannot be achieved through mere digital recipe support, imbuing daily cooking with a feeling of "working hard together with someone." In particular, in today's world where feelings of isolation are common due to the increase in teleworking and single-person households, this kind of "connection with society through cooking" greatly contributes to mental well-being. Furthermore, the network can be used in a wide range of ways, such as exchanging regional food cultures, comparing results among family members, and planning challenges with friends, offering flexibility to accommodate each user's lifestyle. In the future, we plan to introduce more advanced social gamification, such as ranking displays based on posted data and AI-powered compatibility matching recipe challenges.

[0058] This cooking support system features a function that automatically suggests the most suitable recipe based on the user's cooking skill level, making it easy for beginners to advanced users to tackle cooking challenges appropriate to their stage of development. A key feature is that the suggested recipes are not simply ranked by popularity or newness, but are dynamically selected based on each user's technical background and experience. This ensures that users can challenge themselves with dishes of just the right difficulty level.

[0059] The AI ​​determines a user's skill level by analyzing their past cooking records (success rate, accuracy of steps, variety of ingredients used, cooking time, AI score, etc.). The AI ​​also identifies strengths and weaknesses in specific areas, detecting tendencies such as "consistently good at simmering but inconsistent heat control with grilled dishes," and then suggests the next appropriate recipe. The evaluation criteria consider not only the accuracy of the cooking itself but also the user's learning pace and operating tendencies, allowing for personalized suggestions rather than uniform difficulty adjustments.

[0060] For beginners, recipes with clear operation steps and a structure that minimizes the risk of failure are prioritized. For example, basic dishes that display the steps in order, such as "cut, sauté, and plate," are suggested, and users can check their progress step by step on the screen. On the other hand, for users who have demonstrated a certain level of skill, recipes that require timing management or dishes that require the harmony of multiple ingredients are presented in stages, encouraging them to gradually acquire practical skills.

[0061] Furthermore, the suggestion algorithm takes into account the user's cooking genre preferences and past selection trends, enabling suggestions such as, "You've been cooking a lot of spicy dishes lately, so why not try a spicy fish dish?" This goes beyond simple difficulty matching and provides suggestions that are tailored to the user's interests. In addition, integrating with allergy information, ingredient restrictions, and refrigerator inventory information further enhances personalized suggestions that are based on realistic conditions.

[0062] In this cooking support program, users can input simple feedback on suggested recipes, such as "too difficult," "too easy," or "just right," and this feedback information is stored in the system. This information is used as important training data to learn the user's cooking skills, preferences, and reaction tendencies, and through machine learning processing by artificial intelligence, the accuracy of recipe suggestions in the future is continuously improved. For example, if a user repeatedly rates a recipe as "too easy," the AI ​​learns to prioritize suggesting slightly more difficult recipes to that user next time. Conversely, if feedback of "too difficult" is frequently received, the algorithm is adjusted to recommend recipes with simpler steps next time. Furthermore, this feedback is linked not only to difficulty but also to multiple factors such as cooking time, type of ingredients, taste tendencies, complexity of cooking method, and seasonality, enabling a higher level of personalized suggestions for each user. In addition, each user's suggestion history and feedback results are recorded chronologically, allowing users to view an overview of the changes in recipes they have attempted in the past and their reaction trends. Thus, this program is characterized not only by its simple recipe presentation, but also by its interactive and evolving cooking support function that continuously optimizes the accuracy of its suggestions through dialogue with the user.

[0063] This skill-based recipe suggestion feature allows users to naturally improve their skills through daily cooking while taking on challenges that are neither too difficult nor too easy. The exquisite balance of suggestions—not too difficult to cause frustration, and not too easy to become boring—is a core element that supports the continuity and enjoyment of cooking. Furthermore, future plans include expanding the recommendation function to allow users to compare their skill levels with other users and to develop their areas of expertise, providing a medium- to long-term foundation that will support the evolution of the cooking experience.

[0064] This cooking support system features a "food purchase support function" that allows users to clearly understand the necessary ingredients based on the recipe they select, and it streamlines cooking preparation through integration with online shopping. When a user selects a recipe, the system automatically generates a list of the necessary ingredients, displaying quantities, shapes appropriate for the cooking method, and alternative ingredients. This prevents users from forgetting ingredients or making unnecessary purchases, allowing them to start cooking with the minimum necessary shopping.

[0065] This ingredient list is linked to partner online shopping sites, allowing users to add ingredients to their shopping cart with a single click and complete ordering, payment, and delivery arrangements. Furthermore, a user interface is available that allows users to select different price ranges and packaging information (organic products, origin, brand, etc.) for the same ingredient, enabling flexible shopping according to the user's preferences and budget. In the event of ingredient delays or stock shortages, alternative options are offered, ensuring a stress-free shopping experience.

[0066] This online connectivity feature also supports data linkage with the user's IoT refrigerator, automatically matching the type, quantity, and expiration date information of the food currently stored in the refrigerator. For example, it can determine if "there are enough eggs and milk left in the refrigerator, the carrots are nearing their expiration date, and there are few onions," and automatically add only the missing items to the shopping list while simultaneously suggesting recipes that prioritize using ingredients that are nearing their expiration date. This helps to both efficiently utilize food in the household and reduce waste.

[0067] Furthermore, this cooking support system is equipped with a function to collect and provide special sale information from nearby supermarkets and stores based on the user's current location, supporting economical purchasing not only online but also at physical stores. By linking with location information, it obtains the latest flyer information and same-day sales from stores near the user in real time, and displays notifications such as "Pork is half price at XX supermarket" on devices such as smartphones and AR glasses, optimizing the timing and actions of shopping.

[0068] The notifications for special offers go beyond simply displaying prices; they are integrated with the system's recipe suggestion function. For example, if a notification says, "Cabbage is on sale," recipes using cabbage as the main ingredient (such as stuffed cabbage rolls or stir-fried cabbage and pork) are immediately suggested, and the user is encouraged to try a recipe that takes advantage of this sale, either through voice or on-screen display. This integration seamlessly connects the shopping and cooking processes, improving overall planning in daily life.

[0069] Furthermore, the suggested recipes are optimized based on the user's cooking history, preferences, and skill level, ensuring they are enjoyable and easy to follow. For example, a user who has frequently cooked spice dishes in the past might be suggested "Spicy Cabbage Soup," while a user who prefers Japanese food might be suggested "Cabbage Stir-fry with Miso." In this way, the combination of special offer information, personal history, and cooking tendencies creates a personalized and enriching cooking experience that goes beyond mere saving money.

[0070] This cooking support system not only allows users to choose their own shopping routes, but also suggests "shopping routes" that include sale items. For example, if cabbage and chicken are on sale at different nearby supermarkets, the system will display an "efficient route" on the map, taking into account the distance, congestion, and time spent at each location. It can also predict "sales you should buy now" and "sales that are likely to be further discounted if you wait until the weekend," allowing users to control their purchasing timing.

[0071] Furthermore, this cooking support system includes a function that automatically sets expiration dates based on the type of ingredient and purchase date, making it easier for users to manage the ingredients they purchase. For example, the general shelf life of each ingredient is pre-registered in the database, such as 3 days when refrigerated and 2 weeks when frozen for chicken, and 5 days when refrigerated for leafy vegetables, and the expiration date is automatically calculated according to the purchase date and storage method.

[0072] This expiration date setting is automatically retrieved from product information during online purchase, recorded through barcode scanning using IoT refrigerators or smart scanners, or through manual input assistance. Especially when using an IoT refrigerator, internal sensors can detect the insertion and removal of food items, allowing for readjustment of the expiration date based on actual usage. These automated management features allow users to always have accurate inventory status without the hassle of recording or checking expiration dates.

[0073] The system prioritizes detecting ingredients nearing their expiration date and notifies the user. Alerts such as "There is chicken that needs to be used by tomorrow" or "The cabbage expires today" are conveyed not only through on-screen displays but also through voice and vibration, preventing food waste that might otherwise be forgotten in the refrigerator. Furthermore, these ingredients are linked to the recipe suggestion function, which immediately displays recipes that help use them up before their expiration date.

[0074] For example, if you have chicken that's due to expire the next day, the app will suggest recipes that can be cooked immediately, such as "Steamed Chicken and Vegetables" or "Chicken and Bean Tomato Stew," minimizing food waste. Furthermore, if you're missing any other ingredients needed for that recipe, the app automatically displays online purchase links, allowing you to go shopping right away. Through this process, users can effortlessly plan their cooking and consume their ingredients.

[0075] In addition, this cooking support system provides a function that retrieves special sale information from nearby supermarkets and other stores based on the user's location information, and uses this information, along with expiration date information, to help select recipes. The special sale information is obtained in real time from partner stores and is displayed including conditions such as the type of ingredients, price, quantity restrictions, and sales time, so users can always know "what's on sale right now."

[0076] This allows the system to automatically suggest recipes that combine ingredients the user already owns with ingredients currently on sale. For example, using chicken nearing its expiration date in the refrigerator and tomatoes on sale at a nearby supermarket might suggest recipes such as "Chicken Tomato Stew" or "Chicken and Tomato Curry." This enables realistic and economical meal planning suggestions that integrate inventory management with external purchasing information.

[0077] When selecting recipes, the system takes into account the user's preferences, past cooking history, and allergy information. For example, users who like spicy food will be offered recipes that include spicy variations, while users who primarily cook Japanese food will be offered recipes based on miso and soy sauce. The system's AI learns the commonalities between recipes that the user has highly rated in the past and prioritizes presenting menus that the user is interested in, thereby supporting a cooking experience that can be continued without becoming boring.

[0078] This recipe suggestion function, linked to expiration dates and sale information, enables users to make ideal use of ingredients by buying them cheaply and using them without wasting any. Furthermore, recipes that utilize ingredients purchased in excess due to sales are simultaneously suggested, preventing "waste from bulk buying." The system optimizes the user's cooking experience by considering savings, time efficiency, and cooking satisfaction.

[0079] This suite of features goes beyond simple expiration date management and sales notifications, linking a series of daily activities—purchasing, storing, consuming, and cooking—together, resulting in multifaceted benefits such as reducing food waste at home, streamlining shopping, improving family communication, and maintaining motivation for cooking. Future plans include expanding the functionality to include features such as AI-based automatic correction of expiration date accuracy and personalized purchase quantity advice based on individual food consumption trends. It is expected to grow into a next-generation food support platform that contributes to a sustainable and comfortable cooking lifestyle.

[0080] In addition to its basic use in supporting home cooking, the cooking advice system of the present invention possesses versatility that allows for applications in various industrial fields. For example, in cooking classes and culinary schools, it can provide real-time guidance on individual cooking processes as a substitute or assistant to instructors, significantly contributing to the standardization and efficiency of instruction in educational settings. Furthermore, in the food and beverage industry, it can be used to provide cooking instruction to new staff and support standardized work procedures for maintaining quality, and is expected to shorten training periods and standardize work processes. Moreover, in food education classes for children and care facilities for the elderly, it can broaden access to cooking experiences regardless of age or physical ability through multimodal support utilizing visual, auditory, and tactile senses.

[0081] Furthermore, this cooking support system is designed with compatibility with various existing digital devices such as smartphones, wearable glasses, AR devices, and IoT refrigerators in mind, allowing it to flexibly adapt to the user's existing environment. It can be easily deployed not only as a cooking support tool in the home, but also for single-person households without kitchens, in restaurant kitchens, and for cooking demonstrations in commercial facilities. In addition, by linking with online shopping sites and POS systems, it can be deployed for marketing purposes such as purchasing ingredients, inventory management, and even recipe recommendations using purchase history, demonstrating high compatibility with the distribution and retail industries. As a result, this cooking support system has the potential to expand beyond being merely a "cooking support device" into a data-driven lifestyle infrastructure.

[0082] In this way, the cooking advice system according to the present invention visualizes and quantifies the household activity of "cooking" through the collaboration of AI and sensor devices, evolving into a "lifestyle-integrated platform" that connects it with diverse areas such as food management, purchasing, education, and entertainment. Users can centrally manage not only what to eat, but also what to buy, in what order to use it, and how to cook, improving economic and time efficiency while adding enjoyment and peace of mind to their daily cooking experience. In the future, it has high potential for expansion toward solving social issues, such as collaboration with local distribution infrastructure and local government health promotion measures, and application to personalized nutrition management and allergy management, and is expected to make a significant contribution to "improving the quality of daily life" starting from cooking support.

Claims

1. A shooting function that acquires image information, A recording function that allows you to acquire audio information, It has a speaker function that outputs sound, A display function that allows images to be displayed, A memory function that stores image information, cooking procedure information, and cooking action pattern information linked to cooking recipe information, as well as user information and user response information. An artificial intelligence function that uses input information to perform calculations based on machine learning, A cooking support program that enables a mobile terminal to perform the following actions: use the artificial intelligence function to analyze image information acquired by the shooting function and audio information acquired by the recording function while cross-checking them, compare them with image information, cooking procedure information, and cooking action patterns linked to cooking recipe information stored by the memory function to determine a cooking evaluation score, generate feedback information from a virtual character based on the cooking evaluation score, user information, and user reaction information, notify the user of the feedback information using the speaker function with the voice of the virtual character, or visually display the feedback information on the screen along with an image of the virtual character using the display function, and record the cooking evaluation score, feedback information from the virtual character, and user reaction information to the feedback information from the virtual character using the memory function.

2. A cooking support program according to claim 1, wherein a mobile terminal is enabled to analyze heart rate information acquired from a wearable device worn by the user using an artificial intelligence function, estimate the user's state of tension based on changes in heart rate, and use the result of the estimation to generate feedback information by a virtual character.

3. A cooking support program according to claim 1, characterized in that when a dangerous cooking action is detected by the artificial intelligence, a signal is sent from a mobile terminal to a wearable device worn by the user, which will emit a vibration warning.

4. A cooking support program according to claim 1, characterized in that the type of virtual character, way of speaking, or facial expression changes according to the cooking evaluation score.

5. A cooking support program according to claim 1, wherein an artificial intelligence function analyzes the tone of voice, speaking speed, and length of the end of sentences contained in the voice information acquired by the recording function, estimates the user's emotional state, and enables a mobile terminal to use the result of the estimation to generate feedback information by a virtual character.

Images