System

Abstract

A system is provided.SOLUTION: This advice providing system is provided with a means for allowing a user to photograph the process of cooking by using a device, a means for streaming the cooking process to a server in real time, a means for analyzing video data received by the server, and for extracting the information of the size, mixed state and baked color of ingredients, and a means for generating advice based on the extracted information. The system includes a means for notifying the user of the advice, a means for the user to verbally ask a question, a means for converting the verbal question into text and transmitting the text to a server, a means for the server to generate an answer to the question and notify the user, and a means for storing data of the cooking process.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Cooking well requires experience and knowledge, and beginners often feel anxious about certain dishes. Therefore, receiving appropriate advice and feedback in real time can help people cook with greater confidence. However, current technology has limited means for receiving real-time feedback and advice, and there are no systems that provide specific guidance based on visual information. Therefore, there is a need to combine visual information and voice communication to provide appropriate advice and feedback in real time during the cooking process and improve cooking skills. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following means: A user uses a device (e.g., a smartphone) to film the cooking process and streams the video to a server in real time. The server analyzes the received video data and extracts information about the size of ingredients, the degree of mixing, and browning. Based on the analysis results, the server generates appropriate advice and notifies the user in voice and text format. Furthermore, when the user verbally asks a question, the voice is converted into text and sent to the server, and the server generates an answer to the question and notifies the user in voice. The system also includes a means for storing data on the cooking process. In this way, users can receive feedback and advice in real time, allowing them to cook with peace of mind.

[0006] "User" refers to any person or entity that uses the system of the present invention.

[0007] A "device" is a device that a user uses to capture the cooking process, including, for example, a smartphone or tablet.

[0008] The "cooking process" refers to the series of steps and operations involved in preparing a dish.

[0009] "Filming" refers to using a device to record the cooking process as video.

[0010] "Streaming" refers to the continuous transmission of captured video data in real time.

[0011] "Server" refers to a central device that receives and analyzes video data sent by users.

[0012] "Video data" refers to video information that records the cooking process.

[0013] "Analysis" refers to extracting information such as the size of ingredients, how well they are mixed, and how browned they are from the video data.

[0014] "Ingredients" refers to the ingredients used in a dish.

[0015] "Size" refers to the dimensions of the ingredients.

[0016] "Mixing" refers to the degree to which multiple ingredients are mixed together.

[0017] "Browning" refers to the change in color that indicates the doneness of ingredients during cooking.

[0018] "Extraction" refers to extracting necessary information from video data.

[0019] "Advice" refers to the guidance and advice provided to the User regarding the cooking process.

[0020] "Notification" means conveying information to the user.

[0021] "Verbal question" refers to a question that the user asks the system verbally.

[0022] "Convert to text" refers to converting audio data into text information.

[0023] "Answer" refers to the information provided by the server in response to a user's question.

[0024] "Storing" refers to recording and retaining data about the cooking process.

[0025] "System" refers to the collection of all components and functions of the present invention. [Brief explanation of the drawings]

[0026] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0027] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0028] First, the terms used in the following description will be explained.

[0029] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0030] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0031] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0032] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0033] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0034] [First embodiment]

[0035] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0036] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0037] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0038] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0039] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0040] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0041] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0042] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0043] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0044] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0045] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0046] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0047] The system of the present invention provides real-time support for cooking by combining visual information and voice communication. The system allows the user to use a device to film the cooking process and transmit the video in real time to a server, which can then provide appropriate advice on cooking methods and progress.

[0048] The detailed operation of the system is described below.

[0049] First, the user launches the app installed on their smartphone, tablet, or other device. After entering their login information and successful authentication, the device displays a recipe selection screen. The user selects the recipe they want to make and confirms the ingredients and steps required.

[0050] When the user starts cooking, the device's camera captures video. This video is streamed to the server in real time. The server analyzes the received video data and extracts information such as the size of the ingredients, how well they are mixed, and how browned they are. This allows the server to determine whether the ingredients are cut to the right size, mixed thoroughly, and cooked to the right degree.

[0051] The server generates advice based on the analysis results and notifies the user in voice and text format. For example, when frying onions, it may provide specific instructions such as "You need to fry them a little longer." This advice helps the user understand what steps to take next.

[0052] Furthermore, if a user has a question while cooking, they can verbally ask the device. The device uses voice recognition technology to convert the question into text and send it to the server. The server analyzes the question, generates an appropriate answer, and notifies the user by voice. For example, in response to a question such as "What should I do next?", the device will respond with "Next, add the chicken."

[0053] Once cooking is complete, the user ends the session within the app and the server stores the cooking process data for future reference and analysis.

[0054] As a concrete example, consider a user making teriyaki chicken. The user launches the app, logs in, and selects the teriyaki chicken recipe. They prepare the ingredients and use their smartphone camera to film the cooking process. The server receives the video and provides specific advice in real time, such as "You need to fry the onions a little longer." If the user also asks, "Is the chicken browned to the desired degree?" the server analyzes the video and returns an appropriate answer. Finally, when cooking is complete, the user ends the session, and the cooking data is saved to the server.

[0055] As described above, the system of the present invention combines real-time video analysis and voice communication to provide detailed and immediate support to users, allowing even beginners and users who are unsure about certain dishes to cook with confidence.

[0056] The processing flow will be explained below.

[0057] Step 1:

[0058] A user launches an app installed on a device such as a smartphone or tablet, and the device displays a login screen prompting the user to enter their username and password.

[0059] Step 2:

[0060] The user enters their username and password and presses the login button. The device temporarily stores the entered information and prepares to send it to the server. The device then sends an authentication request to the server.

[0061] Step 3:

[0062] The server compares the received user information with the records in its database and generates an authentication result. If the authentication is successful, the server returns an authentication success message to the terminal.

[0063] Step 4:

[0064] The device receives the authentication result and displays success or failure to the user. If authentication is successful, the recipe selection screen is displayed.

[0065] Step 5:

[0066] The user selects the desired recipe from the list and enters the selected recipe ID into the terminal, which temporarily stores the selected recipe ID and prepares to send it to the server.

[0067] Step 6:

[0068] The terminal sends a request for the selected recipe to the server. The server receives the request and retrieves detailed data for the selected recipe from the database. The server then sends the retrieved recipe data to the terminal.

[0069] Step 7:

[0070] The device displays the received recipe data and tells the user the ingredients and steps required, allowing the user to prepare the ingredients and begin cooking.

[0071] Step 8:

[0072] The user initiates the cooking process and captures the video using the device's camera. The device starts recording in real time and streams the video data to the server.

[0073] Step 9:

[0074] The server receives the video data and temporarily stores it for analysis. The server then uses image analysis algorithms to extract information such as the size of the ingredients, how well they are mixed, and how browned they are.

[0075] Step 10:

[0076] The server generates appropriate advice based on the analysis results, and the advice is sent to the device in voice and text format.

[0077] Step 11:

[0078] The device will display and voice the received advice to the user, who will then proceed with the next cooking step based on the advice.

[0079] Step 12:

[0080] If a user has a question while cooking, they can ask it verbally to the device, which will then use voice recognition technology to convert the question into text and send it to the server.

[0081] Step 13:

[0082] The server analyzes the textual content of the question and generates an appropriate answer, which is then sent to the device in voice and text format.

[0083] Step 14:

[0084] The device displays the received answers to the user and notifies them by voice, allowing the user to proceed with cooking while having their questions resolved.

[0085] Step 15:

[0086] When the user finishes cooking, they press the end session button in the app, and the device sends a session end request to the server.

[0087] Step 16:

[0088] The server receives the session data, saves it in the database, and notifies the terminal that the save is complete.

[0089] Step 17:

[0090] The device will display and sound a message to the user that cooking is complete, and the entire cooking process will be completed.

[0091] Example 1

[0092] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0093] The lack of a system that allows users to receive appropriate advice and guidance in real time while cooking poses a challenge, making it difficult for beginners and users who are unsure about certain dishes to cook with confidence. Furthermore, conventional cooking support systems are unable to respond immediately to users' questions, making it difficult to provide appropriate support tailored to the user's actual cooking progress.

[0094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0095] In this invention, the server includes means for a user to use a device to film the cooking process, means for streaming the cooking process to the server in real time, means for the server to analyze the video data received and extract information on the size and mixing of ingredients and browning, means for generating advice based on the extracted information and notifying the user of the advice in audio and text format, means for the user to verbally ask a question, means for converting the verbal question into text and sending it to the server, means for the server to generate an answer to the question and notify the user in audio and text format, and means for saving the cooking process data. This allows users to receive appropriate cooking guidance and advice in real time, allowing even beginners and users who are unsure about certain dishes to cook with confidence.

[0096] A "user" is a person who uses the system to assist with the cooking process.

[0097] "Device" refers to a mobile device such as a smartphone or tablet used by a user.

[0098] The "cooking process" refers to the series of steps involved in preparing ingredients, cutting, mixing, heating, etc., to complete a dish.

[0099] "Streaming" is a technology that transmits captured video data to a server in real time.

[0100] The "server" is a central processing unit that receives and analyzes the video data sent from the user's device and generates corresponding advice and answers.

[0101] "Video data" refers to images and videos of the cooking process captured by the device's camera.

[0102] "Analysis" refers to the process of evaluating the size of ingredients, how well mixed they are, and how browned they are based on the received video data.

[0103] "Ingredients" refer to the individual ingredients that make up a dish.

[0104] "Advice" refers to cooking instructions and advice that are generated based on the analysis results and provided to the user.

[0105] "Audio format" refers to a format in which advice or answers are notified to the user as audio.

[0106] "Text format" refers to a format in which advice or answers are notified to the user in written form.

[0107] "Questions" are questions or problems that users verbally input while cooking.

[0108] "Voice recognition technology" refers to technology that converts a user's verbal questions into text.

[0109] An "answer" is a response generated by a server in response to a question from a user.

[0110] "Data storage" refers to the process of storing data about the cooking process on a server for future reference.

[0111] "Real-time" refers to a time concept in which analysis and notification are carried out immediately as the cooking process progresses.

[0112] "Computer vision technology" is a general term for image recognition technology used to analyze video data.

[0113] The system of the present invention provides real-time support for cooking by combining visual information and voice communication. A specific embodiment of the system will now be described.

[0114] First, the user launches a dedicated app installed on a mobile device such as a smartphone or tablet. After the user enters their login information and is successfully authenticated, the device displays a recipe selection screen. The user selects the recipe they want to make and checks the ingredients and steps. The software used by the device at this stage generally includes the operating system (e.g., iOS, Android) and application software for the smartphone or tablet.

[0115] Next, when the user starts cooking, the device's camera is used to capture video. This video is then streamed to the server in real time using technologies such as the Real-Time Messaging Protocol (RTMP) and HTTP Live Streaming (HLS). This allows the video data to be sent to the server in real time.

[0116] The server uses computer vision technology (e.g., OpenCV or TensorFlow) to analyze the received video data. The server extracts information such as the size of the ingredients, the degree of mixing, and the browning of the ingredients, and performs analysis based on this. Based on the analysis results, the server generates appropriate advice and sends it to the device. The advice is provided in audio (e.g., Google Cloud Text-to-Speech API) and text format. The device notifies the user of the advice it received.

[0117] If a user has a question while cooking, they can verbally ask the device. The device will convert the question into text using voice recognition technology (e.g., Google Cloud Speech-to-Text API) and send it to the server. The server will analyze the question, generate an appropriate answer, and notify the user in voice and text format.

[0118] Once cooking is complete, the user can end the session within the app. The server then stores the cooking process data for future reference and analysis. This data includes analysis results, user questions and answers, and any advice provided.

[0119] As a concrete example, consider a user making teriyaki chicken. The user launches the app, logs in, and selects the teriyaki chicken recipe. They prepare the ingredients and use their smartphone camera to film the cooking process. The server receives the video and provides specific advice in real time, such as "You need to fry the onions a little longer." When the user asks, "Is the chicken browned to the desired degree?" the server analyzes the video and returns an appropriate answer. Finally, when cooking is complete, the user ends the session, and the cooking data is saved to the server.

[0120] Examples of prompts include:

[0121] "I'm making chicken teriyaki. I'm currently frying onions. Can you tell me the proper doneness?"

[0122] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0123] Step 1: Launching the program and authenticating the user

[0124] Users launch a dedicated app on their smartphone or tablet and enter their login information (username and password).

[0125] The terminal sends the entered login information to the server (input: user name, password).

[0126] The server compares the received login information with the authentication database and performs authentication (data processing: comparison with authentication database, output: authentication result).

[0127] If the authentication is successful, the server starts a session and displays the recipe selection screen on the terminal (output: session ID, recipe selection screen).

[0128] Step 2: Select a recipe and check ingredients

[0129] The user selects the recipe they want to make on the device screen (input: select recipe).

[0130] The terminal transmits the selected recipe information to the server (input: recipe information).

[0131] The server sends the ingredients and cooking instructions required for the selected recipe to the terminal (data processing: recipe data extraction, output: ingredient list, procedure list).

[0132] The terminal displays it on the screen, and the user checks the ingredients and proceeds with preparation (output: ingredient list, procedure list).

[0133] Step 3: Record and send footage

[0134] The user starts cooking and takes pictures using the device's camera.

[0135] The terminal streams the captured video data to the server in real time (input: video data).

[0136] The terminal transmits video data using the Real-Time Messaging Protocol (RTMP) or HTTP Live Streaming (HLS) (output: real-time video data).

[0137] Step 4: Video analysis on the server

[0138] The server uses computer vision techniques (e.g., OpenCV or TensorFlow) to analyze the received video data (input: video data).

[0139] The server extracts information such as the size of the ingredients, how well mixed they are, and how browned they are (data processing: video analysis, output: ingredient information).

[0140] The cooking progress is evaluated based on the analysis results (data calculation: cooking progress evaluation, output: progress evaluation results).

[0141] Step 5: Server generates and notifies advice

[0142] The server generates specific advice based on the analysis results (data processing: advice generation, output: advice).

[0143] The server sends the generated advice to the terminal in voice (e.g., Text-to-Speech API) and text format (input: advice, output: voice data, text data).

[0144] The device notifies the user of the received advice (specific actions: playing audio, displaying text).

[0145] Step 6: Respond to user questions

[0146] If the user has a question while cooking, they can ask verbally into the terminal (input: voice question).

[0147] The device uses voice recognition technology (e.g., Speech-to-Text API) to convert the question into text and send it to the server (data processing: voice recognition, output: text question).

[0148] The server analyzes the question and generates an appropriate answer (data processing: question analysis, output: answer).

[0149] The server sends the generated answer to the terminal in voice and text format (input: answer, output: voice data, text data).

[0150] The terminal notifies the user of the answer by voice and text (specific operations: playing voice, displaying text).

[0151] Step 7: Save the data after cooking is complete

[0152] When the user is done cooking, they can end the session within the app (input: end session).

[0153] The server stores the data of the cooking process (data processing: data storage, output: stored data).

[0154] The saved data is used for future reference and analysis (specific operation: saving to a database).

[0155] (Application example 1)

[0156] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0157] With current food delivery services, it is difficult to control the quality of food during delivery, making it difficult for delivery personnel to efficiently deliver food while checking its condition. Furthermore, there is a lack of means to receive real-time feedback on the state of the food, making it a challenge to improve delivery quality.

[0158] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0159] In this invention, the server includes: means for a user and a delivery person to use a device to film the cooking process and the state of the food; means for streaming the cooking process and the state of the food to the server in real time; means for the server to analyze the received video data and evaluate information on the size and mixture of ingredients and browning and the quality of the food; means for generating advice based on the extracted information and evaluation results and notifying the user and the delivery person of the advice; means for the user and the delivery person to ask verbal questions; means for converting the verbal questions into text and sending it to the server; means for the server to generate answers to the questions and notify the user and the delivery person; and means for storing the cooking process data and food state data. This enables real-time quality control of the cooking process and food during delivery, enabling the delivery person to efficiently deliver high-quality food.

[0160] "User" means any person who uses the system to receive real-time assistance and advice during the cooking process.

[0161] "Delivery person" means a person who performs quality control during the delivery of food in a food delivery service.

[0162] The "device" is a hard device that has a camera function and transmits the actions of the user or delivery person to the server in real time.

[0163] A "server" is a computing device that analyzes received video data and generates advice.

[0164] "Video data" refers to visual information captured by the user or delivery person showing the cooking process and the condition of the food.

[0165] "Analysis" is the process of evaluating the size of ingredients, the degree of mixing, the browning, and the quality of the food based on the received video data.

[0166] "Advice" refers to instructions and advice regarding cooking and quality control provided based on the analysis results.

[0167] "Streaming" is a technology for transmitting video data to a server in real time.

[0168] "Questions" are questions or clarifications about the cooking process or the condition of the food that users or delivery personnel ask the system.

[0169] "Text conversion" is the process of converting spoken questions into written information.

[0170] "Notification" is the act of transmitting advice or answers generated by the server to the user or delivery person.

[0171] "Data archiving" is the process of recording information about analysis results and recommendations for future reference.

[0172] "Quality evaluation" is the process of determining the quality of food based on its condition.

[0173] "Audio format" refers to a means of providing advice or answers to users or delivery personnel via voice.

[0174] An "image analysis algorithm" is a mathematical technique used to analyze video data.

[0175] "Food condition" refers to information such as the temperature, appearance, and packaging of the food during delivery.

[0176] The system of the present invention allows users and delivery staff to use devices to film the cooking process and the condition of the food, and sends the footage to a server in real time, thereby providing appropriate advice on cooking methods and progress and managing food quality.

[0177] System Configuration

[0178] 1. Hardware

[0179] Camera-equipped devices such as smartphones, tablets, and smart glasses

[0180] Computing Server

[0181] 2. Software

[0182] Image Analysis Algorithm

[0183] Speech Recognition and Synthesis Software

[0184] Streaming Server

[0185] Data processing and calculation

[0186] Video data collection and streaming

[0187] Users and delivery staff use devices (smartphones, tablets, smart glasses) to record the cooking process and the state of the food, and this video data is sent to a streaming server in real time.

[0188] Video data analysis

[0189] The streaming server analyzes the received video data, using image analysis algorithms to assess the size, consistency, browning, and food quality (temperature, appearance, packaging, etc.) of ingredients.

[0190] Advice generation and notification

[0191] Based on the analysis results, the server generates advice on the cooking progress and food quality, which is communicated to the user and delivery person in voice and text format.

[0192] Questions and Answers

[0193] When a user or delivery person verbally asks a question, the device converts the voice into text and sends it to the server, which analyzes the question, generates an appropriate answer, and notifies the user in voice and text format.

[0194] Data storage

[0195] Analysis results, advice, and question and answer data are stored by the server for future reference and service improvement.

[0196] Specific examples

[0197] Ramen delivery

[0198] A delivery person wears smart glasses and takes a video to check the condition of the ramen during delivery. The streaming server analyzes the video and provides advice such as "The soup has not spilled" and "The ramen ingredients have not fallen apart." If the delivery person asks, "Please check if the ramen ingredients have fallen apart," the voice is converted into text, and the server analyzes the question and provides an appropriate answer.

[0199] Prompt Sentence Examples

[0200] "Please determine whether the ramen ingredients have fallen apart."

[0201] In this way, the system of the present invention controls the quality of food in real time during the cooking process and delivery, allowing delivery personnel to efficiently deliver high-quality food.

[0202] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0203] Step 1:

[0204] Users and delivery personnel activate devices (smartphones, tablets, smart glasses) to capture real-time images of the cooking process or the state of the food.

[0205] Input: Image from the device's camera.

[0206] Output: Real-time video data.

[0207] Specific operation: The user or delivery person points the camera at the target object (food or dish) and presses the capture button. The device acquires the captured data.

[0208] Step 2:

[0209] The video data captured by the terminal is sent to a streaming server.

[0210] Input: Real-time video data.

[0211] Output: Streamed video data.

[0212] Specific operation: Using the device's communication function, video data is uploaded to a server via the Internet in real time. The data is encrypted and transmitted securely.

[0213] Step 3:

[0214] The server analyzes the video data it receives and evaluates the size of the ingredients, how well mixed they are, how browned they are, and the quality of the food (temperature, appearance, packaging, etc.).

[0215] Input: Streamed video data.

[0216] Output: Analysis results (ingredients size, mixing condition, browning, quality evaluation, etc.).

[0217] Specific operation: The image analysis algorithm is executed on the server to analyze the received video data, extracting and evaluating parameters such as the size of ingredients, the degree of mixing, and browning.

[0218] Step 4:

[0219] The server generates advice based on the analysis results and notifies the user and delivery person.

[0220] Input: Analysis results.

[0221] Output: Advice (audio and text format).

[0222] Specific operation: The server generates appropriate advice based on the analysis results and notifies the user and delivery person using speech synthesis and text generation technologies. The advice is sent via a dedicated application.

[0223] Step 5:

[0224] The user or delivery person asks a question verbally, and the device converts the speech into text and sends it to the server.

[0225] Input: Audio question.

[0226] Output: The question in text format.

[0227] Specific operation: The user or delivery person speaks a question into the terminal, and the terminal uses voice recognition technology to convert the speech into text and send it to the server.

[0228] Step 6:

[0229] The server generates an answer to the received question and notifies the user and the delivery person.

[0230] Input: A text question.

[0231] Output: Answers (in audio and text format).

[0232] Specific operation: The server analyzes the received question and generates an appropriate answer using a generative AI model. The generated answer is then notified to the user and delivery person in voice synthesis and text format.

[0233] Step 7:

[0234] The server stores the analysis results, advice, questions and answers data for future reference and to help improve our services.

[0235] Input: Analysis results, advice, questions, answers.

[0236] Output: Saved data.

[0237] Specific operation: The server uses a database system to permanently store analysis results, advice, questions, and answer data. The stored data can be referenced at a later date and used to improve the service.

[0238] These steps allow for real-time quality control of food during the cooking process and delivery, enabling delivery personnel to efficiently deliver high-quality food.

[0239] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0240] The system of the present invention provides real-time support for cooking by combining visual information and voice communication, and also includes an emotion engine that recognizes the user's emotions. This system allows the user to receive appropriate advice in real time and obtain more effective support during the cooking process.

[0241] The detailed operation of the system is described below.

[0242] First, the user launches the app installed on their smartphone, tablet, or other device. After entering their login information and successful authentication, the device displays a recipe selection screen. The user selects the recipe they want to make and confirms the ingredients and steps required.

[0243] When starting cooking, the user uses the device's camera to film the cooking process. This video is streamed to the server in real time. The server analyzes the received video data and extracts information such as the size of the ingredients, how well they are mixed, and how browned they are. Based on the results of this analysis, the server generates appropriate advice and notifies the user in voice and text format. For example, when frying onions, the server may provide specific instructions such as "You need to fry them a little longer."

[0244] Furthermore, if a user has a question while cooking, they can verbally ask the device. The device uses voice recognition technology to convert the question into text and send it to the server. The server analyzes the question, generates an appropriate answer, and notifies the user by voice. For example, in response to a question such as "What should I do next?", the device will respond with "Next, add the chicken."

[0245] The system includes an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's facial expressions and vocal tone. This allows the server to understand the user's emotional state and adjust the content and tone of advice and responses accordingly. For example, if the user is feeling stressed, the server will provide gentle advice such as, "It's okay, try taking it a little slower."

[0246] Once cooking is complete, the user ends the session within the app and the server stores the cooking process data for future reference and analysis.

[0247] As a concrete example, consider a user making teriyaki chicken. The user launches the app, logs in, and selects the teriyaki chicken recipe. They prepare the ingredients and use their smartphone camera to film the cooking process. The server receives the video and provides specific advice in real time, such as "You need to fry the onions a little longer." Furthermore, if the user asks, "Is the chicken browned to the desired degree?" the server analyzes the video and returns an appropriate answer. If the emotion engine recognizes that the user is stressed, it provides supportive advice, such as "Take it slowly, it's fine."

[0248] Finally, when cooking is complete, the user ends the session and the cooking data is saved on the server. This system allows users to cook with peace of mind, receiving detailed support that also responds to emotional changes.

[0249] The processing flow will be explained below.

[0250] Step 1:

[0251] A user launches an app installed on a device such as a smartphone or tablet, and the device displays a login screen prompting the user to enter their username and password.

[0252] Step 2:

[0253] The user enters their username and password and presses the login button. The device temporarily stores the entered information and prepares to send it to the server. The device then sends an authentication request to the server.

[0254] Step 3:

[0255] The server compares the received user information with the records in its database and generates an authentication result. If the authentication is successful, the server returns an authentication success message to the terminal.

[0256] Step 4:

[0257] The device receives the authentication result and displays success or failure to the user. If authentication is successful, the recipe selection screen is displayed.

[0258] Step 5:

[0259] The user selects the desired recipe from the list and enters the selected recipe ID into the terminal, which temporarily stores the selected recipe ID and prepares to send it to the server.

[0260] Step 6:

[0261] The terminal sends a request for the selected recipe to the server. The server receives the request and retrieves detailed data for the selected recipe from the database. The server then sends the retrieved recipe data to the terminal.

[0262] Step 7:

[0263] The device displays the received recipe data and tells the user the ingredients and steps required, allowing the user to prepare the ingredients and begin cooking.

[0264] Step 8:

[0265] The user initiates the cooking process and captures the video using the device's camera. The device starts recording in real time and streams the video data to the server.

[0266] Step 9:

[0267] The server receives the video data and temporarily stores it for analysis. Using an image analysis algorithm, the server extracts information such as the size of the ingredients, how well they are mixed, and how browned they are, allowing it to determine the appropriate cooking status.

[0268] Step 10:

[0269] The server generates appropriate advice based on the analysis results, and the advice is sent to the device in voice and text format.

[0270] Step 11:

[0271] The device will display and voice the received advice to the user, who will then proceed with the next cooking step based on the advice.

[0272] Step 12:

[0273] If a user has a question while cooking, they can ask it verbally to the device, which will then use voice recognition technology to convert the question into text and send it to the server.

[0274] Step 13:

[0275] The server analyzes the textual content of the question and generates an appropriate answer, which is then sent to the device in voice and text format.

[0276] Step 14:

[0277] The device displays the received answers to the user and notifies them by voice, allowing the user to proceed with cooking while having their questions resolved.

[0278] Step 15:

[0279] By analyzing the user's facial expressions and voice tone from the camera footage, the device uses an emotion engine to recognize the user's emotional state, determining whether the user is stressed or relaxed.

[0280] Step 16:

[0281] The server adjusts the tone and content of its advice and responses based on the user's perceived emotional state. For example, if the user is feeling stressed, the server will provide considerate advice in a gentle tone, such as "Try taking it a little slower."

[0282] Step 17:

[0283] When the user finishes cooking, they press the end session button in the app, and the device sends a session end request to the server.

[0284] Step 18:

[0285] The server receives the session data, saves it in the database, and notifies the terminal that the save is complete.

[0286] Step 19:

[0287] The device will display and sound a message to the user that cooking is complete, and the entire cooking process will be completed.

[0288] Through the above process, the system of the present invention provides users with real-time cooking support through visual information and voice communication, and furthermore, by recognizing the user's emotional state using an emotion engine, it realizes detailed support.

[0289] Example 2

[0290] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0291] Conventional cooking assistance systems make it difficult for users to receive appropriate advice during the cooking process. They also lack the ability to respond to the user's emotional state, making it difficult to provide sufficient support to users who are feeling stressed or anxious. This increases the likelihood of inconvenience and mistakes during cooking, potentially resulting in a poor user experience.

[0292] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0293] In this invention, the server includes: means for a user to use a device to film the cooking process; means for streaming the cooking process to the server in real time; means for the server to analyze the received video data and extract information such as the size and mixing condition of ingredients and browning; means for generating advice based on the extracted information and notifying the user of the advice; means for the user to verbally ask a question; means for converting the verbal question into text and sending it to the server; means for the server to generate an answer to the question and notify the user; means for recognizing emotions from the user's facial expression and tone of voice; means for adjusting the content of the advice and answer based on the user's emotional state; and means for saving the cooking process data. This allows the user to receive appropriate advice in real time, quickly respond to any questions they may have while cooking, and receive support tailored to their emotional state, allowing them to cook safely and effectively.

[0294] "User" refers to an individual who uses the system to cook food.

[0295] "Device" refers to a computer terminal used by a user, such as a smartphone or tablet.

[0296] "Server" refers to a central remote computing system that performs processing such as data analysis and advice generation.

[0297] The "cooking process" refers to the series of operations and actions involved in preparing a dish.

[0298] "Means for taking pictures" refers to a method or device for recording images using the camera function of the device.

[0299] "Means of streaming in real time" refers to methods and technologies for transmitting captured video to a server without delay.

[0300] "Video data" refers to video information of the cooking process captured by a camera.

[0301] "Means of analysis" refers to the computational process that processes the video data and extracts important information such as the size of the ingredients, how well they are mixed, and how browned they are.

[0302] "Ingredient size" refers to the physical dimensions or volume of the ingredient being cooked.

[0303] "Mixing condition" refers to the state of how evenly the ingredients are mixed together.

[0304] "Brown" refers to the state in which ingredients change color when heated, particularly the degree of doneness.

[0305] "Advice" refers to specific cooking instructions or suggestions provided to users based on the analyzed information.

[0306] "Verbal means of questioning" refers to the method by which a user uses voice to express questions or instructions to the system.

[0307] "Means for converting to text" refers to speech recognition technology that converts speech into text data.

[0308] "Answer to a question" refers to an answer or explanation generated by the server in response to a user's question.

[0309] "Facial expressions" refer to emotions and states that can be read from the user's facial movements and muscle placement.

[0310] "Voice tone" refers to the pitch and tone of a user's voice when they speak, and is an element that expresses emotion.

[0311] "Means of emotion recognition" refers to technologies and algorithms that determine a user's emotional state from their facial expressions and voice.

[0312] "Emotional state" refers to the emotional state a user is feeling, such as joy, anger, sadness, or fear.

[0313] "Adjustment techniques" refer to methods and techniques for appropriately changing the content and tone of advice or responses depending on the person's emotional state.

[0314] "Data storage means" refers to a method or system that keeps a record of the cooking process for future reference or analysis.

[0315] The system of the present invention provides real-time support for cooking by combining visual information and voice communication. It also includes an emotion engine that recognizes the user's emotions. This system allows the user to receive appropriate advice in real time, providing more effective support during the cooking process.

[0316] First, the user launches the app installed on their smartphone, tablet, or other device. After entering their login information and successful authentication, the device displays a recipe selection screen. The user selects the recipe they want to make and confirms the ingredients and steps required.

[0317] When starting cooking, the user uses the device's camera to film the cooking process. This video is streamed in real time to a server. The server analyzes the received video data and extracts information such as the size of ingredients, the degree of mixing, and browning. This analysis uses image processing techniques (e.g., OpenCV) and generative AI models (e.g., TensorFlow).

[0318] Based on the analysis results, the server generates appropriate advice and notifies the user in voice and text format. For example, when frying onions, it provides specific instructions such as "You need to fry them a little longer."

[0319] Furthermore, if a user has a question while cooking, they can verbally ask the device. In this case, the device uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert the question into text and send it to the server. The server analyzes the question, generates an appropriate answer, and notifies the user by voice. For example, in response to the question, "What should I do next?", the device will respond with, "Next, add the chicken."

[0320] Furthermore, the system includes an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's facial expressions and vocal tone. This recognition uses computer vision and voice analysis technologies (e.g., Azure Cognitive Services). This allows the server to understand the user's emotional state and adjust the content and tone of its advice and responses accordingly. For example, if the user is feeling stressed, the server will provide gentle advice such as, "It's okay, try taking it a little slower."

[0321] Once cooking is complete, the user ends the session within the app and the server stores the cooking process data for future reference and analysis.

[0322] As a concrete example, consider a user making teriyaki chicken. The user launches the app, logs in, and selects the teriyaki chicken recipe. They prepare the ingredients and use their smartphone camera to film the cooking process. The server receives the video and provides specific advice in real time, such as "You need to fry the onions a little longer." If the user asks, "Is the chicken browned properly?", the server analyzes the video and returns an appropriate answer. Additionally, if the emotion engine recognizes that the user is stressed, it provides supportive advice such as "Take it slowly, it's fine."

[0323] Finally, when cooking is complete, the user ends the session and the cooking data is saved on the server. This system allows users to cook with peace of mind, receiving detailed support that also responds to emotional changes.

[0324] Example prompt sentence:

[0325] "Show me the chicken teriyaki recipe."

[0326] "Analyze the results of frying onions."

[0327] "What should I do next?"

[0328] "Is this the right browning for the chicken?"

[0329] "Provide advice on what to do if users are stressed."

[0330] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0331] Step 1:

[0332] App launch and authentication

[0333] Input: The user launches the app on their smartphone or tablet and enters their login information (user ID and password).

[0334] Output: A response from the server indicating authentication success or failure.

[0335] Specific behavior:

[0336] 1. The user launches the app.

[0337] 2. The login screen appears and the user enters their ID and password.

[0338] 3. The device sends the input information to the server.

[0339] 4. The server checks the database for authentication and returns the result to the terminal.

[0340] Step 2:

[0341] Recipe selection and display

[0342] Input: The recipe selected by the user on the recipe selection screen upon successful authentication.

[0343] Output: Ingredient list and instructions for the selected recipe.

[0344] Specific behavior:

[0345] 1. After successful authentication, the device displays the recipe selection screen.

[0346] 2. The user selects the recipe they want to make, and the device requests that information from the server.

[0347] 3. The server receives the request and sends the ingredients list and instructions for the corresponding recipe to the device.

[0348] 4. The terminal displays it on the screen.

[0349] Step 3:

[0350] Filming and streaming the cooking process

[0351] Input: User starts cooking, footage captured by camera.

[0352] Output: Real-time video data sent to the server.

[0353] Specific behavior:

[0354] 1. Users use a camera to record the cooking process in real time.

[0355] 2. The video captured by the device is streamed to the server in real time.

[0356] Step 4:

[0357] Video data analysis

[0358] Input: Streamed video data.

[0359] Output: Analysis results such as size of ingredients, mixing condition, browning, etc.

[0360] Specific behavior:

[0361] 1. The server analyzes the received video data.

[0362] 2. Image processing technology (e.g., OpenCV) and generative AI models (e.g., TensorFlow) are used to determine the size, mixing level, and browning of ingredients.

[0363] 3. The server processes the analysis results in real time.

[0364] Step 5:

[0365] Providing real-time advice

[0366] Input: Analysis results.

[0367] Output: Advice that will be sent to the user.

[0368] Specific behavior:

[0369] 1. The server generates advice based on the analysis results.

[0370] 2. Send advice in voice and text format to the device.

[0371] 3. The device notifies the user of the advice.

[0372] 4. For example, the advice "It needs to be cooked a little longer" will be displayed.

[0373] Step 6:

[0374] Audio questions and answers

[0375] Input: The user's spoken question.

[0376] Output: A suitable response from the server.

[0377] Specific behavior:

[0378] 1. The user has a question and verbally asks it to the device.

[0379] 2. The device uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert the question into text.

[0380] 3. Send the converted text to the server.

[0381] 4. The server analyzes the question, generates an appropriate answer, and sends it to the device.

[0382] 5. The device will notify the user of the answer via voice and text.

[0383] 6. For example, to the question "What should I do next?", the answer announced is "Next, add the chicken."

[0384] Step 7:

[0385] Emotion Recognition and Response

[0386] Input: The user's facial expression and voice tone.

[0387] Output: Advice or answer tailored by the server.

[0388] Specific behavior:

[0389] 1. The device uses a camera and microphone to record the user's facial expressions and voice tone.

[0390] 2. Analyze user emotions using emotion recognition technology (e.g., Azure Cognitive Services).

[0391] 3. Send the analysis results to the server.

[0392] 4. The server adjusts the content and tone of advice and responses based on the user's emotional state.

[0393] 5. For example, if the user is feeling stressed, the device will provide advice such as, "Take it easy, it's okay."

[0394] Step 8:

[0395] Ending a session and saving data

[0396] Input: Session termination information.

[0397] Output: Saved cooking process data.

[0398] Specific behavior:

[0399] 1. After the user finishes cooking, they end the session within the app.

[0400] 2. The terminal sends the session termination information to the server.

[0401] 3. The server stores the cooking process data in a database.

[0402] (Application example 2)

[0403] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0404] Driver stress and reduced attention in autonomous vehicles are a major issue, increasing the risk of traffic accidents. There is also a need to provide a safe driving environment where drivers can receive appropriate advice in real time while driving.

[0405] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to use the device to film the driving process, means for streaming the driving process to the server in real time, means for analyzing the video data received by the server and recognizing the user's emotions and attention state, and means for generating advice or warnings based on the recognized emotions and notifying the user of the advice or warnings. This makes it possible to provide detailed support according to the user's emotional state and improve safety and comfort while driving.

[0406] "User" means an individual or vehicle operator using the system.

[0407] "Devices" are hardware devices such as cameras and microphones installed in autonomous vehicles.

[0408] The "means for photographing the cooking process" is a function that uses a camera inside the self-driving vehicle to photograph the driving situation.

[0409] "Means for streaming to a server in real time" refers to a technology that transmits captured video to a server without delay.

[0410] "Means for analyzing video data and extracting information on the size, mixing level and browning of ingredients" refers to an algorithm that analyzes video data and recognizes the operating status.

[0411] The "means for generating advice based on the extracted information and notifying the user of the advice" is a technology for generating appropriate driving guidance or warnings based on the analysis results and providing them to the user.

[0412] "Means for users to ask questions verbally" refers to a function that allows the driver to ask questions verbally using a microphone.

[0413] The "means for converting an oral question into text and transmitting it to a server" is a technology that uses voice recognition technology to convert an oral question into text format and transmit it to a server.

[0414] "Means for generating answers to questions and notifying users" refers to a technology in which the server generates an appropriate answer based on the question received and notifies the user of the answer by voice or text.

[0415] The "means for recognizing emotions from a user's facial expression and voice tone" is a technology that performs facial expression analysis and voice analysis to recognize the user's emotional state.

[0416] "Means for adjusting the content and tone of advice or responses based on recognized emotions" refers to a technique for adjusting the content of advice or responses provided based on the recognized emotional state, and using a gentle tone or encouraging words as necessary.

[0417] "Means for saving cooking process data" refers to a function that saves the user's operating data on a server so that it can be referenced and analyzed later.

[0418] The present invention relates to a driving assistance system for autonomous vehicles, and aims to provide appropriate advice to users in real time while they are driving, thereby improving the safety and comfort of the driving environment.

[0419] The system consists of the following main components:

[0420] 1. User Device:

[0421] Hardware:

[0422] Camera: Used to record the inside of the vehicle.

[0423] Microphone: Used to capture the user's voice.

[0424] software:

[0425] Speech recognition: Software that converts speech into text.

[0426] Video capture: Software that captures camera footage and streams it to a server.

[0427] 2. Server:

[0428] Hardware:

[0429] A computer equipped with a high-performance processor and large memory capacity.

[0430] software:

[0431] Image analysis algorithm (OpenCV, etc.): Analyzes camera footage and recognizes the user's facial expressions and driving state.

[0432] Emotion recognition engine (deep learning models such as Keras): Recognizes user emotions from facial expressions and voice tone.

[0433] Auto-response generation system (e.g. gTTS): Generates appropriate advice and warnings and notifies them in audio format.

[0434] Speech recognition software (such as SpeechRecognition): Converts spoken questions into text.

[0435] 3. System Operation:

[0436] The user device uses a camera to capture video of the interior of the vehicle and streams it to a server in real time.

[0437] The server analyzes the received video data and recognizes the user's facial expressions and driving state.

[0438] Based on the recognized data, the server generates appropriate advice or warnings and notifies the user in voice format.

[0439] When a user speaks a question, a microphone captures it and speech recognition software converts the question into text and sends it to a server.

[0440] The server generates answers to the questions and communicates them to the user in audio format.

[0441] 4. Example:

[0442] For example, if a user is feeling stressed while driving, the system will recognize this from their facial expression and provide voice advice such as, "You seem a little stressed. Take a deep breath." If the user says, "It's dangerous," the system will warn, "Be careful! There is danger ahead."

[0443] 5. Example prompts:

[0444] "If the user is feeling stressed while driving, the system should recognize facial expressions from the camera footage and provide voice advice such as, 'You seem to be a little stressed. Take a deep breath.' If the user says, 'It's dangerous,' the system should warn them, 'Be careful! There is danger ahead.'"

[0445] This system provides detailed support tailored to the user's emotional state, improving safety and comfort while driving.

[0446] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0447] Step 1:

[0448] The user device uses a camera to capture images of the inside of the car. The input is real-time images of the inside of the car, and the output is camera image data. Specifically, the camera device captures the entire view of the inside of the car and the user's face, and generates this as streaming data.

[0449] Step 2:

[0450] The video data captured by the user device is streamed to the server in real time. The input is the camera video data, and the output is the streaming data to the server. Specifically, the video data is sent to the server via the network.

[0451] Step 3:

[0452] The server analyzes the received video data and recognizes the user's facial expressions and driving state. The input is streaming video data, and the output is data on the recognized facial expressions and driving state. Specifically, the video is analyzed frame by frame using OpenCV's image analysis algorithm, and features are extracted for face detection and emotion recognition.

[0453] Step 4:

[0454] The server uses an emotion recognition engine to recognize the user's emotions from facial expressions and voice tone. The input is the analyzed video data and the user's voice data, and the output is the recognized emotional state. Specifically, a deep learning model using Keras classifies emotions such as joy, anger, and sadness from facial expressions and analyzes voice tone.

[0455] Step 5:

[0456] The server generates appropriate advice or warnings based on the recognized data. The input is the recognized emotional state data, and the output is text data of the advice or warning. Specifically, a pre-defined advice or warning message is selected based on the generated emotional data.

[0457] Step 6:

[0458] The server notifies the user of the generated advice or warning in audio format. The input is the text data of the generated advice or warning, and the output is the audio notification data. Specifically, the server uses gTTS to convert the text data into an audio file and plays it back to the user through the speaker.

[0459] Step 7:

[0460] The user asks a question by voice and the device captures it. The input is the user's voice data, and the output is the captured voice data. Specifically, the microphone device records what the user says.

[0461] Step 8:

[0462] The device converts the voice question into text and sends it to the server. The input is the captured voice data, and the output is the text question data. Specifically, SpeechRecognition software is used to convert the voice into text data and send it to the server.

[0463] Step 9:

[0464] The server generates an answer to the question and notifies the user. The input is text question data, and the output is audio answer data. Specifically, an appropriate answer is generated based on the question content, converted into audio using gTTS, and played back to the user through the speaker.

[0465] Step 10:

[0466] The server stores data on the driving process. The input is all data collected during driving, and the output is the stored data. Specifically, the server stores the user's facial expression data, voice data, advice history, etc. in the database.

[0467] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0468] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search<url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0469] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0470] [Second embodiment]

[0471] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0472] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0473] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0474] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0475] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0476] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0477] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0478] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0479] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0480] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0481] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0482] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0483] The system of the present invention provides real-time support for cooking by combining visual information and voice communication. The system allows the user to use a device to film the cooking process and transmit the video in real time to a server, which can then provide appropriate advice on cooking methods and progress.

[0484] The detailed operation of the system is described below.

[0485] First, the user launches the app installed on their smartphone, tablet, or other device. After entering their login information and successful authentication, the device displays a recipe selection screen. The user selects the recipe they want to make and confirms the ingredients and steps required.

[0486] When the user starts cooking, the device's camera captures video. This video is streamed to the server in real time. The server analyzes the received video data and extracts information such as the size of the ingredients, how well they are mixed, and how browned they are. This allows the server to determine whether the ingredients are cut to the right size, mixed thoroughly, and cooked to the right degree.

[0487] The server generates advice based on the analysis results and notifies the user in voice and text format. For example, when frying onions, it may provide specific instructions such as "You need to fry them a little longer." This advice helps the user understand what steps to take next.

[0488] Furthermore, if a user has a question while cooking, they can verbally ask the device. The device uses voice recognition technology to convert the question into text and send it to the server. The server analyzes the question, generates an appropriate answer, and notifies the user by voice. For example, in response to a question such as "What should I do next?", the device will respond with "Next, add the chicken."

[0489] Once cooking is complete, the user ends the session within the app and the server stores the cooking process data for future reference and analysis.

[0490] As a concrete example, consider a user making teriyaki chicken. The user launches the app, logs in, and selects the teriyaki chicken recipe. They prepare the ingredients and use their smartphone camera to film the cooking process. The server receives the video and provides specific advice in real time, such as "You need to fry the onions a little longer." If the user also asks, "Is the chicken browned to the desired degree?" the server analyzes the video and returns an appropriate answer. Finally, when cooking is complete, the user ends the session, and the cooking data is saved to the server.

[0491] As described above, the system of the present invention combines real-time video analysis and voice communication to provide detailed and immediate support to users, allowing even beginners and users who are unsure about certain dishes to cook with confidence.

[0492] The processing flow will be explained below.

[0493] Step 1:

[0494] A user launches an app installed on a device such as a smartphone or tablet, and the device displays a login screen prompting the user to enter their username and password.

[0495] Step 2:

[0496] The user enters their username and password and presses the login button. The device temporarily stores the entered information and prepares to send it to the server. The device then sends an authentication request to the server.

[0497] Step 3:

[0498] The server compares the received user information with the records in its database and generates an authentication result. If the authentication is successful, the server returns an authentication success message to the terminal.

[0499] Step 4:

[0500] The device receives the authentication result and displays success or failure to the user. If authentication is successful, the recipe selection screen is displayed.

[0501] Step 5:

[0502] The user selects the desired recipe from the list and enters the selected recipe ID into the terminal, which temporarily stores the selected recipe ID and prepares to send it to the server.

[0503] Step 6:

[0504] The terminal sends a request for the selected recipe to the server. The server receives the request and retrieves detailed data for the selected recipe from the database. The server then sends the retrieved recipe data to the terminal.

[0505] Step 7:

[0506] The device displays the received recipe data and tells the user the ingredients and steps required, allowing the user to prepare the ingredients and begin cooking.

[0507] Step 8:

[0508] The user initiates the cooking process and captures the video using the device's camera. The device starts recording in real time and streams the video data to the server.

[0509] Step 9:

[0510] The server receives the video data and temporarily stores it for analysis. The server then uses image analysis algorithms to extract information such as the size of the ingredients, how well they are mixed, and how browned they are.

[0511] Step 10:

[0512] The server generates appropriate advice based on the analysis results, and the advice is sent to the device in voice and text format.

[0513] Step 11:

[0514] The device will display and voice the received advice to the user, who will then proceed with the next cooking step based on the advice.

[0515] Step 12:

[0516] If a user has a question while cooking, they can ask it verbally to the device, which will then use voice recognition technology to convert the question into text and send it to the server.

[0517] Step 13:

[0518] The server analyzes the textual content of the question and generates an appropriate answer, which is then sent to the device in voice and text format.

[0519] Step 14:

[0520] The device displays the received answers to the user and notifies them by voice, allowing the user to proceed with cooking while having their questions resolved.

[0521] Step 15:

[0522] When the user finishes cooking, they press the end session button in the app, and the device sends a session end request to the server.

[0523] Step 16:

[0524] The server receives the session data, saves it in the database, and notifies the terminal that the save is complete.

[0525] Step 17:

[0526] The device will display and sound a message to the user that cooking is complete, and the entire cooking process will be completed.

[0527] Example 1

[0528] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0529] The lack of a system that allows users to receive appropriate advice and guidance in real time while cooking poses a challenge, making it difficult for beginners and users who are unsure about certain dishes to cook with confidence. Furthermore, conventional cooking support systems are unable to respond immediately to users' questions, making it difficult to provide appropriate support tailored to the user's actual cooking progress.

[0530] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0531] In this invention, the server includes means for a user to use a device to film the cooking process, means for streaming the cooking process to the server in real time, means for the server to analyze the video data received and extract information on the size and mixing of ingredients and browning, means for generating advice based on the extracted information and notifying the user of the advice in audio and text format, means for the user to verbally ask a question, means for converting the verbal question into text and sending it to the server, means for the server to generate an answer to the question and notify the user in audio and text format, and means for saving the cooking process data. This allows users to receive appropriate cooking guidance and advice in real time, allowing even beginners and users who are unsure about certain dishes to cook with confidence.

[0532] A "user" is a person who uses the system to assist with the cooking process.

[0533] "Device" refers to a mobile device such as a smartphone or tablet used by a user.

[0534] The "cooking process" refers to the series of steps involved in preparing ingredients, cutting, mixing, heating, etc., to complete a dish.

[0535] "Streaming" is a technology that transmits captured video data to a server in real time.

[0536] The "server" is a central processing unit that receives and analyzes the video data sent from the user's device and generates corresponding advice and answers.

[0537] "Video data" refers to images and videos of the cooking process captured by the device's camera.

[0538] "Analysis" refers to the process of evaluating the size of ingredients, how well mixed they are, and how browned they are based on the received video data.

[0539] "Ingredients" refer to the individual ingredients that make up a dish.

[0540] "Advice" refers to cooking instructions and advice that are generated based on the analysis results and provided to the user.

[0541] "Audio format" refers to a format in which advice or answers are notified to the user as audio.

[0542] "Text format" refers to a format in which advice or answers are notified to the user in written form.

[0543] "Questions" are questions or problems that users verbally input while cooking.

[0544] "Voice recognition technology" refers to technology that converts a user's verbal questions into text.

[0545] An "answer" is a response generated by a server in response to a question from a user.

[0546] "Data storage" refers to the process of storing data about the cooking process on a server for future reference.

[0547] "Real-time" refers to a time concept in which analysis and notification are carried out immediately as the cooking process progresses.

[0548] "Computer vision technology" is a general term for image recognition technology used to analyze video data.

[0549] The system of the present invention provides real-time support for cooking by combining visual information and voice communication. A specific embodiment of the system will now be described.

[0550] First, the user launches a dedicated app installed on a mobile device such as a smartphone or tablet. After the user enters their login information and is successfully authenticated, the device displays a recipe selection screen. The user selects the recipe they want to make and checks the ingredients and steps. The software used by the device at this stage generally includes the operating system (e.g., iOS, Android) and application software for the smartphone or tablet.

[0551] Next, when the user starts cooking, the device's camera is used to capture video. This video is then streamed to the server in real time using technologies such as the Real-Time Messaging Protocol (RTMP) and HTTP Live Streaming (HLS). This allows the video data to be sent to the server in real time.

[0552] The server uses computer vision technology (e.g., OpenCV or TensorFlow) to analyze the received video data. The server extracts information such as the size of the ingredients, the degree of mixing, and the browning of the ingredients, and performs analysis based on this. Based on the analysis results, the server generates appropriate advice and sends it to the device. The advice is provided in audio (e.g., Google Cloud Text-to-Speech API) and text format. The device notifies the user of the advice it received.

[0553] If a user has a question while cooking, they can verbally ask the device. The device will convert the question into text using voice recognition technology (e.g., Google Cloud Speech-to-Text API) and send it to the server. The server will analyze the question, generate an appropriate answer, and notify the user in voice and text format.

[0554] Once cooking is complete, the user can end the session within the app. The server then stores the cooking process data for future reference and analysis. This data includes analysis results, user questions and answers, and any advice provided.

[0555] As a concrete example, consider a user making teriyaki chicken. The user launches the app, logs in, and selects the teriyaki chicken recipe. They prepare the ingredients and use their smartphone camera to film the cooking process. The server receives the video and provides specific advice in real time, such as "You need to fry the onions a little longer." When the user asks, "Is the chicken browned to the desired degree?" the server analyzes the video and returns an appropriate answer. Finally, when cooking is complete, the user ends the session, and the cooking data is saved to the server.

[0556] Examples of prompts include:

[0557] "I'm making chicken teriyaki. I'm currently frying onions. Can you tell me the proper doneness?"

[0558] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0559] Step 1: Launching the program and authenticating the user

[0560] Users launch a dedicated app on their smartphone or tablet and enter their login information (username and password).

[0561] The terminal sends the entered login information to the server (input: user name, password).

[0562] The server compares the received login information with the authentication database and performs authentication (data processing: comparison with authentication database, output: authentication result).

[0563] If the authentication is successful, the server starts a session and displays the recipe selection screen on the terminal (output: session ID, recipe selection screen).

[0564] Step 2: Select a recipe and check ingredients

[0565] The user selects the recipe they want to make on the device screen (input: select recipe).

[0566] The terminal transmits the selected recipe information to the server (input: recipe information).

[0567] The server sends the ingredients and cooking instructions required for the selected recipe to the terminal (data processing: recipe data extraction, output: ingredient list, procedure list).

[0568] The terminal displays it on the screen, and the user checks the ingredients and proceeds with preparation (output: ingredient list, procedure list).

[0569] Step 3: Record and send footage

[0570] The user starts cooking and takes pictures using the device's camera.

[0571] The terminal streams the captured video data to the server in real time (input: video data).

[0572] The terminal transmits video data using the Real-Time Messaging Protocol (RTMP) or HTTP Live Streaming (HLS) (output: real-time video data).

[0573] Step 4: Video analysis on the server

[0574] The server uses computer vision techniques (e.g., OpenCV or TensorFlow) to analyze the received video data (input: video data).

[0575] The server extracts information such as the size of the ingredients, how well mixed they are, and how browned they are (data processing: video analysis, output: ingredient information).

[0576] The cooking progress is evaluated based on the analysis results (data calculation: cooking progress evaluation, output: progress evaluation results).

[0577] Step 5: Server generates and notifies advice

[0578] The server generates specific advice based on the analysis results (data processing: advice generation, output: advice).

[0579] The server sends the generated advice to the terminal in voice (e.g., Text-to-Speech API) and text format (input: advice, output: voice data, text data).

[0580] The device notifies the user of the received advice (specific actions: playing audio, displaying text).

[0581] Step 6: Respond to user questions

[0582] If the user has a question while cooking, they can ask verbally into the terminal (input: voice question).

[0583] The device uses voice recognition technology (e.g., Speech-to-Text API) to convert the question into text and send it to the server (data processing: voice recognition, output: text question).

[0584] The server analyzes the question and generates an appropriate answer (data processing: question analysis, output: answer).

[0585] The server sends the generated answer to the terminal in voice and text format (input: answer, output: voice data, text data).

[0586] The terminal notifies the user of the answer by voice and text (specific operations: playing voice, displaying text).

[0587] Step 7: Save the data after cooking is complete

[0588] When the user is done cooking, they can end the session within the app (input: end session).

[0589] The server stores the data of the cooking process (data processing: data storage, output: stored data).

[0590] The saved data is used for future reference and analysis (specific operation: saving to a database).

[0591] (Application example 1)

[0592] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0593] With current food delivery services, it is difficult to control the quality of food during delivery, making it difficult for delivery personnel to efficiently deliver food while checking its condition. Furthermore, there is a lack of means to receive real-time feedback on the state of the food, making it a challenge to improve delivery quality.

[0594] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0595] In this invention, the server includes: means for a user and a delivery person to use a device to film the cooking process and the state of the food; means for streaming the cooking process and the state of the food to the server in real time; means for the server to analyze the received video data and evaluate information on the size and mixture of ingredients and browning and the quality of the food; means for generating advice based on the extracted information and evaluation results and notifying the user and the delivery person of the advice; means for the user and the delivery person to ask verbal questions; means for converting the verbal questions into text and sending it to the server; means for the server to generate answers to the questions and notify the user and the delivery person; and means for storing the cooking process data and food state data. This enables real-time quality control of the cooking process and food during delivery, enabling the delivery person to efficiently deliver high-quality food.

[0596] "User" means any person who uses the system to receive real-time assistance and advice during the cooking process.

[0597] "Delivery person" means a person who performs quality control during the delivery of food in a food delivery service.

[0598] The "device" is a hard device that has a camera function and transmits the actions of the user or delivery person to the server in real time.

[0599] A "server" is a computing device that analyzes received video data and generates advice.

[0600] "Video data" refers to visual information captured by the user or delivery person showing the cooking process and the condition of the food.

[0601] "Analysis" is the process of evaluating the size of ingredients, the degree of mixing, the browning, and the quality of the food based on the received video data.

[0602] "Advice" refers to instructions and advice regarding cooking and quality control provided based on the analysis results.

[0603] "Streaming" is a technology for transmitting video data to a server in real time.

[0604] "Questions" are questions or clarifications about the cooking process or the condition of the food that users or delivery personnel ask the system.

[0605] "Text conversion" is the process of converting spoken questions into written information.

[0606] "Notification" is the act of transmitting advice or answers generated by the server to the user or delivery person.

[0607] "Data archiving" is the process of recording information about analysis results and recommendations for future reference.

[0608] "Quality evaluation" is the process of determining the quality of food based on its condition.

[0609] "Audio format" refers to a means of providing advice or answers to users or delivery personnel via voice.

[0610] An "image analysis algorithm" is a mathematical technique used to analyze video data.

[0611] "Food condition" refers to information such as the temperature, appearance, and packaging of the food during delivery.

[0612] The system of the present invention allows users and delivery staff to use devices to film the cooking process and the condition of the food, and sends the footage to a server in real time, thereby providing appropriate advice on cooking methods and progress and managing food quality.

[0613] System Configuration

[0614] 1. Hardware

[0615] Camera-equipped devices such as smartphones, tablets, and smart glasses

[0616] Computing Server

[0617] 2. Software

[0618] Image Analysis Algorithm

[0619] Speech Recognition and Synthesis Software

[0620] Streaming Server

[0621] Data processing and calculation

[0622] Video data collection and streaming

[0623] Users and delivery staff use devices (smartphones, tablets, smart glasses) to record the cooking process and the state of the food, and this video data is sent to a streaming server in real time.

[0624] Video data analysis

[0625] The streaming server analyzes the received video data, using image analysis algorithms to assess the size, consistency, browning, and food quality (temperature, appearance, packaging, etc.) of ingredients.

[0626] Advice generation and notification

[0627] Based on the analysis results, the server generates advice on the cooking progress and food quality, which is communicated to the user and delivery person in voice and text format.

[0628] Questions and Answers

[0629] When a user or delivery person verbally asks a question, the device converts the voice into text and sends it to the server, which analyzes the question, generates an appropriate answer, and notifies the user in voice and text format.

[0630] Data storage

[0631] Analysis results, advice, and question and answer data are stored by the server for future reference and service improvement.

[0632] Specific examples

[0633] Ramen delivery

[0634] A delivery person wears smart glasses and takes a video to check the condition of the ramen during delivery. The streaming server analyzes the video and provides advice such as "The soup has not spilled" and "The ramen ingredients have not fallen apart." If the delivery person asks, "Please check if the ramen ingredients have fallen apart," the voice is converted into text, and the server analyzes the question and provides an appropriate answer.

[0635] Prompt Sentence Examples

[0636] "Please determine whether the ramen ingredients have fallen apart."

[0637] In this way, the system of the present invention controls the quality of food in real time during the cooking process and delivery, allowing delivery personnel to efficiently deliver high-quality food.

[0638] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0639] Step 1:

[0640] Users and delivery personnel activate devices (smartphones, tablets, smart glasses) to capture real-time images of the cooking process or the state of the food.

[0641] Input: Image from the device's camera.

[0642] Output: Real-time video data.

[0643] Specific operation: The user or delivery person points the camera at the target object (food or dish) and presses the capture button. The device acquires the captured data.

[0644] Step 2:

[0645] The video data captured by the terminal is sent to a streaming server.

[0646] Input: Real-time video data.

[0647] Output: Streamed video data.

[0648] Specific operation: Using the device's communication function, video data is uploaded to a server via the Internet in real time. The data is encrypted and transmitted securely.

[0649] Step 3:

[0650] The server analyzes the video data it receives and evaluates the size of the ingredients, how well mixed they are, how browned they are, and the quality of the food (temperature, appearance, packaging, etc.).

[0651] Input: Streamed video data.

[0652] Output: Analysis results (ingredients size, mixing condition, browning, quality evaluation, etc.).

[0653] Specific operation: The image analysis algorithm is executed on the server to analyze the received video data, extracting and evaluating parameters such as the size of ingredients, the degree of mixing, and browning.

[0654] Step 4:

[0655] The server generates advice based on the analysis results and notifies the user and delivery person.

[0656] Input: Analysis results.

[0657] Output: Advice (audio and text format).

[0658] Specific operation: The server generates appropriate advice based on the analysis results and notifies the user and delivery person using speech synthesis and text generation technologies. The advice is sent via a dedicated application.

[0659] Step 5:

[0660] The user or delivery person asks a question verbally, and the device converts the speech into text and sends it to the server.

[0661] Input: Audio question.

[0662] Output: The question in text format.

[0663] Specific operation: The user or delivery person speaks a question into the terminal, and the terminal uses voice recognition technology to convert the speech into text and send it to the server.

[0664] Step 6:

[0665] The server generates an answer to the received question and notifies the user and the delivery person.

[0666] Input: A text question.

[0667] Output: Answers (in audio and text format).

[0668] Specific operation: The server analyzes the received question and generates an appropriate answer using a generative AI model. The generated answer is then notified to the user and delivery person in voice synthesis and text format.

[0669] Step 7:

[0670] The server stores the analysis results, advice, questions and answers data for future reference and to help improve our services.

[0671] Input: Analysis results, advice, questions, answers.

[0672] Output: Saved data.

[0673] Specific operation: The server uses a database system to permanently store analysis results, advice, questions, and answer data. The stored data can be referenced at a later date and used to improve the service.

[0674] These steps allow for real-time quality control of food during the cooking process and delivery, enabling delivery personnel to efficiently deliver high-quality food.

[0675] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0676] The system of the present invention provides real-time support for cooking by combining visual information and voice communication, and also includes an emotion engine that recognizes the user's emotions. This system allows the user to receive appropriate advice in real time and obtain more effective support during the cooking process.

[0677] The detailed operation of the system is described below.

[0678] First, the user launches the app installed on their smartphone, tablet, or other device. After entering their login information and successful authentication, the device displays a recipe selection screen. The user selects the recipe they want to make and confirms the ingredients and steps required.

[0679] When starting cooking, the user uses the device's camera to film the cooking process. This video is streamed to the server in real time. The server analyzes the received video data and extracts information such as the size of the ingredients, how well they are mixed, and how browned they are. Based on the results of this analysis, the server generates appropriate advice and notifies the user in voice and text format. For example, when frying onions, the server may provide specific instructions such as "You need to fry them a little longer."

[0680] Furthermore, if a user has a question while cooking, they can verbally ask the device. The device uses voice recognition technology to convert the question into text and send it to the server. The server analyzes the question, generates an appropriate answer, and notifies the user by voice. For example, in response to a question such as "What should I do next?", the device will respond with "Next, add the chicken."

[0681] The system includes an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's facial expressions and vocal tone. This allows the server to understand the user's emotional state and adjust the content and tone of advice and responses accordingly. For example, if the user is feeling stressed, the server will provide gentle advice such as, "It's okay, try taking it a little slower."

[0682] Once cooking is complete, the user ends the session within the app and the server stores the cooking process data for future reference and analysis.

[0683] As a concrete example, consider a user making teriyaki chicken. The user launches the app, logs in, and selects the teriyaki chicken recipe. They prepare the ingredients and use their smartphone camera to film the cooking process. The server receives the video and provides specific advice in real time, such as "You need to fry the onions a little longer." Furthermore, if the user asks, "Is the chicken browned to the desired degree?" the server analyzes the video and returns an appropriate answer. If the emotion engine recognizes that the user is stressed, it provides supportive advice, such as "Take it slowly, it's fine."

[0684] Finally, when cooking is complete, the user ends the session and the cooking data is saved on the server. This system allows users to cook with peace of mind, receiving detailed support that also responds to emotional changes.

[0685] The processing flow will be explained below.

[0686] Step 1:

[0687] A user launches an app installed on a device such as a smartphone or tablet, and the device displays a login screen prompting the user to enter their username and password.

[0688] Step 2:

[0689] The user enters their username and password and presses the login button. The device temporarily stores the entered information and prepares to send it to the server. The device then sends an authentication request to the server.

[0690] Step 3:

[0691] The server compares the received user information with the records in its database and generates an authentication result. If the authentication is successful, the server returns an authentication success message to the terminal.

[0692] Step 4:

[0693] The device receives the authentication result and displays success or failure to the user. If authentication is successful, the recipe selection screen is displayed.

[0694] Step 5:

[0695] The user selects the desired recipe from the list and enters the selected recipe ID into the terminal, which temporarily stores the selected recipe ID and prepares to send it to the server.

[0696] Step 6:

[0697] The terminal sends a request for the selected recipe to the server. The server receives the request and retrieves detailed data for the selected recipe from the database. The server then sends the retrieved recipe data to the terminal.

[0698] Step 7:

[0699] The device displays the received recipe data and tells the user the ingredients and steps required, allowing the user to prepare the ingredients and begin cooking.

[0700] Step 8:

[0701] The user initiates the cooking process and captures the video using the device's camera. The device starts recording in real time and streams the video data to the server.

[0702] Step 9:

[0703] The server receives the video data and temporarily stores it for analysis. Using an image analysis algorithm, the server extracts information such as the size of the ingredients, how well they are mixed, and how browned they are, allowing it to determine the appropriate cooking status.

[0704] Step 10:

[0705] The server generates appropriate advice based on the analysis results, and the advice is sent to the device in voice and text format.

[0706] Step 11:

[0707] The device will display and voice the received advice to the user, who will then proceed with the next cooking step based on the advice.

[0708] Step 12:

[0709] If a user has a question while cooking, they can ask it verbally to the device, which will then use voice recognition technology to convert the question into text and send it to the server.

[0710] Step 13:

[0711] The server analyzes the textual content of the question and generates an appropriate answer, which is then sent to the device in voice and text format.

[0712] Step 14:

[0713] The device displays the received answers to the user and notifies them by voice, allowing the user to proceed with cooking while having their questions resolved.

[0714] Step 15:

[0715] By analyzing the user's facial expressions and voice tone from the camera footage, the device uses an emotion engine to recognize the user's emotional state, determining whether the user is stressed or relaxed.

[0716] Step 16:

[0717] The server adjusts the tone and content of its advice and responses based on the user's perceived emotional state. For example, if the user is feeling stressed, the server will provide considerate advice in a gentle tone, such as "Try taking it a little slower."

[0718] Step 17:

[0719] When the user finishes cooking, they press the end session button in the app, and the device sends a session end request to the server.

[0720] Step 18:

[0721] The server receives the session data, saves it in the database, and notifies the terminal that the save is complete.

[0722] Step 19:

[0723] The device will display and sound a message to the user that cooking is complete, and the entire cooking process will be completed.

[0724] Through the above process, the system of the present invention provides users with real-time cooking support through visual information and voice communication, and furthermore, by recognizing the user's emotional state using an emotion engine, it realizes detailed support.

[0725] Example 2

[0726] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0727] Conventional cooking assistance systems make it difficult for users to receive appropriate advice during the cooking process. They also lack the ability to respond to the user's emotional state, making it difficult to provide sufficient support to users who are feeling stressed or anxious. This increases the likelihood of inconvenience and mistakes during cooking, potentially resulting in a poor user experience.

[0728] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0729] In this invention, the server includes: means for a user to use a device to film the cooking process; means for streaming the cooking process to the server in real time; means for the server to analyze the received video data and extract information such as the size and mixing condition of ingredients and browning; means for generating advice based on the extracted information and notifying the user of the advice; means for the user to verbally ask a question; means for converting the verbal question into text and sending it to the server; means for the server to generate an answer to the question and notify the user; means for recognizing emotions from the user's facial expression and tone of voice; means for adjusting the content of the advice and answer based on the user's emotional state; and means for saving the cooking process data. This allows the user to receive appropriate advice in real time, quickly respond to any questions they may have while cooking, and receive support tailored to their emotional state, allowing them to cook safely and effectively.

[0730] "User" refers to an individual who uses the system to cook food.

[0731] "Device" refers to a computer terminal used by a user, such as a smartphone or tablet.

[0732] "Server" refers to a central remote computing system that performs processing such as data analysis and advice generation.

[0733] The "cooking process" refers to the series of operations and actions involved in preparing a dish.

[0734] "Means for taking pictures" refers to a method or device for recording images using the camera function of the device.

[0735] "Means of streaming in real time" refers to methods and technologies for transmitting captured video to a server without delay.

[0736] "Video data" refers to video information of the cooking process captured by a camera.

[0737] "Means of analysis" refers to the computational process that processes the video data and extracts important information such as the size of the ingredients, how well they are mixed, and how browned they are.

[0738] "Ingredient size" refers to the physical dimensions or volume of the ingredient being cooked.

[0739] "Mixing condition" refers to the state of how evenly the ingredients are mixed together.

[0740] "Brown" refers to the state in which ingredients change color when heated, particularly the degree of doneness.

[0741] "Advice" refers to specific cooking instructions or suggestions provided to users based on the analyzed information.

[0742] "Verbal means of questioning" refers to the method by which a user uses voice to express questions or instructions to the system.

[0743] "Means for converting to text" refers to speech recognition technology that converts speech into text data.

[0744] "Answer to a question" refers to an answer or explanation generated by the server in response to a user's question.

[0745] "Facial expressions" refer to emotions and states that can be read from the user's facial movements and muscle placement.

[0746] "Voice tone" refers to the pitch and tone of a user's voice when they speak, and is an element that expresses emotion.

[0747] "Means of emotion recognition" refers to technologies and algorithms that determine a user's emotional state from their facial expressions and voice.

[0748] "Emotional state" refers to the emotional state a user is feeling, such as joy, anger, sadness, or fear.

[0749] "Adjustment techniques" refer to methods and techniques for appropriately changing the content and tone of advice or responses depending on the person's emotional state.

[0750] "Data storage means" refers to a method or system that keeps a record of the cooking process for future reference or analysis.

[0751] The system of the present invention provides real-time support for cooking by combining visual information and voice communication. It also includes an emotion engine that recognizes the user's emotions. This system allows the user to receive appropriate advice in real time, providing more effective support during the cooking process.

[0752] First, the user launches the app installed on their smartphone, tablet, or other device. After entering their login information and successful authentication, the device displays a recipe selection screen. The user selects the recipe they want to make and confirms the ingredients and steps required.

[0753] When starting cooking, the user uses the device's camera to film the cooking process. This video is streamed in real time to a server. The server analyzes the received video data and extracts information such as the size of ingredients, the degree of mixing, and browning. This analysis uses image processing techniques (e.g., OpenCV) and generative AI models (e.g., TensorFlow).

[0754] Based on the analysis results, the server generates appropriate advice and notifies the user in voice and text format. For example, when frying onions, it provides specific instructions such as "You need to fry them a little longer."

[0755] Furthermore, if a user has a question while cooking, they can verbally ask the device. In this case, the device uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert the question into text and send it to the server. The server analyzes the question, generates an appropriate answer, and notifies the user by voice. For example, in response to the question, "What should I do next?", the device will respond with, "Next, add the chicken."

[0756] Furthermore, the system includes an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's facial expressions and vocal tone. This recognition uses computer vision and voice analysis technologies (e.g., Azure Cognitive Services). This allows the server to understand the user's emotional state and adjust the content and tone of its advice and responses accordingly. For example, if the user is feeling stressed, the server will provide gentle advice such as, "It's okay, try taking it a little slower."

[0757] Once cooking is complete, the user ends the session within the app and the server stores the cooking process data for future reference and analysis.

[0758] As a concrete example, consider a user making teriyaki chicken. The user launches the app, logs in, and selects the teriyaki chicken recipe. They prepare the ingredients and use their smartphone camera to film the cooking process. The server receives the video and provides specific advice in real time, such as "You need to fry the onions a little longer." If the user asks, "Is the chicken browned properly?", the server analyzes the video and returns an appropriate answer. Additionally, if the emotion engine recognizes that the user is stressed, it provides supportive advice such as "Take it slowly, it's fine."

[0759] Finally, when cooking is complete, the user ends the session and the cooking data is saved on the server. This system allows users to cook with peace of mind, receiving detailed support that also responds to emotional changes.

[0760] Example prompt sentence:

[0761] "Show me the chicken teriyaki recipe."

[0762] "Analyze the results of frying onions."

[0763] "What should I do next?"

[0764] "Is this the right browning for the chicken?"

[0765] "Provide advice on what to do if users are stressed."

[0766] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0767] Step 1:

[0768] App launch and authentication

[0769] Input: The user launches the app on their smartphone or tablet and enters their login information (user ID and password).

[0770] Output: A response from the server indicating authentication success or failure.

[0771] Specific behavior:

[0772] 1. The user launches the app.

[0773] 2. The login screen appears and the user enters their ID and password.

[0774] 3. The device sends the input information to the server.

[0775] 4. The server checks the database for authentication and returns the result to the terminal.

[0776] Step 2:

[0777] Recipe selection and display

[0778] Input: The recipe selected by the user on the recipe selection screen upon successful authentication.

[0779] Output: Ingredient list and instructions for the selected recipe.

[0780] Specific behavior:

[0781] 1. After successful authentication, the device displays the recipe selection screen.

[0782] 2. The user selects the recipe they want to make, and the device requests that information from the server.

[0783] 3. The server receives the request and sends the ingredients list and instructions for the corresponding recipe to the device.

[0784] 4. The terminal displays it on the screen.

[0785] Step 3:

[0786] Filming and streaming the cooking process

[0787] Input: User starts cooking, footage captured by camera.

[0788] Output: Real-time video data sent to the server.

[0789] Specific behavior:

[0790] 1. Users use a camera to record the cooking process in real time.

[0791] 2. The video captured by the device is streamed to the server in real time.

[0792] Step 4:

[0793] Video data analysis

[0794] Input: Streamed video data.

[0795] Output: Analysis results such as size of ingredients, mixing condition, browning, etc.

[0796] Specific behavior:

[0797] 1. The server analyzes the received video data.

[0798] 2. Image processing technology (e.g., OpenCV) and generative AI models (e.g., TensorFlow) are used to determine the size, mixing level, and browning of ingredients.

[0799] 3. The server processes the analysis results in real time.

[0800] Step 5:

[0801] Providing real-time advice

[0802] Input: Analysis results.

[0803] Output: Advice that will be sent to the user.

[0804] Specific behavior:

[0805] 1. The server generates advice based on the analysis results.

[0806] 2. Send advice in voice and text format to the device.

[0807] 3. The device notifies the user of the advice.

[0808] 4. For example, the advice "It needs to be cooked a little longer" will be displayed.

[0809] Step 6:

[0810] Audio questions and answers

[0811] Input: The user's spoken question.

[0812] Output: A suitable response from the server.

[0813] Specific behavior:

[0814] 1. The user has a question and verbally asks it to the device.

[0815] 2. The device uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert the question into text.

[0816] 3. Send the converted text to the server.

[0817] 4. The server analyzes the question, generates an appropriate answer, and sends it to the device.

[0818] 5. The device will notify the user of the answer via voice and text.

[0819] 6. For example, to the question "What should I do next?", the answer announced is "Next, add the chicken."

[0820] Step 7:

[0821] Emotion Recognition and Response

[0822] Input: The user's facial expression and voice tone.

[0823] Output: Advice or answer tailored by the server.

[0824] Specific behavior:

[0825] 1. The device uses a camera and microphone to record the user's facial expressions and voice tone.

[0826] 2. Analyze user emotions using emotion recognition technology (e.g., Azure Cognitive Services).

[0827] 3. Send the analysis results to the server.

[0828] 4. The server adjusts the content and tone of advice and responses based on the user's emotional state.

[0829] 5. For example, if the user is feeling stressed, the device will provide advice such as, "Take it easy, it's okay."

[0830] Step 8:

[0831] Ending a session and saving data

[0832] Input: Session termination information.

[0833] Output: Saved cooking process data.

[0834] Specific behavior:

[0835] 1. After the user finishes cooking, they end the session within the app.

[0836] 2. The terminal sends the session termination information to the server.

[0837] 3. The server stores the cooking process data in a database.

[0838] (Application example 2)

[0839] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0840] Driver stress and reduced attention in autonomous vehicles are a major issue, increasing the risk of traffic accidents. There is also a need to provide a safe driving environment where drivers can receive appropriate advice in real time while driving.

[0841] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to use the device to film the driving process, means for streaming the driving process to the server in real time, means for analyzing the video data received by the server and recognizing the user's emotions and attention state, and means for generating advice or warnings based on the recognized emotions and notifying the user of the advice or warnings. This makes it possible to provide detailed support according to the user's emotional state and improve safety and comfort while driving.

[0842] "User" means an individual or vehicle operator using the system.

[0843] "Devices" are hardware devices such as cameras and microphones installed in autonomous vehicles.

[0844] The "means for photographing the cooking process" is a function that uses a camera inside the self-driving vehicle to photograph the driving situation.

[0845] "Means for streaming to a server in real time" refers to a technology that transmits captured video to a server without delay.

[0846] "Means for analyzing video data and extracting information on the size, mixing level and browning of ingredients" refers to an algorithm that analyzes video data and recognizes the operating status.

[0847] The "means for generating advice based on the extracted information and notifying the user of the advice" is a technology for generating appropriate driving guidance or warnings based on the analysis results and providing them to the user.

[0848] "Means for users to ask questions verbally" refers to a function that allows the driver to ask questions verbally using a microphone.

[0849] The "means for converting an oral question into text and transmitting it to a server" is a technology that uses voice recognition technology to convert an oral question into text format and transmit it to a server.

[0850] "Means for generating answers to questions and notifying users" refers to a technology in which the server generates an appropriate answer based on the question received and notifies the user of the answer by voice or text.

[0851] The "means for recognizing emotions from a user's facial expression and voice tone" is a technology that performs facial expression analysis and voice analysis to recognize the user's emotional state.

[0852] "Means for adjusting the content and tone of advice or responses based on recognized emotions" refers to a technique for adjusting the content of advice or responses provided based on the recognized emotional state, and using a gentle tone or encouraging words as necessary.

[0853] "Means for saving cooking process data" refers to a function that saves the user's operating data on a server so that it can be referenced and analyzed later.

[0854] The present invention relates to a driving assistance system for autonomous vehicles, and aims to provide appropriate advice to users in real time while they are driving, thereby improving the safety and comfort of the driving environment.

[0855] The system consists of the following main components:

[0856] 1. User Device:

[0857] Hardware:

[0858] Camera: Used to record the inside of the vehicle.

[0859] Microphone: Used to capture the user's voice.

[0860] software:

[0861] Speech recognition: Software that converts speech into text.

[0862] Video capture: Software that captures camera footage and streams it to a server.

[0863] 2. Server:

[0864] Hardware:

[0865] A computer equipped with a high-performance processor and large memory capacity.

[0866] software:

[0867] Image analysis algorithm (OpenCV, etc.): Analyzes camera footage and recognizes the user's facial expressions and driving state.

[0868] Emotion recognition engine (deep learning models such as Keras): Recognizes user emotions from facial expressions and voice tone.

[0869] Auto-response generation system (e.g. gTTS): Generates appropriate advice and warnings and notifies them in audio format.

[0870] Speech recognition software (such as SpeechRecognition): Converts spoken questions into text.

[0871] 3. System Operation:

[0872] The user device uses a camera to capture video of the interior of the vehicle and streams it to a server in real time.

[0873] The server analyzes the received video data and recognizes the user's facial expressions and driving state.

[0874] Based on the recognized data, the server generates appropriate advice or warnings and notifies the user in voice format.

[0875] When a user speaks a question, a microphone captures it and speech recognition software converts the question into text and sends it to a server.

[0876] The server generates answers to the questions and communicates them to the user in audio format.

[0877] 4. Example:

[0878] For example, if a user is feeling stressed while driving, the system will recognize this from their facial expression and provide voice advice such as, "You seem a little stressed. Take a deep breath." If the user says, "It's dangerous," the system will warn, "Be careful! There is danger ahead."

[0879] 5. Example prompts:

[0880] "If the user is feeling stressed while driving, the system should recognize facial expressions from the camera footage and provide voice advice such as, 'You seem to be a little stressed. Take a deep breath.' If the user says, 'It's dangerous,' the system should warn them, 'Be careful! There is danger ahead.'"

[0881] This system provides detailed support tailored to the user's emotional state, improving safety and comfort while driving.

[0882] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0883] Step 1:

[0884] The user device uses a camera to capture images of the inside of the car. The input is real-time images of the inside of the car, and the output is camera image data. Specifically, the camera device captures the entire view of the inside of the car and the user's face, and generates this as streaming data.

[0885] Step 2:

[0886] The video data captured by the user device is streamed to the server in real time. The input is the camera video data, and the output is the streaming data to the server. Specifically, the video data is sent to the server via the network.

[0887] Step 3:

[0888] The server analyzes the received video data and recognizes the user's facial expressions and driving state. The input is streaming video data, and the output is data on the recognized facial expressions and driving state. Specifically, the video is analyzed frame by frame using OpenCV's image analysis algorithm, and features are extracted for face detection and emotion recognition.

[0889] Step 4:

[0890] The server uses an emotion recognition engine to recognize the user's emotions from facial expressions and voice tone. The input is the analyzed video data and the user's voice data, and the output is the recognized emotional state. Specifically, a deep learning model using Keras classifies emotions such as joy, anger, and sadness from facial expressions and analyzes voice tone.

[0891] Step 5:

[0892] The server generates appropriate advice or warnings based on the recognized data. The input is the recognized emotional state data, and the output is text data of the advice or warning. Specifically, a pre-defined advice or warning message is selected based on the generated emotional data.

[0893] Step 6:

[0894] The server notifies the user of the generated advice or warning in audio format. The input is the text data of the generated advice or warning, and the output is the audio notification data. Specifically, the server uses gTTS to convert the text data into an audio file and plays it back to the user through the speaker.

[0895] Step 7:

[0896] The user asks a question by voice and the device captures it. The input is the user's voice data, and the output is the captured voice data. Specifically, the microphone device records what the user says.

[0897] Step 8:

[0898] The device converts the voice question into text and sends it to the server. The input is the captured voice data, and the output is the text question data. Specifically, SpeechRecognition software is used to convert the voice into text data and send it to the server.

[0899] Step 9:

[0900] The server generates an answer to the question and notifies the user. The input is text question data, and the output is audio answer data. Specifically, an appropriate answer is generated based on the question content, converted into audio using gTTS, and played back to the user through the speaker.

[0901] Step 10:

[0902] The server stores data on the driving process. The input is all data collected during driving, and the output is the stored data. Specifically, the server stores the user's facial expression data, voice data, advice history, etc. in the database.

[0903] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0904] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0905] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0906] [Third embodiment]

[0907] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0908] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0909] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0910] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0911] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0912] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0913] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0914] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0915] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0916] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0917] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0918] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0919] The system of the present invention provides real-time support for cooking by combining visual information and voice communication. The system allows the user to use a device to film the cooking process and transmit the video in real time to a server, which can then provide appropriate advice on cooking methods and progress.

[0920] The detailed operation of the system is described below.

[0921] First, the user launches the app installed on their smartphone, tablet, or other device. After entering their login information and successful authentication, the device displays a recipe selection screen. The user selects the recipe they want to make and confirms the ingredients and steps required.

[0922] When the user starts cooking, the device's camera captures video. This video is streamed to the server in real time. The server analyzes the received video data and extracts information such as the size of the ingredients, how well they are mixed, and how browned they are. This allows the server to determine whether the ingredients are cut to the right size, mixed thoroughly, and cooked to the right degree.

[0923] The server generates advice based on the analysis results and notifies the user in voice and text format. For example, when frying onions, it may provide specific instructions such as "You need to fry them a little longer." This advice helps the user understand what steps to take next.

[0924] Furthermore, if a user has a question while cooking, they can verbally ask the device. The device uses voice recognition technology to convert the question into text and send it to the server. The server analyzes the question, generates an appropriate answer, and notifies the user by voice. For example, in response to a question such as "What should I do next?", the device will respond with "Next, add the chicken."

[0925] Once cooking is complete, the user ends the session within the app and the server stores the cooking process data for future reference and analysis.

[0926] As a concrete example, consider a user making teriyaki chicken. The user launches the app, logs in, and selects the teriyaki chicken recipe. They prepare the ingredients and use their smartphone camera to film the cooking process. The server receives the video and provides specific advice in real time, such as "You need to fry the onions a little longer." If the user also asks, "Is the chicken browned to the desired degree?" the server analyzes the video and returns an appropriate answer. Finally, when cooking is complete, the user ends the session, and the cooking data is saved to the server.

[0927] As described above, the system of the present invention combines real-time video analysis and voice communication to provide detailed and immediate support to users, allowing even beginners and users who are unsure about certain dishes to cook with confidence.

[0928] The processing flow will be explained below.

[0929] Step 1:

[0930] A user launches an app installed on a device such as a smartphone or tablet, and the device displays a login screen prompting the user to enter their username and password.

[0931] Step 2:

[0932] The user enters their username and password and presses the login button. The device temporarily stores the entered information and prepares to send it to the server. The device then sends an authentication request to the server.

[0933] Step 3:

[0934] The server compares the received user information with the records in its database and generates an authentication result. If the authentication is successful, the server returns an authentication success message to the terminal.

[0935] Step 4:

[0936] The device receives the authentication result and displays success or failure to the user. If authentication is successful, the recipe selection screen is displayed.

[0937] Step 5:

[0938] The user selects the desired recipe from the list and enters the selected recipe ID into the terminal, which temporarily stores the selected recipe ID and prepares to send it to the server.

[0939] Step 6:

[0940] The terminal sends a request for the selected recipe to the server. The server receives the request and retrieves detailed data for the selected recipe from the database. The server then sends the retrieved recipe data to the terminal.

[0941] Step 7:

[0942] The device displays the received recipe data and tells the user the ingredients and steps required, allowing the user to prepare the ingredients and begin cooking.

[0943] Step 8:

[0944] The user initiates the cooking process and captures the video using the device's camera. The device starts recording in real time and streams the video data to the server.

[0945] Step 9:

[0946] The server receives the video data and temporarily stores it for analysis. The server then uses image analysis algorithms to extract information such as the size of the ingredients, how well they are mixed, and how browned they are.

[0947] Step 10:

[0948] The server generates appropriate advice based on the analysis results, and the advice is sent to the device in voice and text format.

[0949] Step 11:

[0950] The device will display and voice the received advice to the user, who will then proceed with the next cooking step based on the advice.

[0951] Step 12:

[0952] If a user has a question while cooking, they can ask it verbally to the device, which will then use voice recognition technology to convert the question into text and send it to the server.

[0953] Step 13:

[0954] The server analyzes the textual content of the question and generates an appropriate answer, which is then sent to the device in voice and text format.

[0955] Step 14:

[0956] The device displays the received answers to the user and notifies them by voice, allowing the user to proceed with cooking while having their questions resolved.

[0957] Step 15:

[0958] When the user finishes cooking, they press the end session button in the app, and the device sends a session end request to the server.

[0959] Step 16:

[0960] The server receives the session data, saves it in the database, and notifies the terminal that the save is complete.

[0961] Step 17:

[0962] The device will display and sound a message to the user that cooking is complete, and the entire cooking process will be completed.

[0963] Example 1

[0964] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0965] The lack of a system that allows users to receive appropriate advice and guidance in real time while cooking poses a challenge, making it difficult for beginners and users who are unsure about certain dishes to cook with confidence. Furthermore, conventional cooking support systems are unable to respond immediately to users' questions, making it difficult to provide appropriate support tailored to the user's actual cooking progress.

[0966] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0967] In this invention, the server includes means for a user to use a device to film the cooking process, means for streaming the cooking process to the server in real time, means for the server to analyze the video data received and extract information on the size and mixing of ingredients and browning, means for generating advice based on the extracted information and notifying the user of the advice in audio and text format, means for the user to verbally ask a question, means for converting the verbal question into text and sending it to the server, means for the server to generate an answer to the question and notify the user in audio and text format, and means for saving the cooking process data. This allows users to receive appropriate cooking guidance and advice in real time, allowing even beginners and users who are unsure about certain dishes to cook with confidence.

[0968] A "user" is a person who uses the system to assist with the cooking process.

[0969] "Device" refers to a mobile device such as a smartphone or tablet used by a user.

[0970] The "cooking process" refers to the series of steps involved in preparing ingredients, cutting, mixing, heating, etc., to complete a dish.

[0971] "Streaming" is a technology that transmits captured video data to a server in real time.

[0972] The "server" is a central processing unit that receives and analyzes the video data sent from the user's device and generates corresponding advice and answers.

[0973] "Video data" refers to images and videos of the cooking process captured by the device's camera.

[0974] "Analysis" refers to the process of evaluating the size of ingredients, how well mixed they are, and how browned they are based on the received video data.

[0975] "Ingredients" refer to the individual ingredients that make up a dish.

[0976] "Advice" refers to cooking instructions and advice that are generated based on the analysis results and provided to the user.

[0977] "Audio format" refers to a format in which advice or answers are notified to the user as audio.

[0978] "Text format" refers to a format in which advice or answers are notified to the user in written form.

[0979] "Questions" are questions or problems that users verbally input while cooking.

[0980] "Voice recognition technology" refers to technology that converts a user's verbal questions into text.

[0981] An "answer" is a response generated by a server in response to a question from a user.

[0982] "Data storage" refers to the process of storing data about the cooking process on a server for future reference.

[0983] "Real-time" refers to a time concept in which analysis and notification are carried out immediately as the cooking process progresses.

[0984] "Computer vision technology" is a general term for image recognition technology used to analyze video data.

[0985] The system of the present invention provides real-time support for cooking by combining visual information and voice communication. A specific embodiment of the system will now be described.

[0986] First, the user launches a dedicated app installed on a mobile device such as a smartphone or tablet. After the user enters their login information and is successfully authenticated, the device displays a recipe selection screen. The user selects the recipe they want to make and checks the ingredients and steps. The software used by the device at this stage generally includes the operating system (e.g., iOS, Android) and application software for the smartphone or tablet.

[0987] Next, when the user starts cooking, the device's camera is used to capture video. This video is then streamed to the server in real time using technologies such as the Real-Time Messaging Protocol (RTMP) and HTTP Live Streaming (HLS). This allows the video data to be sent to the server in real time.

[0988] The server uses computer vision technology (e.g., OpenCV or TensorFlow) to analyze the received video data. The server extracts information such as the size of the ingredients, the degree of mixing, and the browning of the ingredients, and performs analysis based on this. Based on the analysis results, the server generates appropriate advice and sends it to the device. The advice is provided in audio (e.g., Google Cloud Text-to-Speech API) and text format. The device notifies the user of the advice it received.

[0989] If a user has a question while cooking, they can verbally ask the device. The device will convert the question into text using voice recognition technology (e.g., Google Cloud Speech-to-Text API) and send it to the server. The server will analyze the question, generate an appropriate answer, and notify the user in voice and text format.

[0990] Once cooking is complete, the user can end the session within the app. The server then stores the cooking process data for future reference and analysis. This data includes analysis results, user questions and answers, and any advice provided.

[0991] As a concrete example, consider a user making teriyaki chicken. The user launches the app, logs in, and selects the teriyaki chicken recipe. They prepare the ingredients and use their smartphone camera to film the cooking process. The server receives the video and provides specific advice in real time, such as "You need to fry the onions a little longer." When the user asks, "Is the chicken browned to the desired degree?" the server analyzes the video and returns an appropriate answer. Finally, when cooking is complete, the user ends the session, and the cooking data is saved to the server.

[0992] Examples of prompts include:

[0993] "I'm making chicken teriyaki. I'm currently frying onions. Can you tell me the proper doneness?"

[0994] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0995] Step 1: Launching the program and authenticating the user

[0996] Users launch a dedicated app on their smartphone or tablet and enter their login information (username and password).

[0997] The terminal sends the entered login information to the server (input: user name, password).

[0998] The server compares the received login information with the authentication database and performs authentication (data processing: comparison with authentication database, output: authentication result).

[0999] If the authentication is successful, the server starts a session and displays the recipe selection screen on the terminal (output: session ID, recipe selection screen).

[1000] Step 2: Select a recipe and check ingredients

[1001] The user selects the recipe they want to make on the device screen (input: select recipe).

[1002] The terminal transmits the selected recipe information to the server (input: recipe information).

[1003] The server sends the ingredients and cooking instructions required for the selected recipe to the terminal (data processing: recipe data extraction, output: ingredient list, procedure list).

[1004] The terminal displays it on the screen, and the user checks the ingredients and proceeds with preparation (output: ingredient list, procedure list).

[1005] Step 3: Record and send footage

[1006] The user starts cooking and takes pictures using the device's camera.

[1007] The terminal streams the captured video data to the server in real time (input: video data).

[1008] The terminal transmits video data using the Real-Time Messaging Protocol (RTMP) or HTTP Live Streaming (HLS) (output: real-time video data).

[1009] Step 4: Video analysis on the server

[1010] The server uses computer vision techniques (e.g., OpenCV or TensorFlow) to analyze the received video data (input: video data).

[1011] The server extracts information such as the size of the ingredients, how well mixed they are, and how browned they are (data processing: video analysis, output: ingredient information).

[1012] The cooking progress is evaluated based on the analysis results (data calculation: cooking progress evaluation, output: progress evaluation results).

[1013] Step 5: Server generates and notifies advice

[1014] The server generates specific advice based on the analysis results (data processing: advice generation, output: advice).

[1015] The server sends the generated advice to the terminal in voice (e.g., Text-to-Speech API) and text format (input: advice, output: voice data, text data).

[1016] The device notifies the user of the received advice (specific actions: playing audio, displaying text).

[1017] Step 6: Respond to user questions

[1018] If the user has a question while cooking, they can ask verbally into the terminal (input: voice question).

[1019] The device uses voice recognition technology (e.g., Speech-to-Text API) to convert the question into text and send it to the server (data processing: voice recognition, output: text question).

[1020] The server analyzes the question and generates an appropriate answer (data processing: question analysis, output: answer).

[1021] The server sends the generated answer to the terminal in voice and text format (input: answer, output: voice data, text data).

[1022] The terminal notifies the user of the answer by voice and text (specific operations: playing voice, displaying text).

[1023] Step 7: Save the data after cooking is complete

[1024] When the user is done cooking, they can end the session within the app (input: end session).

[1025] The server stores the data of the cooking process (data processing: data storage, output: stored data).

[1026] The saved data is used for future reference and analysis (specific operation: saving to a database).

[1027] (Application example 1)

[1028] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1029] With current food delivery services, it is difficult to control the quality of food during delivery, making it difficult for delivery personnel to efficiently deliver food while checking its condition. Furthermore, there is a lack of means to receive real-time feedback on the state of the food, making it a challenge to improve delivery quality.

[1030] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1031] In this invention, the server includes: means for a user and a delivery person to use a device to film the cooking process and the state of the food; means for streaming the cooking process and the state of the food to the server in real time; means for the server to analyze the received video data and evaluate information on the size and mixture of ingredients and browning and the quality of the food; means for generating advice based on the extracted information and evaluation results and notifying the user and the delivery person of the advice; means for the user and the delivery person to ask verbal questions; means for converting the verbal questions into text and sending it to the server; means for the server to generate answers to the questions and notify the user and the delivery person; and means for storing the cooking process data and food state data. This enables real-time quality control of the cooking process and food during delivery, enabling the delivery person to efficiently deliver high-quality food.

[1032] "User" means any person who uses the system to receive real-time assistance and advice during the cooking process.

[1033] "Delivery person" means a person who performs quality control during the delivery of food in a food delivery service.

[1034] The "device" is a hard device that has a camera function and transmits the actions of the user or delivery person to the server in real time.

[1035] A "server" is a computing device that analyzes received video data and generates advice.

[1036] "Video data" refers to visual information captured by the user or delivery person showing the cooking process and the condition of the food.

[1037] "Analysis" is the process of evaluating the size of ingredients, the degree of mixing, the browning, and the quality of the food based on the received video data.

[1038] "Advice" refers to instructions and advice regarding cooking and quality control provided based on the analysis results.

[1039] "Streaming" is a technology for transmitting video data to a server in real time.

[1040] "Questions" are questions or clarifications about the cooking process or the condition of the food that users or delivery personnel ask the system.

[1041] "Text conversion" is the process of converting spoken questions into written information.

[1042] "Notification" is the act of transmitting advice or answers generated by the server to the user or delivery person.

[1043] "Data archiving" is the process of recording information about analysis results and recommendations for future reference.

[1044] "Quality evaluation" is the process of determining the quality of food based on its condition.

[1045] "Audio format" refers to a means of providing advice or answers to users or delivery personnel via voice.

[1046] An "image analysis algorithm" is a mathematical technique used to analyze video data.

[1047] "Food condition" refers to information such as the temperature, appearance, and packaging of the food during delivery.

[1048] The system of the present invention allows users and delivery staff to use devices to film the cooking process and the condition of the food, and sends the footage to a server in real time, thereby providing appropriate advice on cooking methods and progress and managing food quality.

[1049] System Configuration

[1050] 1. Hardware

[1051] Camera-equipped devices such as smartphones, tablets, and smart glasses

[1052] Computing Server

[1053] 2. Software

[1054] Image Analysis Algorithm

[1055] Speech Recognition and Synthesis Software

[1056] Streaming Server

[1057] Data processing and calculation

[1058] Video data collection and streaming

[1059] Users and delivery staff use devices (smartphones, tablets, smart glasses) to record the cooking process and the state of the food, and this video data is sent to a streaming server in real time.

[1060] Video data analysis

[1061] The streaming server analyzes the received video data, using image analysis algorithms to assess the size, consistency, browning, and food quality (temperature, appearance, packaging, etc.) of ingredients.

[1062] Advice generation and notification

[1063] Based on the analysis results, the server generates advice on the cooking progress and food quality, which is communicated to the user and delivery person in voice and text format.

[1064] Questions and Answers

[1065] When a user or delivery person verbally asks a question, the device converts the voice into text and sends it to the server, which analyzes the question, generates an appropriate answer, and notifies the user in voice and text format.

[1066] Data storage

[1067] Analysis results, advice, and question and answer data are stored by the server for future reference and service improvement.

[1068] Specific examples

[1069] Ramen delivery

[1070] A delivery person wears smart glasses and takes a video to check the condition of the ramen during delivery. The streaming server analyzes the video and provides advice such as "The soup has not spilled" and "The ramen ingredients have not fallen apart." If the delivery person asks, "Please check if the ramen ingredients have fallen apart," the voice is converted into text, and the server analyzes the question and provides an appropriate answer.

[1071] Prompt Sentence Examples

[1072] "Please determine whether the ramen ingredients have fallen apart."

[1073] In this way, the system of the present invention controls the quality of food in real time during the cooking process and delivery, allowing delivery personnel to efficiently deliver high-quality food.

[1074] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1075] Step 1:

[1076] Users and delivery personnel activate devices (smartphones, tablets, smart glasses) to capture real-time images of the cooking process or the state of the food.

[1077] Input: Image from the device's camera.

[1078] Output: Real-time video data.

[1079] Specific operation: The user or delivery person points the camera at the target object (food or dish) and presses the capture button. The device acquires the captured data.

[1080] Step 2:

[1081] The video data captured by the terminal is sent to a streaming server.

[1082] Input: Real-time video data.

[1083] Output: Streamed video data.

[1084] Specific operation: Using the device's communication function, video data is uploaded to a server via the Internet in real time. The data is encrypted and transmitted securely.

[1085] Step 3:

[1086] The server analyzes the video data it receives and evaluates the size of the ingredients, how well mixed they are, how browned they are, and the quality of the food (temperature, appearance, packaging, etc.).

[1087] Input: Streamed video data.

[1088] Output: Analysis results (ingredients size, mixing condition, browning, quality evaluation, etc.).

[1089] Specific operation: The image analysis algorithm is executed on the server to analyze the received video data, extracting and evaluating parameters such as the size of ingredients, the degree of mixing, and browning.

[1090] Step 4:

[1091] The server generates advice based on the analysis results and notifies the user and delivery person.

[1092] Input: Analysis results.

[1093] Output: Advice (audio and text format).

[1094] Specific operation: The server generates appropriate advice based on the analysis results and notifies the user and delivery person using speech synthesis and text generation technologies. The advice is sent via a dedicated application.

[1095] Step 5:

[1096] The user or delivery person asks a question verbally, and the device converts the speech into text and sends it to the server.

[1097] Input: Audio question.

[1098] Output: The question in text format.

[1099] Specific operation: The user or delivery person speaks a question into the terminal, and the terminal uses voice recognition technology to convert the speech into text and send it to the server.

[1100] Step 6:

[1101] The server generates an answer to the received question and notifies the user and the delivery person.

[1102] Input: A text question.

[1103] Output: Answers (in audio and text format).

[1104] Specific operation: The server analyzes the received question and generates an appropriate answer using a generative AI model. The generated answer is then notified to the user and delivery person in voice synthesis and text format.

[1105] Step 7:

[1106] The server stores the analysis results, advice, questions and answers data for future reference and to help improve our services.

[1107] Input: Analysis results, advice, questions, answers.

[1108] Output: Saved data.

[1109] Specific operation: The server uses a database system to permanently store analysis results, advice, questions, and answer data. The stored data can be referenced at a later date and used to improve the service.

[1110] These steps allow for real-time quality control of food during the cooking process and delivery, enabling delivery personnel to efficiently deliver high-quality food.

[1111] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1112] The system of the present invention provides real-time support for cooking by combining visual information and voice communication, and also includes an emotion engine that recognizes the user's emotions. This system allows the user to receive appropriate advice in real time and obtain more effective support during the cooking process.

[1113] The detailed operation of the system is described below.

[1114] First, the user launches the app installed on their smartphone, tablet, or other device. After entering their login information and successful authentication, the device displays a recipe selection screen. The user selects the recipe they want to make and confirms the ingredients and steps required.

[1115] When starting cooking, the user uses the device's camera to film the cooking process. This video is streamed to the server in real time. The server analyzes the received video data and extracts information such as the size of the ingredients, how well they are mixed, and how browned they are. Based on the results of this analysis, the server generates appropriate advice and notifies the user in voice and text format. For example, when frying onions, the server may provide specific instructions such as "You need to fry them a little longer."

[1116] Furthermore, if a user has a question while cooking, they can verbally ask the device. The device uses voice recognition technology to convert the question into text and send it to the server. The server analyzes the question, generates an appropriate answer, and notifies the user by voice. For example, in response to a question such as "What should I do next?", the device will respond with "Next, add the chicken."

[1117] The system includes an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's facial expressions and vocal tone. This allows the server to understand the user's emotional state and adjust the content and tone of advice and responses accordingly. For example, if the user is feeling stressed, the server will provide gentle advice such as, "It's okay, try taking it a little slower."

[1118] Once cooking is complete, the user ends the session within the app and the server stores the cooking process data for future reference and analysis.

[1119] As a concrete example, consider a user making teriyaki chicken. The user launches the app, logs in, and selects the teriyaki chicken recipe. They prepare the ingredients and use their smartphone camera to film the cooking process. The server receives the video and provides specific advice in real time, such as "You need to fry the onions a little longer." Furthermore, if the user asks, "Is the chicken browned to the desired degree?" the server analyzes the video and returns an appropriate answer. If the emotion engine recognizes that the user is stressed, it provides supportive advice, such as "Take it slowly, it's fine."

[1120] Finally, when cooking is complete, the user ends the session and the cooking data is saved on the server. This system allows users to cook with peace of mind, receiving detailed support that also responds to emotional changes.

[1121] The processing flow will be explained below.

[1122] Step 1:

[1123] A user launches an app installed on a device such as a smartphone or tablet, and the device displays a login screen prompting the user to enter their username and password.

[1124] Step 2:

[1125] The user enters their username and password and presses the login button. The device temporarily stores the entered information and prepares to send it to the server. The device then sends an authentication request to the server.

[1126] Step 3:

[1127] The server compares the received user information with the records in its database and generates an authentication result. If the authentication is successful, the server returns an authentication success message to the terminal.

[1128] Step 4:

[1129] The device receives the authentication result and displays success or failure to the user. If authentication is successful, the recipe selection screen is displayed.

[1130] Step 5:

[1131] The user selects the desired recipe from the list and enters the selected recipe ID into the terminal, which temporarily stores the selected recipe ID and prepares to send it to the server.

[1132] Step 6:

[1133] The terminal sends a request for the selected recipe to the server. The server receives the request and retrieves detailed data for the selected recipe from the database. The server then sends the retrieved recipe data to the terminal.

[1134] Step 7:

[1135] The device displays the received recipe data and tells the user the ingredients and steps required, allowing the user to prepare the ingredients and begin cooking.

[1136] Step 8:

[1137] The user initiates the cooking process and captures the video using the device's camera. The device starts recording in real time and streams the video data to the server.

[1138] Step 9:

[1139] The server receives the video data and temporarily stores it for analysis. Using an image analysis algorithm, the server extracts information such as the size of the ingredients, how well they are mixed, and how browned they are, allowing it to determine the appropriate cooking status.

[1140] Step 10:

[1141] The server generates appropriate advice based on the analysis results, and the advice is sent to the device in voice and text format.

[1142] Step 11:

[1143] The device will display and voice the received advice to the user, who will then proceed with the next cooking step based on the advice.

[1144] Step 12:

[1145] If a user has a question while cooking, they can ask it verbally to the device, which will then use voice recognition technology to convert the question into text and send it to the server.

[1146] Step 13:

[1147] The server analyzes the textual content of the question and generates an appropriate answer, which is then sent to the device in voice and text format.

[1148] Step 14:

[1149] The device displays the received answers to the user and notifies them by voice, allowing the user to proceed with cooking while having their questions resolved.

[1150] Step 15:

[1151] By analyzing the user's facial expressions and voice tone from the camera footage, the device uses an emotion engine to recognize the user's emotional state, determining whether the user is stressed or relaxed.

[1152] Step 16:

[1153] The server adjusts the tone and content of its advice and responses based on the user's perceived emotional state. For example, if the user is feeling stressed, the server will provide considerate advice in a gentle tone, such as "Try taking it a little slower."

[1154] Step 17:

[1155] When the user finishes cooking, they press the end session button in the app, and the device sends a session end request to the server.

[1156] Step 18:

[1157] The server receives the session data, saves it in the database, and notifies the terminal that the save is complete.

[1158] Step 19:

[1159] The device will display and sound a message to the user that cooking is complete, and the entire cooking process will be completed.

[1160] Through the above process, the system of the present invention provides users with real-time cooking support through visual information and voice communication, and furthermore, by recognizing the user's emotional state using an emotion engine, it realizes detailed support.

[1161] Example 2

[1162] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1163] Conventional cooking assistance systems make it difficult for users to receive appropriate advice during the cooking process. They also lack the ability to respond to the user's emotional state, making it difficult to provide sufficient support to users who are feeling stressed or anxious. This increases the likelihood of inconvenience and mistakes during cooking, potentially resulting in a poor user experience.

[1164] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1165] In this invention, the server includes: means for a user to use a device to film the cooking process; means for streaming the cooking process to the server in real time; means for the server to analyze the received video data and extract information such as the size and mixing condition of ingredients and browning; means for generating advice based on the extracted information and notifying the user of the advice; means for the user to verbally ask a question; means for converting the verbal question into text and sending it to the server; means for the server to generate an answer to the question and notify the user; means for recognizing emotions from the user's facial expression and tone of voice; means for adjusting the content of the advice and answer based on the user's emotional state; and means for saving the cooking process data. This allows the user to receive appropriate advice in real time, quickly respond to any questions they may have while cooking, and receive support tailored to their emotional state, allowing them to cook safely and effectively.

[1166] "User" refers to an individual who uses the system to cook food.

[1167] "Device" refers to a computer terminal used by a user, such as a smartphone or tablet.

[1168] "Server" refers to a central remote computing system that performs processing such as data analysis and advice generation.

[1169] The "cooking process" refers to the series of operations and actions involved in preparing a dish.

[1170] "Means for taking pictures" refers to a method or device for recording images using the camera function of the device.

[1171] "Means of streaming in real time" refers to methods and technologies for transmitting captured video to a server without delay.

[1172] "Video data" refers to video information of the cooking process captured by a camera.

[1173] "Means of analysis" refers to the computational process that processes the video data and extracts important information such as the size of the ingredients, how well they are mixed, and how browned they are.

[1174] "Ingredient size" refers to the physical dimensions or volume of the ingredient being cooked.

[1175] "Mixing condition" refers to the state of how evenly the ingredients are mixed together.

[1176] "Brown" refers to the state in which ingredients change color when heated, particularly the degree of doneness.

[1177] "Advice" refers to specific cooking instructions or suggestions provided to users based on the analyzed information.

[1178] "Verbal means of questioning" refers to the method by which a user uses voice to express questions or instructions to the system.

[1179] "Means for converting to text" refers to speech recognition technology that converts speech into text data.

[1180] "Answer to a question" refers to an answer or explanation generated by the server in response to a user's question.

[1181] "Facial expressions" refer to emotions and states that can be read from the user's facial movements and muscle placement.

[1182] "Voice tone" refers to the pitch and tone of a user's voice when they speak, and is an element that expresses emotion.

[1183] "Means of emotion recognition" refers to technologies and algorithms that determine a user's emotional state from their facial expressions and voice.

[1184] "Emotional state" refers to the emotional state a user is feeling, such as joy, anger, sadness, or fear.

[1185] "Adjustment techniques" refer to methods and techniques for appropriately changing the content and tone of advice or responses depending on the person's emotional state.

[1186] "Data storage means" refers to a method or system that keeps a record of the cooking process for future reference or analysis.

[1187] The system of the present invention provides real-time support for cooking by combining visual information and voice communication. It also includes an emotion engine that recognizes the user's emotions. This system allows the user to receive appropriate advice in real time, providing more effective support during the cooking process.

[1188] First, the user launches the app installed on their smartphone, tablet, or other device. After entering their login information and successful authentication, the device displays a recipe selection screen. The user selects the recipe they want to make and confirms the ingredients and steps required.

[1189] When starting cooking, the user uses the device's camera to film the cooking process. This video is streamed in real time to a server. The server analyzes the received video data and extracts information such as the size of ingredients, the degree of mixing, and browning. This analysis uses image processing techniques (e.g., OpenCV) and generative AI models (e.g., TensorFlow).

[1190] Based on the analysis results, the server generates appropriate advice and notifies the user in voice and text format. For example, when frying onions, it provides specific instructions such as "You need to fry them a little longer."

[1191] Furthermore, if a user has a question while cooking, they can verbally ask the device. In this case, the device uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert the question into text and send it to the server. The server analyzes the question, generates an appropriate answer, and notifies the user by voice. For example, in response to the question, "What should I do next?", the device will respond with, "Next, add the chicken."

[1192] Furthermore, the system includes an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's facial expressions and vocal tone. This recognition uses computer vision and voice analysis technologies (e.g., Azure Cognitive Services). This allows the server to understand the user's emotional state and adjust the content and tone of its advice and responses accordingly. For example, if the user is feeling stressed, the server will provide gentle advice such as, "It's okay, try taking it a little slower."

[1193] Once cooking is complete, the user ends the session within the app and the server stores the cooking process data for future reference and analysis.

[1194] As a concrete example, consider a user making teriyaki chicken. The user launches the app, logs in, and selects the teriyaki chicken recipe. They prepare the ingredients and use their smartphone camera to film the cooking process. The server receives the video and provides specific advice in real time, such as "You need to fry the onions a little longer." If the user asks, "Is the chicken browned properly?", the server analyzes the video and returns an appropriate answer. Additionally, if the emotion engine recognizes that the user is stressed, it provides supportive advice such as "Take it slowly, it's fine."

[1195] Finally, when cooking is complete, the user ends the session and the cooking data is saved on the server. This system allows users to cook with peace of mind, receiving detailed support that also responds to emotional changes.

[1196] Example prompt sentence:

[1197] "Show me the chicken teriyaki recipe."

[1198] "Analyze the results of frying onions."

[1199] "What should I do next?"

[1200] "Is this the right browning for the chicken?"

[1201] "Provide advice on what to do if users are stressed."

[1202] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1203] Step 1:

[1204] App launch and authentication

[1205] Input: The user launches the app on their smartphone or tablet and enters their login information (user ID and password).

[1206] Output: A response from the server indicating authentication success or failure.

[1207] Specific behavior:

[1208] 1. The user launches the app.

[1209] 2. The login screen appears and the user enters their ID and password.

[1210] 3. The device sends the input information to the server.

[1211] 4. The server checks the database for authentication and returns the result to the terminal.

[1212] Step 2:

[1213] Recipe selection and display

[1214] Input: The recipe selected by the user on the recipe selection screen upon successful authentication.

[1215] Output: Ingredient list and instructions for the selected recipe.

[1216] Specific behavior:

[1217] 1. After successful authentication, the device displays the recipe selection screen.

[1218] 2. The user selects the recipe they want to make, and the device requests that information from the server.

[1219] 3. The server receives the request and sends the ingredients list and instructions for the corresponding recipe to the device.

[1220] 4. The terminal displays it on the screen.

[1221] Step 3:

[1222] Filming and streaming the cooking process

[1223] Input: User starts cooking, footage captured by camera.

[1224] Output: Real-time video data sent to the server.

[1225] Specific behavior:

[1226] 1. Users use a camera to record the cooking process in real time.

[1227] 2. The video captured by the device is streamed to the server in real time.

[1228] Step 4:

[1229] Video data analysis

[1230] Input: Streamed video data.

[1231] Output: Analysis results such as size of ingredients, mixing condition, browning, etc.

[1232] Specific behavior:

[1233] 1. The server analyzes the received video data.

[1234] 2. Image processing technology (e.g., OpenCV) and generative AI models (e.g., TensorFlow) are used to determine the size, mixing level, and browning of ingredients.

[1235] 3. The server processes the analysis results in real time.

[1236] Step 5:

[1237] Providing real-time advice

[1238] Input: Analysis results.

[1239] Output: Advice that will be sent to the user.

[1240] Specific behavior:

[1241] 1. The server generates advice based on the analysis results.

[1242] 2. Send advice in voice and text format to the device.

[1243] 3. The device notifies the user of the advice.

[1244] 4. For example, the advice "It needs to be cooked a little longer" will be displayed.

[1245] Step 6:

[1246] Audio questions and answers

[1247] Input: The user's spoken question.

[1248] Output: A suitable response from the server.

[1249] Specific behavior:

[1250] 1. The user has a question and verbally asks it to the device.

[1251] 2. The device uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert the question into text.

[1252] 3. Send the converted text to the server.

[1253] 4. The server analyzes the question, generates an appropriate answer, and sends it to the device.

[1254] 5. The device will notify the user of the answer via voice and text.

[1255] 6. For example, to the question "What should I do next?", the answer announced is "Next, add the chicken."

[1256] Step 7:

[1257] Emotion Recognition and Response

[1258] Input: The user's facial expression and voice tone.

[1259] Output: Advice or answer tailored by the server.

[1260] Specific behavior:

[1261] 1. The device uses a camera and microphone to record the user's facial expressions and voice tone.

[1262] 2. Analyze user emotions using emotion recognition technology (e.g., Azure Cognitive Services).

[1263] 3. Send the analysis results to the server.

[1264] 4. The server adjusts the content and tone of advice and responses based on the user's emotional state.

[1265] 5. For example, if the user is feeling stressed, the device will provide advice such as, "Take it easy, it's okay."

[1266] Step 8:

[1267] Ending a session and saving data

[1268] Input: Session termination information.

[1269] Output: Saved cooking process data.

[1270] Specific behavior:

[1271] 1. After the user finishes cooking, they end the session within the app.

[1272] 2. The terminal sends the session termination information to the server.

[1273] 3. The server stores the cooking process data in a database.

[1274] (Application example 2)

[1275] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1276] Driver stress and reduced attention in autonomous vehicles are a major issue, increasing the risk of traffic accidents. There is also a need to provide a safe driving environment where drivers can receive appropriate advice in real time while driving.

[1277] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to use the device to film the driving process, means for streaming the driving process to the server in real time, means for analyzing the video data received by the server and recognizing the user's emotions and attention state, and means for generating advice or warnings based on the recognized emotions and notifying the user of the advice or warnings. This makes it possible to provide detailed support according to the user's emotional state and improve safety and comfort while driving.

[1278] "User" means an individual or vehicle operator using the system.

[1279] "Devices" are hardware devices such as cameras and microphones installed in autonomous vehicles.

[1280] The "means for photographing the cooking process" is a function that uses a camera inside the self-driving vehicle to photograph the driving situation.

[1281] "Means for streaming to a server in real time" refers to a technology that transmits captured video to a server without delay.

[1282] "Means for analyzing video data and extracting information on the size, mixing level and browning of ingredients" refers to an algorithm that analyzes video data and recognizes the operating status.

[1283] The "means for generating advice based on the extracted information and notifying the user of the advice" is a technology for generating appropriate driving guidance or warnings based on the analysis results and providing them to the user.

[1284] "Means for users to ask questions verbally" refers to a function that allows the driver to ask questions verbally using a microphone.

[1285] The "means for converting an oral question into text and transmitting it to a server" is a technology that uses voice recognition technology to convert an oral question into text format and transmit it to a server.

[1286] "Means for generating answers to questions and notifying users" refers to a technology in which the server generates an appropriate answer based on the question received and notifies the user of the answer by voice or text.

[1287] The "means for recognizing emotions from a user's facial expression and voice tone" is a technology that performs facial expression analysis and voice analysis to recognize the user's emotional state.

[1288] "Means for adjusting the content and tone of advice or responses based on recognized emotions" refers to a technique for adjusting the content of advice or responses provided based on the recognized emotional state, and using a gentle tone or encouraging words as necessary.

[1289] "Means for saving cooking process data" refers to a function that saves the user's operating data on a server so that it can be referenced and analyzed later.

[1290] The present invention relates to a driving assistance system for autonomous vehicles, and aims to provide appropriate advice to users in real time while they are driving, thereby improving the safety and comfort of the driving environment.

[1291] The system consists of the following main components:

[1292] 1. User Device:

[1293] Hardware:

[1294] Camera: Used to record the inside of the vehicle.

[1295] Microphone: Used to capture the user's voice.

[1296] software:

[1297] Speech recognition: Software that converts speech into text.

[1298] Video capture: Software that captures camera footage and streams it to a server.

[1299] 2. Server:

[1300] Hardware:

[1301] A computer equipped with a high-performance processor and large memory capacity.

[1302] software:

[1303] Image analysis algorithm (OpenCV, etc.): Analyzes camera footage and recognizes the user's facial expressions and driving state.

[1304] Emotion recognition engine (deep learning models such as Keras): Recognizes user emotions from facial expressions and voice tone.

[1305] Auto-response generation system (e.g. gTTS): Generates appropriate advice and warnings and notifies them in audio format.

[1306] Speech recognition software (such as SpeechRecognition): Converts spoken questions into text.

[1307] 3. System Operation:

[1308] The user device uses a camera to capture video of the interior of the vehicle and streams it to a server in real time.

[1309] The server analyzes the received video data and recognizes the user's facial expressions and driving state.

[1310] Based on the recognized data, the server generates appropriate advice or warnings and notifies the user in voice format.

[1311] When a user speaks a question, a microphone captures it and speech recognition software converts the question into text and sends it to a server.

[1312] The server generates answers to the questions and communicates them to the user in audio format.

[1313] 4. Example:

[1314] For example, if a user is feeling stressed while driving, the system will recognize this from their facial expression and provide voice advice such as, "You seem a little stressed. Take a deep breath." If the user says, "It's dangerous," the system will warn, "Be careful! There is danger ahead."

[1315] 5. Example prompts:

[1316] "If the user is feeling stressed while driving, the system should recognize facial expressions from the camera footage and provide voice advice such as, 'You seem to be a little stressed. Take a deep breath.' If the user says, 'It's dangerous,' the system should warn them, 'Be careful! There is danger ahead.'"

[1317] This system provides detailed support tailored to the user's emotional state, improving safety and comfort while driving.

[1318] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1319] Step 1:

[1320] The user device uses a camera to capture images of the inside of the car. The input is real-time images of the inside of the car, and the output is camera image data. Specifically, the camera device captures the entire view of the inside of the car and the user's face, and generates this as streaming data.

[1321] Step 2:

[1322] The video data captured by the user device is streamed to the server in real time. The input is the camera video data, and the output is the streaming data to the server. Specifically, the video data is sent to the server via the network.

[1323] Step 3:

[1324] The server analyzes the received video data and recognizes the user's facial expressions and driving state. The input is streaming video data, and the output is data on the recognized facial expressions and driving state. Specifically, the video is analyzed frame by frame using OpenCV's image analysis algorithm, and features are extracted for face detection and emotion recognition.

[1325] Step 4:

[1326] The server uses an emotion recognition engine to recognize the user's emotions from facial expressions and voice tone. The input is the analyzed video data and the user's voice data, and the output is the recognized emotional state. Specifically, a deep learning model using Keras classifies emotions such as joy, anger, and sadness from facial expressions and analyzes voice tone.

[1327] Step 5:

[1328] The server generates appropriate advice or warnings based on the recognized data. The input is the recognized emotional state data, and the output is text data of the advice or warning. Specifically, a pre-defined advice or warning message is selected based on the generated emotional data.

[1329] Step 6:

[1330] The server notifies the user of the generated advice or warning in audio format. The input is the text data of the generated advice or warning, and the output is the audio notification data. Specifically, the server uses gTTS to convert the text data into an audio file and plays it back to the user through the speaker.

[1331] Step 7:

[1332] The user asks a question by voice and the device captures it. The input is the user's voice data, and the output is the captured voice data. Specifically, the microphone device records what the user says.

[1333] Step 8:

[1334] The device converts the voice question into text and sends it to the server. The input is the captured voice data, and the output is the text question data. Specifically, SpeechRecognition software is used to convert the voice into text data and send it to the server.

[1335] Step 9:

[1336] The server generates an answer to the question and notifies the user. The input is text question data, and the output is audio answer data. Specifically, an appropriate answer is generated based on the question content, converted into audio using gTTS, and played back to the user through the speaker.

[1337] Step 10:

[1338] The server stores data on the driving process. The input is all data collected during driving, and the output is the stored data. Specifically, the server stores the user's facial expression data, voice data, advice history, etc. in the database.

[1339] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1340] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1341] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1342] [Fourth embodiment]

[1343] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1344] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1345] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1346] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1347] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1348] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1349] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1350] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1351] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1352] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1353] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1354] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1355] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1356] The system of the present invention provides real-time support for cooking by combining visual information and voice communication. The system allows the user to use a device to film the cooking process and transmit the video in real time to a server, which can then provide appropriate advice on cooking methods and progress.

[1357] The detailed operation of the system is described below.

[1358] First, the user launches the app installed on their smartphone, tablet, or other device. After entering their login information and successful authentication, the device displays a recipe selection screen. The user selects the recipe they want to make and confirms the ingredients and steps required.

[1359] When the user starts cooking, the device's camera captures video. This video is streamed to the server in real time. The server analyzes the received video data and extracts information such as the size of the ingredients, how well they are mixed, and how browned they are. This allows the server to determine whether the ingredients are cut to the right size, mixed thoroughly, and cooked to the right degree.

[1360] The server generates advice based on the analysis results and notifies the user in voice and text format. For example, when frying onions, it may provide specific instructions such as "You need to fry them a little longer." This advice helps the user understand what steps to take next.

[1361] Furthermore, if a user has a question while cooking, they can verbally ask the device. The device uses voice recognition technology to convert the question into text and send it to the server. The server analyzes the question, generates an appropriate answer, and notifies the user by voice. For example, in response to a question such as "What should I do next?", the device will respond with "Next, add the chicken."

[1362] Once cooking is complete, the user ends the session within the app and the server stores the cooking process data for future reference and analysis.

[1363] As a concrete example, consider a user making teriyaki chicken. The user launches the app, logs in, and selects the teriyaki chicken recipe. They prepare the ingredients and use their smartphone camera to film the cooking process. The server receives the video and provides specific advice in real time, such as "You need to fry the onions a little longer." If the user also asks, "Is the chicken browned to the desired degree?" the server analyzes the video and returns an appropriate answer. Finally, when cooking is complete, the user ends the session, and the cooking data is saved to the server.

[1364] As described above, the system of the present invention combines real-time video analysis and voice communication to provide detailed and immediate support to users, allowing even beginners and users who are unsure about certain dishes to cook with confidence.

[1365] The processing flow will be explained below.

[1366] Step 1:

[1367] A user launches an app installed on a device such as a smartphone or tablet, and the device displays a login screen prompting the user to enter their username and password.

[1368] Step 2:

[1369] The user enters their username and password and presses the login button. The device temporarily stores the entered information and prepares to send it to the server. The device then sends an authentication request to the server.

[1370] Step 3:

[1371] The server compares the received user information with the records in its database and generates an authentication result. If the authentication is successful, the server returns an authentication success message to the terminal.

[1372] Step 4:

[1373] The device receives the authentication result and displays success or failure to the user. If authentication is successful, the recipe selection screen is displayed.

[1374] Step 5:

[1375] The user selects the desired recipe from the list and enters the selected recipe ID into the terminal, which temporarily stores the selected recipe ID and prepares to send it to the server.

[1376] Step 6:

[1377] The terminal sends a request for the selected recipe to the server. The server receives the request and retrieves detailed data for the selected recipe from the database. The server then sends the retrieved recipe data to the terminal.

[1378] Step 7:

[1379] The device displays the received recipe data and tells the user the ingredients and steps required, allowing the user to prepare the ingredients and begin cooking.

[1380] Step 8:

[1381] The user initiates the cooking process and captures the video using the device's camera. The device starts recording in real time and streams the video data to the server.

[1382] Step 9:

[1383] The server receives the video data and temporarily stores it for analysis. The server then uses image analysis algorithms to extract information such as the size of the ingredients, how well they are mixed, and how browned they are.

[1384] Step 10:

[1385] The server generates appropriate advice based on the analysis results, and the advice is sent to the device in voice and text format.

[1386] Step 11:

[1387] The device will display and voice the received advice to the user, who will then proceed with the next cooking step based on the advice.

[1388] Step 12:

[1389] If a user has a question while cooking, they can ask it verbally to the device, which will then use voice recognition technology to convert the question into text and send it to the server.

[1390] Step 13:

[1391] The server analyzes the textual content of the question and generates an appropriate answer, which is then sent to the device in voice and text format.

[1392] Step 14:

[1393] The device displays the received answers to the user and notifies them by voice, allowing the user to proceed with cooking while having their questions resolved.

[1394] Step 15:

[1395] When the user finishes cooking, they press the end session button in the app, and the device sends a session end request to the server.

[1396] Step 16:

[1397] The server receives the session data, saves it in the database, and notifies the terminal that the save is complete.

[1398] Step 17:

[1399] The device will display and sound a message to the user that cooking is complete, and the entire cooking process will be completed.

[1400] Example 1

[1401] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1402] The lack of a system that allows users to receive appropriate advice and guidance in real time while cooking poses a challenge, making it difficult for beginners and users who are unsure about certain dishes to cook with confidence. Furthermore, conventional cooking support systems are unable to respond immediately to users' questions, making it difficult to provide appropriate support tailored to the user's actual cooking progress.

[1403] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1404] In this invention, the server includes means for a user to use a device to film the cooking process, means for streaming the cooking process to the server in real time, means for the server to analyze the video data received and extract information on the size and mixing of ingredients and browning, means for generating advice based on the extracted information and notifying the user of the advice in audio and text format, means for the user to verbally ask a question, means for converting the verbal question into text and sending it to the server, means for the server to generate an answer to the question and notify the user in audio and text format, and means for saving the cooking process data. This allows users to receive appropriate cooking guidance and advice in real time, allowing even beginners and users who are unsure about certain dishes to cook with confidence.

[1405] A "user" is a person who uses the system to assist with the cooking process.

[1406] "Device" refers to a mobile device such as a smartphone or tablet used by a user.

[1407] The "cooking process" refers to the series of steps involved in preparing ingredients, cutting, mixing, heating, etc., to complete a dish.

[1408] "Streaming" is a technology that transmits captured video data to a server in real time.

[1409] The "server" is a central processing unit that receives and analyzes the video data sent from the user's device and generates corresponding advice and answers.

[1410] "Video data" refers to images and videos of the cooking process captured by the device's camera.

[1411] "Analysis" refers to the process of evaluating the size of ingredients, how well mixed they are, and how browned they are based on the received video data.

[1412] "Ingredients" refer to the individual ingredients that make up a dish.

[1413] "Advice" refers to cooking instructions and advice that are generated based on the analysis results and provided to the user.

[1414] "Audio format" refers to a format in which advice or answers are notified to the user as audio.

[1415] "Text format" refers to a format in which advice or answers are notified to the user in written form.

[1416] "Questions" are questions or problems that users verbally input while cooking.

[1417] "Voice recognition technology" refers to technology that converts a user's verbal questions into text.

[1418] An "answer" is a response generated by a server in response to a question from a user.

[1419] "Data storage" refers to the process of storing data about the cooking process on a server for future reference.

[1420] "Real-time" refers to a time concept in which analysis and notification are carried out immediately as the cooking process progresses.

[1421] "Computer vision technology" is a general term for image recognition technology used to analyze video data.

[1422] The system of the present invention provides real-time support for cooking by combining visual information and voice communication. A specific embodiment of the system will now be described.

[1423] First, the user launches a dedicated app installed on a mobile device such as a smartphone or tablet. After the user enters their login information and is successfully authenticated, the device displays a recipe selection screen. The user selects the recipe they want to make and checks the ingredients and steps. The software used by the device at this stage generally includes the operating system (e.g., iOS, Android) and application software for the smartphone or tablet.

[1424] Next, when the user starts cooking, the device's camera is used to capture video. This video is then streamed to the server in real time using technologies such as the Real-Time Messaging Protocol (RTMP) and HTTP Live Streaming (HLS). This allows the video data to be sent to the server in real time.

[1425] The server uses computer vision technology (e.g., OpenCV or TensorFlow) to analyze the received video data. The server extracts information such as the size of the ingredients, the degree of mixing, and the browning of the ingredients, and performs analysis based on this. Based on the analysis results, the server generates appropriate advice and sends it to the device. The advice is provided in audio (e.g., Google Cloud Text-to-Speech API) and text format. The device notifies the user of the advice it received.

[1426] If a user has a question while cooking, they can verbally ask the device. The device will convert the question into text using voice recognition technology (e.g., Google Cloud Speech-to-Text API) and send it to the server. The server will analyze the question, generate an appropriate answer, and notify the user in voice and text format.

[1427] Once cooking is complete, the user can end the session within the app. The server then stores the cooking process data for future reference and analysis. This data includes analysis results, user questions and answers, and any advice provided.

[1428] As a concrete example, consider a user making teriyaki chicken. The user launches the app, logs in, and selects the teriyaki chicken recipe. They prepare the ingredients and use their smartphone camera to film the cooking process. The server receives the video and provides specific advice in real time, such as "You need to fry the onions a little longer." When the user asks, "Is the chicken browned to the desired degree?" the server analyzes the video and returns an appropriate answer. Finally, when cooking is complete, the user ends the session, and the cooking data is saved to the server.

[1429] Examples of prompts include:

[1430] "I'm making chicken teriyaki. I'm currently frying onions. Can you tell me the proper doneness?"

[1431] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1432] Step 1: Launching the program and authenticating the user

[1433] Users launch a dedicated app on their smartphone or tablet and enter their login information (username and password).

[1434] The terminal sends the entered login information to the server (input: user name, password).

[1435] The server compares the received login information with the authentication database and performs authentication (data processing: comparison with authentication database, output: authentication result).

[1436] If the authentication is successful, the server starts a session and displays the recipe selection screen on the terminal (output: session ID, recipe selection screen).

[1437] Step 2: Select a recipe and check ingredients

[1438] The user selects the recipe they want to make on the device screen (input: select recipe).

[1439] The terminal transmits the selected recipe information to the server (input: recipe information).

[1440] The server sends the ingredients and cooking instructions required for the selected recipe to the terminal (data processing: recipe data extraction, output: ingredient list, procedure list).

[1441] The terminal displays it on the screen, and the user checks the ingredients and proceeds with preparation (output: ingredient list, procedure list).

[1442] Step 3: Record and send footage

[1443] The user starts cooking and takes pictures using the device's camera.

[1444] The terminal streams the captured video data to the server in real time (input: video data).

[1445] The terminal transmits video data using the Real-Time Messaging Protocol (RTMP) or HTTP Live Streaming (HLS) (output: real-time video data).

[1446] Step 4: Video analysis on the server

[1447] The server uses computer vision techniques (e.g., OpenCV or TensorFlow) to analyze the received video data (input: video data).

[1448] The server extracts information such as the size of the ingredients, how well mixed they are, and how browned they are (data processing: video analysis, output: ingredient information).

[1449] The cooking progress is evaluated based on the analysis results (data calculation: cooking progress evaluation, output: progress evaluation results).

[1450] Step 5: Server generates and notifies advice

[1451] The server generates specific advice based on the analysis results (data processing: advice generation, output: advice).

[1452] The server sends the generated advice to the terminal in voice (e.g., Text-to-Speech API) and text format (input: advice, output: voice data, text data).

[1453] The device notifies the user of the received advice (specific actions: playing audio, displaying text).

[1454] Step 6: Respond to user questions

[1455] If the user has a question while cooking, they can ask verbally into the terminal (input: voice question).

[1456] The device uses voice recognition technology (e.g., Speech-to-Text API) to convert the question into text and send it to the server (data processing: voice recognition, output: text question).

[1457] The server analyzes the question and generates an appropriate answer (data processing: question analysis, output: answer).

[1458] The server sends the generated answer to the terminal in voice and text format (input: answer, output: voice data, text data).

[1459] The terminal notifies the user of the answer by voice and text (specific operations: playing voice, displaying text).

[1460] Step 7: Save the data after cooking is complete

[1461] When the user is done cooking, they can end the session within the app (input: end session).

[1462] The server stores the data of the cooking process (data processing: data storage, output: stored data).

[1463] The saved data is used for future reference and analysis (specific operation: saving to a database).

[1464] (Application example 1)

[1465] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1466] With current food delivery services, it is difficult to control the quality of food during delivery, making it difficult for delivery personnel to efficiently deliver food while checking its condition. Furthermore, there is a lack of means to receive real-time feedback on the state of the food, making it a challenge to improve delivery quality.

[1467] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1468] In this invention, the server includes: means for a user and a delivery person to use a device to film the cooking process and the state of the food; means for streaming the cooking process and the state of the food to the server in real time; means for the server to analyze the received video data and evaluate information on the size and mixture of ingredients and browning and the quality of the food; means for generating advice based on the extracted information and evaluation results and notifying the user and the delivery person of the advice; means for the user and the delivery person to ask verbal questions; means for converting the verbal questions into text and sending it to the server; means for the server to generate answers to the questions and notify the user and the delivery person; and means for storing the cooking process data and food state data. This enables real-time quality control of the cooking process and food during delivery, enabling the delivery person to efficiently deliver high-quality food.

[1469] "User" means any person who uses the system to receive real-time assistance and advice during the cooking process.

[1470] "Delivery person" means a person who performs quality control during the delivery of food in a food delivery service.

[1471] The "device" is a hard device that has a camera function and transmits the actions of the user or delivery person to the server in real time.

[1472] A "server" is a computing device that analyzes received video data and generates advice.

[1473] "Video data" refers to visual information captured by the user or delivery person showing the cooking process and the condition of the food.

[1474] "Analysis" is the process of evaluating the size of ingredients, the degree of mixing, the browning, and the quality of the food based on the received video data.

[1475] "Advice" refers to instructions and advice regarding cooking and quality control provided based on the analysis results.

[1476] "Streaming" is a technology for transmitting video data to a server in real time.

[1477] "Questions" are questions or clarifications about the cooking process or the condition of the food that users or delivery personnel ask the system.

[1478] "Text conversion" is the process of converting spoken questions into written information.

[1479] "Notification" is the act of transmitting advice or answers generated by the server to the user or delivery person.

[1480] "Data archiving" is the process of recording information about analysis results and recommendations for future reference.

[1481] "Quality evaluation" is the process of determining the quality of food based on its condition.

[1482] "Audio format" refers to a means of providing advice or answers to users or delivery personnel via voice.

[1483] An "image analysis algorithm" is a mathematical technique used to analyze video data.

[1484] "Food condition" refers to information such as the temperature, appearance, and packaging of the food during delivery.

[1485] The system of the present invention allows users and delivery staff to use devices to film the cooking process and the condition of the food, and sends the footage to a server in real time, thereby providing appropriate advice on cooking methods and progress and managing food quality.

[1486] System Configuration

[1487] 1. Hardware

[1488] Camera-equipped devices such as smartphones, tablets, and smart glasses

[1489] Computing Server

[1490] 2. Software

[1491] Image Analysis Algorithm

[1492] Speech Recognition and Synthesis Software

[1493] Streaming Server

[1494] Data processing and calculation

[1495] Video data collection and streaming

[1496] Users and delivery staff use devices (smartphones, tablets, smart glasses) to record the cooking process and the state of the food, and this video data is sent to a streaming server in real time.

[1497] Video data analysis

[1498] The streaming server analyzes the received video data, using image analysis algorithms to assess the size, consistency, browning, and food quality (temperature, appearance, packaging, etc.) of ingredients.

[1499] Advice generation and notification

[1500] Based on the analysis results, the server generates advice on the cooking progress and food quality, which is communicated to the user and delivery person in voice and text format.

[1501] Questions and Answers

[1502] When a user or delivery person verbally asks a question, the device converts the voice into text and sends it to the server, which analyzes the question, generates an appropriate answer, and notifies the user in voice and text format.

[1503] Data storage

[1504] Analysis results, advice, and question and answer data are stored by the server for future reference and service improvement.

[1505] Specific examples

[1506] Ramen delivery

[1507] A delivery person wears smart glasses and takes a video to check the condition of the ramen during delivery. The streaming server analyzes the video and provides advice such as "The soup has not spilled" and "The ramen ingredients have not fallen apart." If the delivery person asks, "Please check if the ramen ingredients have fallen apart," the voice is converted into text, and the server analyzes the question and provides an appropriate answer.

[1508] Prompt Sentence Examples

[1509] "Please determine whether the ramen ingredients have fallen apart."

[1510] In this way, the system of the present invention controls the quality of food in real time during the cooking process and delivery, allowing delivery personnel to efficiently deliver high-quality food.

[1511] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1512] Step 1:

[1513] Users and delivery personnel activate devices (smartphones, tablets, smart glasses) to capture real-time images of the cooking process or the state of the food.

[1514] Input: Image from the device's camera.

[1515] Output: Real-time video data.

[1516] Specific operation: The user or delivery person points the camera at the target object (food or dish) and presses the capture button. The device acquires the captured data.

[1517] Step 2:

[1518] The video data captured by the terminal is sent to a streaming server.

[1519] Input: Real-time video data.

[1520] Output: Streamed video data.

[1521] Specific operation: Using the device's communication function, video data is uploaded to a server via the Internet in real time. The data is encrypted and transmitted securely.

[1522] Step 3:

[1523] The server analyzes the video data it receives and evaluates the size of the ingredients, how well mixed they are, how browned they are, and the quality of the food (temperature, appearance, packaging, etc.).

[1524] Input: Streamed video data.

[1525] Output: Analysis results (ingredients size, mixing condition, browning, quality evaluation, etc.).

[1526] Specific operation: The image analysis algorithm is executed on the server to analyze the received video data, extracting and evaluating parameters such as the size of ingredients, the degree of mixing, and browning.

[1527] Step 4:

[1528] The server generates advice based on the analysis results and notifies the user and delivery person.

[1529] Input: Analysis results.

[1530] Output: Advice (audio and text format).

[1531] Specific operation: The server generates appropriate advice based on the analysis results and notifies the user and delivery person using speech synthesis and text generation technologies. The advice is sent via a dedicated application.

[1532] Step 5:

[1533] The user or delivery person asks a question verbally, and the device converts the speech into text and sends it to the server.

[1534] Input: Audio question.

[1535] Output: The question in text format.

[1536] Specific operation: The user or delivery person speaks a question into the terminal, and the terminal uses voice recognition technology to convert the speech into text and send it to the server.

[1537] Step 6:

[1538] The server generates an answer to the received question and notifies the user and the delivery person.

[1539] Input: A text question.

[1540] Output: Answers (in audio and text format).

[1541] Specific operation: The server analyzes the received question and generates an appropriate answer using a generative AI model. The generated answer is then notified to the user and delivery person in voice synthesis and text format.

[1542] Step 7:

[1543] The server stores the analysis results, advice, questions and answers data for future reference and to help improve our services.

[1544] Input: Analysis results, advice, questions, answers.

[1545] Output: Saved data.

[1546] Specific operation: The server uses a database system to permanently store analysis results, advice, questions, and answer data. The stored data can be referenced at a later date and used to improve the service.

[1547] These steps allow for real-time quality control of food during the cooking process and delivery, enabling delivery personnel to efficiently deliver high-quality food.

[1548] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1549] The system of the present invention provides real-time support for cooking by combining visual information and voice communication, and also includes an emotion engine that recognizes the user's emotions. This system allows the user to receive appropriate advice in real time and obtain more effective support during the cooking process.

[1550] The detailed operation of the system is described below.

[1551] First, the user launches the app installed on their smartphone, tablet, or other device. After entering their login information and successful authentication, the device displays a recipe selection screen. The user selects the recipe they want to make and confirms the ingredients and steps required.

[1552] When starting cooking, the user uses the device's camera to film the cooking process. This video is streamed to the server in real time. The server analyzes the received video data and extracts information such as the size of the ingredients, how well they are mixed, and how browned they are. Based on the results of this analysis, the server generates appropriate advice and notifies the user in voice and text format. For example, when frying onions, the server may provide specific instructions such as "You need to fry them a little longer."

[1553] Furthermore, if a user has a question while cooking, they can verbally ask the device. The device uses voice recognition technology to convert the question into text and send it to the server. The server analyzes the question, generates an appropriate answer, and notifies the user by voice. For example, in response to a question such as "What should I do next?", the device will respond with "Next, add the chicken."

[1554] The system includes an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's facial expressions and vocal tone. This allows the server to understand the user's emotional state and adjust the content and tone of advice and responses accordingly. For example, if the user is feeling stressed, the server will provide gentle advice such as, "It's okay, try taking it a little slower."

[1555] Once cooking is complete, the user ends the session within the app and the server stores the cooking process data for future reference and analysis.

[1556] As a concrete example, consider a user making teriyaki chicken. The user launches the app, logs in, and selects the teriyaki chicken recipe. They prepare the ingredients and use their smartphone camera to film the cooking process. The server receives the video and provides specific advice in real time, such as "You need to fry the onions a little longer." Furthermore, if the user asks, "Is the chicken browned to the desired degree?" the server analyzes the video and returns an appropriate answer. If the emotion engine recognizes that the user is stressed, it provides supportive advice, such as "Take it slowly, it's fine."

[1557] Finally, when cooking is complete, the user ends the session and the cooking data is saved on the server. This system allows users to cook with peace of mind, receiving detailed support that also responds to emotional changes.

[1558] The processing flow will be explained below.

[1559] Step 1:

[1560] A user launches an app installed on a device such as a smartphone or tablet, and the device displays a login screen prompting the user to enter their username and password.

[1561] Step 2:

[1562] The user enters their username and password and presses the login button. The device temporarily stores the entered information and prepares to send it to the server. The device then sends an authentication request to the server.

[1563] Step 3:

[1564] The server compares the received user information with the records in its database and generates an authentication result. If the authentication is successful, the server returns an authentication success message to the terminal.

[1565] Step 4:

[1566] The device receives the authentication result and displays success or failure to the user. If authentication is successful, the recipe selection screen is displayed.

[1567] Step 5:

[1568] The user selects the desired recipe from the list and enters the selected recipe ID into the terminal, which temporarily stores the selected recipe ID and prepares to send it to the server.

[1569] Step 6:

[1570] The terminal sends a request for the selected recipe to the server. The server receives the request and retrieves detailed data for the selected recipe from the database. The server then sends the retrieved recipe data to the terminal.

[1571] Step 7:

[1572] The device displays the received recipe data and tells the user the ingredients and steps required, allowing the user to prepare the ingredients and begin cooking.

[1573] Step 8:

[1574] The user initiates the cooking process and captures the video using the device's camera. The device starts recording in real time and streams the video data to the server.

[1575] Step 9:

[1576] The server receives the video data and temporarily stores it for analysis. Using an image analysis algorithm, the server extracts information such as the size of the ingredients, how well they are mixed, and how browned they are, allowing it to determine the appropriate cooking status.

[1577] Step 10:

[1578] The server generates appropriate advice based on the analysis results, and the advice is sent to the device in voice and text format.

[1579] Step 11:

[1580] The device will display and voice the received advice to the user, who will then proceed with the next cooking step based on the advice.

[1581] Step 12:

[1582] If a user has a question while cooking, they can ask it verbally to the device, which will then use voice recognition technology to convert the question into text and send it to the server.

[1583] Step 13:

[1584] The server analyzes the textual content of the question and generates an appropriate answer, which is then sent to the device in voice and text format.

[1585] Step 14:

[1586] The device displays the received answers to the user and notifies them by voice, allowing the user to proceed with cooking while having their questions resolved.

[1587] Step 15:

[1588] By analyzing the user's facial expressions and voice tone from the camera footage, the device uses an emotion engine to recognize the user's emotional state, determining whether the user is stressed or relaxed.

[1589] Step 16:

[1590] The server adjusts the tone and content of its advice and responses based on the user's perceived emotional state. For example, if the user is feeling stressed, the server will provide considerate advice in a gentle tone, such as "Try taking it a little slower."

[1591] Step 17:

[1592] When the user finishes cooking, they press the end session button in the app, and the device sends a session end request to the server.

[1593] Step 18:

[1594] The server receives the session data, saves it in the database, and notifies the terminal that the save is complete.

[1595] Step 19:

[1596] The device will display and sound a message to the user that cooking is complete, and the entire cooking process will be completed.

[1597] Through the above process, the system of the present invention provides users with real-time cooking support through visual information and voice communication, and furthermore, by recognizing the user's emotional state using an emotion engine, it realizes detailed support.

[1598] Example 2

[1599] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1600] Conventional cooking assistance systems make it difficult for users to receive appropriate advice during the cooking process. They also lack the ability to respond to the user's emotional state, making it difficult to provide sufficient support to users who are feeling stressed or anxious. This increases the likelihood of inconvenience and mistakes during cooking, potentially resulting in a poor user experience.

[1601] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1602] In this invention, the server includes: means for a user to use a device to film the cooking process; means for streaming the cooking process to the server in real time; means for the server to analyze the received video data and extract information such as the size and mixing condition of ingredients and browning; means for generating advice based on the extracted information and notifying the user of the advice; means for the user to verbally ask a question; means for converting the verbal question into text and sending it to the server; means for the server to generate an answer to the question and notify the user; means for recognizing emotions from the user's facial expression and tone of voice; means for adjusting the content of the advice and answer based on the user's emotional state; and means for saving the cooking process data. This allows the user to receive appropriate advice in real time, quickly respond to any questions they may have while cooking, and receive support tailored to their emotional state, allowing them to cook safely and effectively.

[1603] "User" refers to an individual who uses the system to cook food.

[1604] "Device" refers to a computer terminal used by a user, such as a smartphone or tablet.

[1605] "Server" refers to a central remote computing system that performs processing such as data analysis and advice generation.

[1606] The "cooking process" refers to the series of operations and actions involved in preparing a dish.

[1607] "Means for taking pictures" refers to a method or device for recording images using the camera function of the device.

[1608] "Means of streaming in real time" refers to methods and technologies for transmitting captured video to a server without delay.

[1609] "Video data" refers to video information of the cooking process captured by a camera.

[1610] "Means of analysis" refers to the computational process that processes the video data and extracts important information such as the size of the ingredients, how well they are mixed, and how browned they are.

[1611] "Ingredient size" refers to the physical dimensions or volume of the ingredient being cooked.

[1612] "Mixing condition" refers to the state of how evenly the ingredients are mixed together.

[1613] "Brown" refers to the state in which ingredients change color when heated, particularly the degree of doneness.

[1614] "Advice" refers to specific cooking instructions or suggestions provided to users based on the analyzed information.

[1615] "Verbal means of questioning" refers to the method by which a user uses voice to express questions or instructions to the system.

[1616] "Means for converting to text" refers to speech recognition technology that converts speech into text data.

[1617] "Answer to a question" refers to an answer or explanation generated by the server in response to a user's question.

[1618] "Facial expressions" refer to emotions and states that can be read from the user's facial movements and muscle placement.

[1619] "Voice tone" refers to the pitch and tone of a user's voice when they speak, and is an element that expresses emotion.

[1620] "Means of emotion recognition" refers to technologies and algorithms that determine a user's emotional state from their facial expressions and voice.

[1621] "Emotional state" refers to the emotional state a user is feeling, such as joy, anger, sadness, or fear.

[1622] "Adjustment techniques" refer to methods and techniques for appropriately changing the content and tone of advice or responses depending on the person's emotional state.

[1623] "Data storage means" refers to a method or system that keeps a record of the cooking process for future reference or analysis.

[1624] The system of the present invention provides real-time support for cooking by combining visual information and voice communication. It also includes an emotion engine that recognizes the user's emotions. This system allows the user to receive appropriate advice in real time, providing more effective support during the cooking process.

[1625] First, the user launches the app installed on their smartphone, tablet, or other device. After entering their login information and successful authentication, the device displays a recipe selection screen. The user selects the recipe they want to make and confirms the ingredients and steps required.

[1626] When starting cooking, the user uses the device's camera to film the cooking process. This video is streamed in real time to a server. The server analyzes the received video data and extracts information such as the size of ingredients, the degree of mixing, and browning. This analysis uses image processing techniques (e.g., OpenCV) and generative AI models (e.g., TensorFlow).

[1627] Based on the analysis results, the server generates appropriate advice and notifies the user in voice and text format. For example, when frying onions, it provides specific instructions such as "You need to fry them a little longer."

[1628] Furthermore, if a user has a question while cooking, they can verbally ask the device. In this case, the device uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert the question into text and send it to the server. The server analyzes the question, generates an appropriate answer, and notifies the user by voice. For example, in response to the question, "What should I do next?", the device will respond with, "Next, add the chicken."

[1629] Furthermore, the system includes an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's facial expressions and vocal tone. This recognition uses computer vision and voice analysis technologies (e.g., Azure Cognitive Services). This allows the server to understand the user's emotional state and adjust the content and tone of its advice and responses accordingly. For example, if the user is feeling stressed, the server will provide gentle advice such as, "It's okay, try taking it a little slower."

[1630] Once cooking is complete, the user ends the session within the app and the server stores the cooking process data for future reference and analysis.

[1631] As a concrete example, consider a user making teriyaki chicken. The user launches the app, logs in, and selects the teriyaki chicken recipe. They prepare the ingredients and use their smartphone camera to film the cooking process. The server receives the video and provides specific advice in real time, such as "You need to fry the onions a little longer." If the user asks, "Is the chicken browned properly?", the server analyzes the video and returns an appropriate answer. Additionally, if the emotion engine recognizes that the user is stressed, it provides supportive advice such as "Take it slowly, it's fine."

[1632] Finally, when cooking is complete, the user ends the session and the cooking data is saved on the server. This system allows users to cook with peace of mind, receiving detailed support that also responds to emotional changes.

[1633] Example prompt sentence:

[1634] "Show me the chicken teriyaki recipe."

[1635] "Analyze the results of frying onions."

[1636] "What should I do next?"

[1637] "Is this the right browning for the chicken?"

[1638] "Provide advice on what to do if users are stressed."

[1639] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1640] Step 1:

[1641] App launch and authentication

[1642] Input: The user launches the app on their smartphone or tablet and enters their login information (user ID and password).

[1643] Output: A response from the server indicating authentication success or failure.

[1644] Specific behavior:

[1645] 1. The user launches the app.

[1646] 2. The login screen appears and the user enters their ID and password.

[1647] 3. The device sends the input information to the server.

[1648] 4. The server checks the database for authentication and returns the result to the terminal.

[1649] Step 2:

[1650] Recipe selection and display

[1651] Input: The recipe selected by the user on the recipe selection screen upon successful authentication.

[1652] Output: Ingredient list and instructions for the selected recipe.

[1653] Specific behavior:

[1654] 1. After successful authentication, the device displays the recipe selection screen.

[1655] 2. The user selects the recipe they want to make, and the device requests that information from the server.

[1656] 3. The server receives the request and sends the ingredients list and instructions for the corresponding recipe to the device.

[1657] 4. The terminal displays it on the screen.

[1658] Step 3:

[1659] Filming and streaming the cooking process

[1660] Input: User starts cooking, footage captured by camera.

[1661] Output: Real-time video data sent to the server.

[1662] Specific behavior:

[1663] 1. Users use a camera to record the cooking process in real time.

[1664] 2. The video captured by the device is streamed to the server in real time.

[1665] Step 4:

[1666] Video data analysis

[1667] Input: Streamed video data.

[1668] Output: Analysis results such as size of ingredients, mixing condition, browning, etc.

[1669] Specific behavior:

[1670] 1. The server analyzes the received video data.

[1671] 2. Image processing technology (e.g., OpenCV) and generative AI models (e.g., TensorFlow) are used to determine the size, mixing level, and browning of ingredients.

[1672] 3. The server processes the analysis results in real time.

[1673] Step 5:

[1674] Providing real-time advice

[1675] Input: Analysis results.

[1676] Output: Advice that will be sent to the user.

[1677] Specific behavior:

[1678] 1. The server generates advice based on the analysis results.

[1679] 2. Send advice in voice and text format to the device.

[1680] 3. The device notifies the user of the advice.

[1681] 4. For example, the advice "It needs to be cooked a little longer" will be displayed.

[1682] Step 6:

[1683] Audio questions and answers

[1684] Input: The user's spoken question.

[1685] Output: A suitable response from the server.

[1686] Specific behavior:

[1687] 1. The user has a question and verbally asks it to the device.

[1688] 2. The device uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert the question into text.

[1689] 3. Send the converted text to the server.

[1690] 4. The server analyzes the question, generates an appropriate answer, and sends it to the device.

[1691] 5. The device will notify the user of the answer via voice and text.

[1692] 6. For example, to the question "What should I do next?", the answer announced is "Next, add the chicken."

[1693] Step 7:

[1694] Emotion Recognition and Response

[1695] Input: The user's facial expression and voice tone.

[1696] Output: Advice or answer tailored by the server.

[1697] Specific behavior:

[1698] 1. The device uses a camera and microphone to record the user's facial expressions and voice tone.

[1699] 2. Analyze user emotions using emotion recognition technology (e.g., Azure Cognitive Services).

[1700] 3. Send the analysis results to the server.

[1701] 4. The server adjusts the content and tone of advice and responses based on the user's emotional state.

[1702] 5. For example, if the user is feeling stressed, the device will provide advice such as, "Take it easy, it's okay."

[1703] Step 8:

[1704] Ending a session and saving data

[1705] Input: Session termination information.

[1706] Output: Saved cooking process data.

[1707] Specific behavior:

[1708] 1. After the user finishes cooking, they end the session within the app.

[1709] 2. The terminal sends the session termination information to the server.

[1710] 3. The server stores the cooking process data in a database.

[1711] (Application example 2)

[1712] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1713] Driver stress and reduced attention in autonomous vehicles are a major issue, increasing the risk of traffic accidents. There is also a need to provide a safe driving environment where drivers can receive appropriate advice in real time while driving.

[1714] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to use the device to film the driving process, means for streaming the driving process to the server in real time, means for analyzing the video data received by the server and recognizing the user's emotions and attention state, and means for generating advice or warnings based on the recognized emotions and notifying the user of the advice or warnings. This makes it possible to provide detailed support according to the user's emotional state and improve safety and comfort while driving.

[1715] "User" means an individual or vehicle operator using the system.

[1716] "Devices" are hardware devices such as cameras and microphones installed in autonomous vehicles.

[1717] The "means for photographing the cooking process" is a function that uses a camera inside the self-driving vehicle to photograph the driving situation.

[1718] "Means for streaming to a server in real time" refers to a technology that transmits captured video to a server without delay.

[1719] "Means for analyzing video data and extracting information on the size, mixing level and browning of ingredients" refers to an algorithm that analyzes video data and recognizes the operating status.

[1720] The "means for generating advice based on the extracted information and notifying the user of the advice" is a technology for generating appropriate driving guidance or warnings based on the analysis results and providing them to the user.

[1721] "Means for users to ask questions verbally" refers to a function that allows the driver to ask questions verbally using a microphone.

[1722] The "means for converting an oral question into text and transmitting it to a server" is a technology that uses voice recognition technology to convert an oral question into text format and transmit it to a server.

[1723] "Means for generating answers to questions and notifying users" refers to a technology in which the server generates an appropriate answer based on the question received and notifies the user of the answer by voice or text.

[1724] The "means for recognizing emotions from a user's facial expression and voice tone" is a technology that performs facial expression analysis and voice analysis to recognize the user's emotional state.

[1725] "Means for adjusting the content and tone of advice or responses based on recognized emotions" refers to a technique for adjusting the content of advice or responses provided based on the recognized emotional state, and using a gentle tone or encouraging words as necessary.

[1726] "Means for saving cooking process data" refers to a function that saves the user's operating data on a server so that it can be referenced and analyzed later.

[1727] The present invention relates to a driving assistance system for autonomous vehicles, and aims to provide appropriate advice to users in real time while they are driving, thereby improving the safety and comfort of the driving environment.

[1728] The system consists of the following main components:

[1729] 1. User Device:

[1730] Hardware:

[1731] Camera: Used to record the inside of the vehicle.

[1732] Microphone: Used to capture the user's voice.

[1733] software:

[1734] Speech recognition: Software that converts speech into text.

[1735] Video capture: Software that captures camera footage and streams it to a server.

[1736] 2. Server:

[1737] Hardware:

[1738] A computer equipped with a high-performance processor and large memory capacity.

[1739] software:

[1740] Image analysis algorithm (OpenCV, etc.): Analyzes camera footage and recognizes the user's facial expressions and driving state.

[1741] Emotion recognition engine (deep learning models such as Keras): Recognizes user emotions from facial expressions and voice tone.

[1742] Auto-response generation system (e.g. gTTS): Generates appropriate advice and warnings and notifies them in audio format.

[1743] Speech recognition software (such as SpeechRecognition): Converts spoken questions into text.

[1744] 3. System Operation:

[1745] The user device uses a camera to capture video of the interior of the vehicle and streams it to a server in real time.

[1746] The server analyzes the received video data and recognizes the user's facial expressions and driving state.

[1747] Based on the recognized data, the server generates appropriate advice or warnings and notifies the user in voice format.

[1748] When a user speaks a question, a microphone captures it and speech recognition software converts the question into text and sends it to a server.

[1749] The server generates answers to the questions and communicates them to the user in audio format.

[1750] 4. Example:

[1751] For example, if a user is feeling stressed while driving, the system will recognize this from their facial expression and provide voice advice such as, "You seem a little stressed. Take a deep breath." If the user says, "It's dangerous," the system will warn, "Be careful! There is danger ahead."

[1752] 5. Example prompts:

[1753] "If the user is feeling stressed while driving, the system should recognize facial expressions from the camera footage and provide voice advice such as, 'You seem to be a little stressed. Take a deep breath.' If the user says, 'It's dangerous,' the system should warn them, 'Be careful! There is danger ahead.'"

[1754] This system provides detailed support tailored to the user's emotional state, improving safety and comfort while driving.

[1755] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1756] Step 1:

[1757] The user device uses a camera to capture images of the inside of the car. The input is real-time images of the inside of the car, and the output is camera image data. Specifically, the camera device captures the entire view of the inside of the car and the user's face, and generates this as streaming data.

[1758] Step 2:

[1759] The video data captured by the user device is streamed to the server in real time. The input is the camera video data, and the output is the streaming data to the server. Specifically, the video data is sent to the server via the network.

[1760] Step 3:

[1761] The server analyzes the received video data and recognizes the user's facial expressions and driving state. The input is streaming video data, and the output is data on the recognized facial expressions and driving state. Specifically, the video is analyzed frame by frame using OpenCV's image analysis algorithm, and features are extracted for face detection and emotion recognition.

[1762] Step 4:

[1763] The server uses an emotion recognition engine to recognize the user's emotions from facial expressions and voice tone. The input is the analyzed video data and the user's voice data, and the output is the recognized emotional state. Specifically, a deep learning model using Keras classifies emotions such as joy, anger, and sadness from facial expressions and analyzes voice tone.

[1764] Step 5:

[1765] The server generates appropriate advice or warnings based on the recognized data. The input is the recognized emotional state data, and the output is text data of the advice or warning. Specifically, a pre-defined advice or warning message is selected based on the generated emotional data.

[1766] Step 6:

[1767] The server notifies the user of the generated advice or warning in audio format. The input is the text data of the generated advice or warning, and the output is the audio notification data. Specifically, the server uses gTTS to convert the text data into an audio file and plays it back to the user through the speaker.

[1768] Step 7:

[1769] The user asks a question by voice and the device captures it. The input is the user's voice data, and the output is the captured voice data. Specifically, the microphone device records what the user says.

[1770] Step 8:

[1771] The device converts the voice question into text and sends it to the server. The input is the captured voice data, and the output is the text question data. Specifically, SpeechRecognition software is used to convert the voice into text data and send it to the server.

[1772] Step 9:

[1773] The server generates an answer to the question and notifies the user. The input is text question data, and the output is audio answer data. Specifically, an appropriate answer is generated based on the question content, converted into audio using gTTS, and played back to the user through the speaker.

[1774] Step 10:

[1775] The server stores data on the driving process. The input is all data collected during driving, and the output is the stored data. Specifically, the server stores the user's facial expression data, voice data, advice history, etc. in the database.

[1776] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1777] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1778] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1779] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1780] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1781] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1782] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1783] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, motorcycles, and other devices, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1784] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1785] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1786] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1787] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1788] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1789] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1790] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1791] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1792] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1793] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1794] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1795] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1796] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1797] The following is further disclosed regarding the above embodiment.

[1798] (Claim 1)

[1799] a means for a user to use the device to photograph the cooking process;

[1800] means for streaming the cooking process to a server in real time;

[1801] A means for analyzing the video data received by the server and extracting information on the size and mixing condition of ingredients and browning;

[1802] means for generating advice based on the extracted information and notifying the user of the advice;

[1803] means for said user to verbally ask a question;

[1804] means for converting the verbal question into text and transmitting the text to a server;

[1805] A means for the server to generate an answer to the question and notify the user;

[1806] A means for storing data on the cooking process;

[1807] A system including:

[1808] (Claim 2)

[1809] 10. The system of claim 1, wherein the means for analyzing the video data of the cooking process uses an image analysis algorithm.

[1810] (Claim 3)

[1811] 10. The system of claim 1, further comprising means for announcing said advice and answers to said user in audio form.

[1812] "Example 1"

[1813] (Claim 1)

[1814] a means for a user to use the device to photograph the cooking process;

[1815] means for streaming the cooking process to a server in real time;

[1816] A means for analyzing the video data received by the server and extracting information on the size and mixing condition of ingredients and browning;

[1817] means for generating advice based on the extracted information and notifying the user of the advice in audio and text formats;

[1818] means for said user to verbally ask a question;

[1819] means for converting the verbal question into text and transmitting the text to a server;

[1820] means for the server to generate answers to the questions and notify the user in voice and text format;

[1821] A means for storing data on the cooking process;

[1822] A system including:

[1823] (Claim 2)

[1824] 10. The system of claim 1, wherein the means for analyzing the video data of the cooking process uses image analysis techniques.

[1825] (Claim 3)

[1826] 10. The system of claim 1, further comprising means for notifying the user of said advice and answers in audio and text formats.

[1827] "Application Example 1"

[1828] (Claim 1)

[1829] a means for a user to use the device to photograph the cooking process;

[1830] means for streaming the cooking process to a server in real time;

[1831] A means for analyzing the video data received by the server and extracting information on the size and mixing condition of ingredients and browning;

[1832] means for generating advice based on the extracted information and notifying the user of the advice;

[1833] means for said user to verbally ask a question;

[1834] means for converting the verbal question into text and transmitting the text to a server;

[1835] A means for the server to generate an answer to the question and notify the user;

[1836] A means for storing data on the cooking process;

[1837] A means for the delivery person to take a photo of the condition of the food;

[1838] means for analyzing the food condition data received by the server and evaluating the quality of the food;

[1839] A means for providing advice to a delivery person based on the evaluation result;

[1840] A system including:

[1841] (Claim 2)

[1842] 2. The system of claim 1, wherein the means for analyzing the video data of the cooking process and the food state data uses an image analysis algorithm.

[1843] (Claim 3)

[1844] 2. The system of claim 1, further comprising means for notifying the user and delivery person of the advice and answer in audio form.

[1845] "Example 2: Combining Emotion Engines"

[1846] (Claim 1)

[1847] a means for a user to use the device to photograph the cooking process;

[1848] means for streaming the cooking process to a server in real time;

[1849] A means for analyzing the video data received by the server and extracting information on the size and mixing condition of ingredients and browning;

[1850] means for generating advice based on the extracted information and notifying the user of the advice;

[1851] a means for the user to verbally ask a question;

[1852] means for converting the verbal question into text and transmitting the text to a server;

[1853] A means for the server to generate an answer to the question and notify the user;

[1854] means for recognizing emotions from the user's facial expressions and vocal tones;

[1855] means for adjusting the content of advice and answers based on said emotional state;

[1856] A means for storing data on the cooking process;

[1857] A system including:

[1858] (Claim 2)

[1859] 10. The system of claim 1, wherein the means for analyzing the video data of the cooking process uses image analysis algorithms and generative AI models.

[1860] (Claim 3)

[1861] 10. The system of claim 1, further comprising means for announcing said advice and answers to said user in audio form.

[1862] "Application example 2 when combining emotion engines"

[1863] (Claim 1)

[1864] a means for a user to use the device to photograph the cooking process;

[1865] means for streaming the cooking process to a server in real time;

[1866] A means for analyzing the video data received by the server and extracting information on the size and mixing condition of ingredients and browning;

[1867] means for generating advice based on the extracted information and notifying the user of the advice;

[1868] means for said user to verbally ask a question;

[1869] means for converting the verbal question into text and transmitting the text to a server;

[1870] A means for the server to generate an answer to the question and notify the user;

[1871] means for recognizing emotions from the user's facial expressions and vocal tones;

[1872] means for adjusting the content and tone of advice or responses based on the perceived emotions;

[1873] A means for storing data on the cooking process;

[1874] A system including:

[1875] (Claim 2)

[1876] 10. The system of claim 1, wherein the means for analyzing the video data of the cooking process uses an image analysis algorithm.

[1877] (Claim 3)

[1878] 10. The system of claim 1, further comprising means for announcing said advice and answers to said user in audio form. [Explanation of symbols]

[1879] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for a user to use the device to photograph the cooking process; means for streaming the cooking process to a server in real time; A means for analyzing the video data received by the server and extracting information on the size and mixing condition of ingredients and browning; means for generating advice based on the extracted information and notifying the user of the advice; means for said user to verbally ask a question; means for converting the verbal question into text and transmitting the text to a server; A means for the server to generate an answer to the question and notify the user; A means for storing data on the cooking process; A system including:

2. 10. The system of claim 1, wherein the means for analyzing the video data of the cooking process uses an image analysis algorithm.

3. 2. The system of claim 1, further comprising means for announcing said advice and answers to said user in audio form.

Images