System

Abstract

To provide a system capable of efficiently and accurately learning a traditional technique and transmitting it to the next generation.SOLUTION: Means for recording a process of cooking by a user, means for uploading data of the recorded process of cooking to a server, means for converting the data received by the server into a format that can be analyzed, means for analyzing the video data by the server and extracting each process, means for analyzing the audio data by the server and extracting an instruction or a comment, means for causing the AI model to learn the analysis result by the server, means for transmitting a reproduction request by the user, and means for generating a specific cooking procedure and recipe by the server; A means for providing to the terminal, a means for the user to feed back the cooking result, and a means for the server to analyze the feedback and improve the AI model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] It is often difficult to pass on traditional culinary techniques. It is especially difficult to teach others the techniques and flavors that chefs rely on experience and intuition. Furthermore, the tendency for young people to avoid rigorous training means that the skills are not being passed down. As a result, there is a risk that valuable techniques and flavors will be lost. Therefore, a system for effectively passing on these skills is needed. [Means for solving the problem]

[0005] The present invention solves the above problems by providing a system including the following means.

[0006] The system has a means for users to record the cooking process, a means for uploading the recorded cooking process data to a server, a means for the server to convert the received data into an analyzable format, a means for the server to analyze the video data and extract each process, a means for the server to analyze the audio data and extract instructions and comments, a means for the server to train an AI model on the analysis results, a means for the user to send a reproduction request, a means for the server to generate specific cooking steps and recipes and provide them to the device, a means for the user to provide feedback on the cooking results, and a means for the server to analyze the feedback and improve the AI ​​model. This makes it possible to efficiently and accurately learn traditional techniques and pass them on to future generations.

[0007] A "user" is an individual or organization that is responsible for recording the cooking process and uploading the data to the server.

[0008] "Recorded data" is digital data including video and audio that is taken by the user and records the cooking process.

[0009] A "server" is a computer system that receives recorded data and performs processes such as analysis, training of AI models, and generation of cooking instructions.

[0010] A "terminal" is a device (e.g., a smartphone or PC) that a user uses to upload recorded data and receive and display cooking instructions and recipes from the server.

[0011] "Analyzable format" refers to a data format that has been converted into a state that allows the server to properly process and analyze the recorded data.

[0012] "Video data" refers to the visual information portion of the recorded data, that is, data that records the cooking process frame by frame.

[0013] "Audio data" refers to the auditory information portion of the video recording data, that is, audio data that records instructions and comments given by the chef.

[0014] A "process" refers to a series of steps or actions in cooking (e.g., cutting ingredients, baking, boiling, etc.).

[0015] "Instructions and comments" refer to words and explanations given by the chef during the cooking process, and are part of the audio data.

[0016] An "AI model" is an artificial intelligence algorithm and its trained dataset that uses machine learning techniques to reproduce cooking techniques and flavors.

[0017] A "reproduction request" is a request sent by a user to the server to reproduce a particular dish or technique.

[0018] "Cooking instructions and recipes" refers to specific steps and a list of ingredients and seasonings needed to recreate a particular dish.

[0019] "Feedback" refers to the user reporting their evaluation of the cooking results and areas for improvement to the server via their terminal. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The system of the present invention allows users to easily record and analyze the cooking process, and based on that, learn and reproduce cooking techniques. An embodiment of the system will be described in detail below.

[0042] System configuration

[0043] The system mainly consists of the following elements:

[0044] 1. User Recording Method

[0045] 2. Data upload method

[0046] 3. Server data analysis method

[0047] 4. AI learning methods

[0048] 5. How to Request Reproduction

[0049] 6. Feedback channels

[0050] Program processing

[0051] The system program does the following:

[0052] Acquiring Recording Data

[0053] The user uses a means to record the cooking process. For example, the user uses a smartphone or video camera to record the cooking process. After the recording is finished, the user imports the recorded data into the device.

[0054] Uploading data

[0055] The device uploads the recorded data to the server. Data can be easily sent to the server using a dedicated application.

[0056] Data preprocessing and analysis

[0057] The server converts the received recorded data into an analyzable format. First, the video data and audio data are separated and processed separately. The video data is analyzed frame by frame to extract information such as the type of ingredients, how to use cooking utensils, and the cooking process. Meanwhile, the audio data is converted into text to extract the chef's instructions and comments.

[0058] AI Learning

[0059] The server analyzes the data and trains the AI ​​model, which allows it to accurately reproduce cooking techniques and flavors. Specifically, even minute details such as cutting methods, heat levels, and the use of seasonings are reflected in the model.

[0060] Reproduction requests and submissions

[0061] When a user wants to recreate a specific dish or technique, they send a request to the server through their device. The server generates cooking instructions and recipes based on the request and provides them to the device. The user can then follow these instructions to recreate the dish.

[0062] Feedback and model improvement

[0063] The user recreates the dish and provides feedback on the results. The device then sends the feedback to the server, which analyzes the information and improves the AI ​​model, further improving the accuracy of future attempts.

[0064] Specific examples

[0065] For example, if a user wants to make "seared bonito", the steps would be as follows:

[0066] 1. A user records a chef "searing bonito."

[0067] 2. The device uploads the recorded data to the server.

[0068] 3. The server analyzes information from the video data, such as the steps for cutting the bonito, how well it is cooked, and how to pound it, and extracts supplementary explanations from the audio data.

[0069] 4. The server trains the AI ​​model on the analysis results.

[0070] 5. The user sends a request to recreate "Seared Bonito."

[0071] 6. The server generates the optimal cooking procedure and necessary recipes and provides them to the terminal.

[0072] 7. The user follows the provided instructions to cook the food and provides feedback on the results.

[0073] 8. The server analyzes the feedback and improves the AI ​​model.

[0074] This allows users to efficiently learn and reproduce first-class cooking techniques. This system also enables the efficient and accurate inheritance of traditional cooking techniques.

[0075] The processing flow will be explained below.

[0076] Step 1:

[0077] The user records the cooking process. The user uses a smartphone or video camera to record the cooking process of a top chef in detail. The recording is done so that every process is clearly visible.

[0078] Step 2:

[0079] The user imports the recorded data to the device. Once recording is complete, the user transfers the video file to a device such as a smartphone or PC and opens the dedicated application.

[0080] Step 3:

[0081] The device uploads the recorded data to the server. A dedicated application is used to select the recorded data and issue an upload command to the server. The data is then securely transferred to the server.

[0082] Step 4:

[0083] The server receives the recorded data and prepares it for conversion into an analyzable format.

[0084] Step 5:

[0085] The server converts the recorded data into an analyzable format, splits the video file into frames, and separates the video data from the audio data.

[0086] Step 6:

[0087] The server analyzes the video data, analyzing each individual frame to extract each step of the cooking process (e.g., type of ingredient, cutting method, heat level), and even to identify the smallest actions and steps visible in the video.

[0088] Step 7:

[0089] The server analyzes the audio data, converts it into text, and extracts the chef's instructions and comments, providing a detailed textual description of each process.

[0090] Step 8:

[0091] The server trains the AI ​​model with the analysis results, and the extracted text from the video and audio data is input into the AI ​​model, which then learns how to reproduce the cooking techniques and flavors.

[0092] Step 9:

[0093] The user sends a reproduction request to the server. The user sends a request to reproduce a specific dish (e.g., "seared bonito") to the server via the terminal.

[0094] Step 10:

[0095] The server generates cooking instructions and recipes based on the request, calculates the optimal steps based on an AI model, and generates detailed recipes and cooking instructions.

[0096] Step 11:

[0097] The terminal provides the cooking instructions and recipes to the user, who can then view the cooking instructions and recipes obtained from the server through the terminal.

[0098] Step 12:

[0099] The user follows the instructions to cook the food, and then recreates the dish based on the instructions displayed on the device.

[0100] Step 13:

[0101] The user provides feedback on the cooking results, sending their evaluation of the cooking results and taste to the server via their device.

[0102] Step 14:

[0103] The server receives and analyzes the feedback sent by users and uses it to improve the AI ​​model.

[0104] Step 15:

[0105] The server improves the AI ​​model, adjusting it based on the feedback to improve the accuracy of the next replay.

[0106] This process allows users to accurately learn and recreate traditional cooking techniques.

[0107] Example 1

[0108] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0109] Traditional cooking skill learning methods often rely on watching videos or reading books, making it difficult to fully understand the details of the actual cooking process. It is particularly difficult to accurately reproduce the techniques of professional chefs, making it difficult for learners to improve their skills. Furthermore, there are limited ways to improve one's own cooking skills through feedback, making it difficult to efficiently acquire new cooking skills.

[0110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0111] In this invention, the server includes means for a user to record the cooking process, means for uploading data of the recorded cooking process to an information processing device, means for the information processing device to convert the received data into an analyzable format, means for the information processing device to analyze the video data and extract each process, means for the information processing device to analyze the audio data and extract instructions and comments, means for the information processing device to train a learning model based on the analysis results, means for a user to send a reproduction request, means for the information processing device to generate specific cooking steps and recipes and provide them to the terminal, means for a user to provide feedback on the cooking results, and means for the information processing device to analyze the feedback and improve the learning model. This enables users to accurately learn and reproduce professional cooking techniques and improve their own skills.

[0112] "User" refers to any individual or legal entity that records the cooking process and uploads the data to the system.

[0113] The "cooking process" refers to the series of steps involved in preparing ingredients, cooking, and finishing.

[0114] "Recording means" refers to a device for recording video and audio, such as a smartphone or video camera.

[0115] "Data" refers to video data and audio data acquired by a recording means.

[0116] "Information processing device" refers to a device that analyzes, processes, and stores data, such as a server or cloud service.

[0117] "Uploading means" refers to the means by which a user sends recorded data to an information processing device, including dedicated applications and web forms.

[0118] The term "analyzable format" refers to a format required for an information processing device to process data appropriately.

[0119] "Video data" refers to video frame information captured by a recording means.

[0120] "Audio data" refers to audio information captured by a recording means.

[0121] The "means for extracting each process" refers to a means for analyzing and identifying each stage of cooking from video data and audio data.

[0122] "Instructions and comments" refers to statements and explanations made by the chef or user regarding the cooking process.

[0123] "Learning model" refers to analytical algorithms and data models that use artificial intelligence to learn cooking techniques and processes.

[0124] A "reproduction request" refers to a user requesting the system to reproduce a specific dish or technique.

[0125] "Cooking instructions and recipes" refers to the specific steps and ingredients needed to recreate a dish.

[0126] "Terminal" refers to a device used by a user, such as a smartphone, tablet, or PC.

[0127] "Feedback" refers to information such as cooking results and impressions that users provide to the system.

[0128] "Analysis means" refers to the algorithms and techniques used by the information processing device to process and analyze data.

[0129] The system of the present invention allows users to easily record and analyze the cooking process, and then learn and reproduce cooking techniques based on the recorded footage. To implement this system, the following hardware and software are required: a user's smartphone, a video camera, a personal computer, a dedicated application, an information processing device (server), and a learning model.

[0130] Hardware and software used

[0131] 1. Smartphone / Video Camera: Used as a recording medium.

[0132] 2. Computer / dedicated application: Used to import and upload data.

[0133] 3. Information processing device (server): Used to analyze data, manage learning models, generate cooking instructions and recipes, and analyze feedback.

[0134] 4. Learning model: A model that uses AI technology and frameworks such as TENSORFLOW (registered trademark) or PyTorch.

[0135] Specific actions and processes

[0136] The user uses a smartphone or video camera to record the cooking process. For example, the process of making "seared bonito" is recorded. The recorded data is then imported into the device and uploaded to an information processing device using a dedicated application. When uploading, the user sends the data by pressing the "upload" button in the application.

[0137] After receiving the data, the server separates the video data from the audio data. The video data is analyzed frame by frame to extract information such as the type of ingredients, how to use cooking utensils, and the cooking process. The audio data is converted to text and instructions and comments are extracted using natural language processing (NLP), for example, using the Google® Cloud Speech-to-Text API.

[0138] The server then trains the learning model based on the analysis results. The learning model uses AI technology, such as TensorFlow or PyTorch, to accurately reproduce cooking techniques and flavors.

[0139] When a user wants to recreate a specific dish or technique, they send a request to the server via their device. A request is made by pressing the "reproduce request" button within the application. At this time, by specifying the specific dish name and technique, for example, "I want to recreate seared bonito," the request is sent to the server.

[0140] The server receives requests from users and generates optimal cooking instructions and recipes. The generated information is provided to the device, and the user recreates the dish according to the instructions. For example, the next cooking steps are displayed within the application.

[0141] After the user has finished cooking, they can provide feedback on their results and impressions. By filling out and submitting the feedback form within the dedicated application, the feedback is sent to the server, which then analyzes and improves the model. This improves the accuracy of the reproducibility from the next time onwards.

[0142] Examples and prompts

[0143] For example, a user who wants to make "seared bonito" would follow these steps:

[0144] 1. The user records the cooking process on their smartphone and uploads the recording data to the server using the application.

[0145] 2. The server analyzes the data and processes the video and audio data separately.

[0146] 3. The server trains the AI ​​model on the analysis results.

[0147] 4. The user sends a request to recreate "Seared Bonito" within the app.

[0148] 5. The server generates optimal cooking instructions and recipes and provides them to the device.

[0149] 6. The user follows the provided steps to recreate the dish and receives feedback on the results.

[0150] Prompt Sentence Examples

[0151] "I want to upload video data of the cooking process and have the AI ​​model learn from it. Please tell me the specific steps and tools I need to use."

[0152] "Analyze the cooking steps for seared bonito and generate the optimal recipe."

[0153] "I want to provide feedback to improve the AI ​​model. How do I do that?"

[0154] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0155] Step 1: Obtaining Recording Data

[0156] The user uses a smartphone or video camera to record the cooking process. For example, the process of making "seared bonito" is used as the recording process. The input is the recorded data that includes the entire cooking process. Once the recording is finished, the recorded data is imported to a device such as a smartphone or PC. The output is the recorded data saved on the device.

[0157] Step 2: Upload your data

[0158] The device uses a dedicated application to upload recorded data to the server. The input is the recorded data stored on the device, and the output is the recorded data stored on the server. When the dedicated application is launched on the device and the "upload button" is pressed, the data is sent to the server.

[0159] Step 3: Preprocessing the data

[0160] The recorded data received by the server is converted into an analyzable format. First, the video data and audio data are separated. The input is the recorded data stored on the server, and the output is the separated video and audio data. Specifically, the video data is divided into 30 frames per second, and the audio data is saved as an audio file.

[0161] Step 4: Analyze the data

[0162] The server analyzes the preprocessed data. The input is the separated video and audio data, and the output is the extracted cooking process information and transcribed instructions and comments. Computer vision technology is used to analyze the video data for ingredients, utensil usage, cooking process, etc. Specifically, each frame is analyzed using image recognition technology (e.g., YOLO, OpenCV) to identify ingredients and utensils.

[0163] Step 5: Analyze the audio data

[0164] The server analyzes the voice data and converts it into text. The input is voice data, and the output is text data such as instructions or comments. Voice recognition technology (e.g., Google Cloud Speech-to-Text API) is used to convert the voice data into text format.

[0165] Step 6: Training the AI ​​model

[0166] The server trains the learning model with the analysis results. The input is the extracted cooking process information and text instructions and comments, and the output is the trained model. Specifically, the analysis results are fed into the learning model using TensorFlow or PyTorch to improve the model's reproducibility.

[0167] Step 7: Accepting a reproduction request

[0168] When a user wants to recreate a specific dish or technique, they send a request to the server via their device. The input is the user's request to recreate it, and the output is the server's acceptance of the request. The user presses the "recreate request" button in the dedicated application to send a request to recreate "seared bonito."

[0169] Step 8: Generate cooking instructions and recipes

[0170] The server receives requests from users and generates optimal cooking instructions and the necessary recipes. The input is the reproduction request and the trained model, and the output is the specific cooking instructions and recipes. Specifically, the server uses the trained model to calculate the optimal cooking method for the requested dish and provides it in text format.

[0171] Step 9: Provide and Reproduce to Users

[0172] The server provides the generated cooking instructions and recipes to the device. The input is the cooking instructions and recipe, and the output is the cooking instructions and recipe displayed on the user's device. The user recreates the dish by checking the steps for each step in the application.

[0173] Step 10: Submit your feedback

[0174] The user provides feedback on the cooking results and their impressions. The input is the user's feedback information, and the output is the feedback data sent to the server. After the user finishes cooking, they enter their impressions and suggestions for improvement in the feedback form within the application and submit it.

[0175] Step 11: Improve the model

[0176] The server receives the feedback and analyzes the information to improve the learning model. The input is the feedback data, and the output is an improved learning model. Specifically, the server fine-tunes the model parameters based on the feedback and adds new data for learning, thereby improving the recall accuracy.

[0177] (Application example 1)

[0178] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0179] Traditional methods of learning cooking techniques and recipes require specialized knowledge and numerous manuals, which takes a great deal of time and effort. Furthermore, chefs and cooking staff who want to learn new techniques and recipes have few efficient ways to learn, limiting their ability to improve their skills even when practicing in a brick-and-mortar restaurant. To solve these problems, a system is needed that can easily record and analyze the cooking process, allowing for efficient learning of techniques and recipes.

[0180] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0181] In this invention, the server includes: a means for a user to record the cooking process; a means for uploading the recorded data of the cooking process to the server; a means for the server to convert the received data into an analyzable format; a means for the server to analyze the video data and extract each process; a means for the server to analyze the audio data and extract instructions and comments; a means for the server to train an AI model based on the analysis results; a means for a user to send a reproduction request; a means for the server to generate specific cooking steps and recipes and provide them to a terminal; a means for a user to provide feedback on the cooking results; a means for the server to analyze the feedback and improve the AI ​​model; and a means used by chefs and staff in the restaurant to learn new techniques and recipes. This makes it possible to efficiently and effectively learn cooking techniques and new recipes.

[0182] A "user" is a person or cook who uses the system to record and analyze the cooking process and learn new techniques and recipes.

[0183] "Means for recording" refers to devices or equipment for recording the cooking process on video.

[0184] "Means for uploading data to a server" refers to the function or process of transmitting recorded video data to a server via the Internet.

[0185] "Means of converting into an analyzable format" refers to the process by which the server converts the data it receives into a format that can be processed by AI models or other data analysis tools.

[0186] "Means for analyzing video data" refers to a method for analyzing recorded video data and extracting each cooking process and step.

[0187] The "means for analyzing audio data" is a method for analyzing recorded audio data and extracting verbal instructions and comments.

[0188] "Means for training an AI model" refers to a method of inputting analyzed data into an artificial intelligence model to improve and optimize cooking techniques and recipes.

[0189] The "means for sending a reproduction request" is a function or process by which a user sends a request to the server to reproduce a particular dish or technique.

[0190] "Means for generating specific cooking steps and recipes and providing them to the terminal" refers to a method in which the server generates specific cooking steps and recipes based on the analysis and learning results and provides them to the user's terminal.

[0191] The "means for feeding back cooking results" refers to a function or process that reports the results of cooking performed by the user to the server.

[0192] "Means for analyzing feedback and improving the AI ​​model" refers to the process of analyzing feedback data obtained from users and updating and improving the model.

[0193] "Methods used by in-store chefs and staff to learn new techniques and recipes" refers to methods and tools that allow chefs and culinary staff in physical stores to efficiently learn new techniques and recipes.

[0194] DETAILED DESCRIPTION OF THE INVENTION The following describes an embodiment of the present invention.

[0195] The system's program is designed to allow users to learn and reproduce new techniques and recipes by recording the cooking process and uploading the data to a server for analysis. Specific hardware and software configurations and processing are explained below.

[0196] Hardware and software used

[0197] 1. Hardware:

[0198] Smartphone or head-mounted display (HMD): Used to record and play back the cooking process.

[0199] Server: Stores data, analyzes it, performs AI learning, and processes feedback.

[0200] 2. Software:

[0201] Recording and uploading application: An application installed on a smartphone or HMD to upload recorded data to a server.

[0202] Data analysis software: Video and audio data analysis is performed on the server, using, for example, OpenCV (video processing) and SpeechRecognition (audio processing) libraries.

[0203] AI Model: A generative AI model built using the PyTorch framework.

[0204] User Interface (UI): An application that allows for easy cooking instructions requests and feedback.

[0205] Data processing and data calculation

[0206] The user records the cooking process using a smartphone or HMD. The recorded data is sent to the server via an upload application. The data received by the server is processed as follows:

[0207] 1. Data preprocessing:

[0208] The server splits the video data into frames and extracts detailed information such as the type of ingredients, the use of cooking utensils, the cooking process, etc. It analyzes the video using the OpenCV library.

[0209] At the same time, the voice data is analyzed and instructions and comments are converted into text using the SpeechRecognition library.

[0210] 2. AI Learning:

[0211] The analyzed data is fed into an AI model that learns cooking techniques and recipes. Specifically, PyTorch is used to train and update the model, which reflects subtleties such as cutting, heat control, and seasoning use.

[0212] 3. Providing cooking instructions and recipes:

[0213] When a user requests the reproduction of a specific dish, the server uses an AI model to generate optimal cooking instructions and recipes and provides them to the device.

[0214] 4. Feedback Processing:

[0215] Users provide feedback on their cooking results, and the server analyzes the data to further improve the AI ​​model.

[0216] Specific use cases

[0217] For example, if a user wants to perfectly recreate "seared bonito," they would follow these steps:

[0218] 1. Record the cooking process of seared bonito using a smartphone or HMD.

[0219] 2. Use an upload application to send the recorded data to the server.

[0220] 3. The server analyzes the video and audio data and trains the AI ​​model to learn the necessary cooking techniques and recipe details.

[0221] 4. When a user requests to recreate seared bonito, the server generates the optimal steps and recipe and provides it to the device.

[0222] 5. The user follows the instructions to cook the food and provides feedback on the results.

[0223] Prompt Sentence Examples

[0224] The user can request cooking instructions using prompts such as:

[0225] Instruct the AI ​​to reconstruct the cooking process for "Seared Bonito" and provide detailed step-by-step instructions, including how to cut the bonito, how to cook it, and how to press it into a firm consistency.

[0226] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0227] Step 1:

[0228] The user records the cooking process using a smartphone or head-mounted display (HMD). Video and audio are recorded during the recording. The input is real-time video and audio of the cooking process, and the output is the recorded video and audio files. Specifically, the entire cooking process is recorded by pressing the start recording button.

[0229] Step 2:

[0230] The device uploads the recorded data to the server. After the user finishes recording, they press the upload button and the video and audio files are sent to the server. The input is the recorded video and audio files, and the output is the files saved on the server. Specifically, the upload process runs in the background and the progress is displayed to the user.

[0231] Step 3:

[0232] The server converts the received data into an analyzable format. The server separates the video and audio data and temporarily stores them. The input is the original uploaded video and audio files, and the output is the separated video and audio data. Specifically, the video frames are extracted as continuous still images, and the audio data is stored as a separate audio file.

[0233] Step 4:

[0234] The server analyzes the video data and extracts each process. It uses OpenCV to analyze each frame and automatically extracts the type of ingredients, how to use cooking utensils, and steps in the cooking process. The input is the separated video data, and the output is the analyzed process information. Specifically, it performs object recognition on a frame-by-frame basis to classify ingredients and cooking utensils.

[0235] Step 5:

[0236] The server analyzes the audio data and extracts instructions and comments. It uses the SpeechRecognition library to convert the audio into text and extracts cooking instructions and explanations. The input is separated audio data, and the output is text data. Specifically, the audio file is read and converted to text through a speech recognition API.

[0237] Step 6:

[0238] The server trains the AI ​​model with the analysis results. The analyzed video and audio data is used to train the generative AI model on cooking techniques and recipes. The input is the analyzed process information and instruction text, and the output is an updated AI model. Specifically, the PyTorch framework is used to train the model based on the analyzed data.

[0239] Step 7:

[0240] The user submits a reproduction request. If the user wants to reproduce a specific dish or technique, the request is sent to the server via the device. The input is the reproduction request information, and the output is a notification that the request has been accepted. Specifically, the user enters the name of the dish or a specific technique in the app and submits it.

[0241] Step 8:

[0242] The server generates specific cooking steps and recipes and provides them to the device. Based on the learning results of the AI ​​model, the server generates optimal steps and provides them to the user. The input is the reproduction request and the trained AI model, and the output is specific cooking steps and recipe information. Specifically, a detailed recipe generated based on the prompt text is displayed on the user interface.

[0243] Step 9:

[0244] The user provides feedback on the cooking results. The results of the recreated dish are sent to the server, and the success rate of the cooking and areas for improvement are reported. The input is feedback information (photos, comments), and the output is feedback data stored on the server. Specifically, after cooking, the results are photographed and entered, and sent via the app.

[0245] Step 10:

[0246] The server analyzes the feedback and improves the AI ​​model. The feedback data is analyzed and reflected in the next learning. The input is feedback data from the user, and the output is a further improved AI model. Specifically, the feedback is aggregated and analyzed, and the model's hyperparameters are adjusted and additional learning is performed.

[0247] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0248] The system of the present invention allows users to record the cooking process, analyze the data, learn cooking techniques, and then reproduce the recipes. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system aims to increase user satisfaction. An embodiment of this system will be described in detail below.

[0249] System configuration

[0250] The system mainly consists of the following elements:

[0251] 1. User Recording Method

[0252] 2. Data upload method

[0253] 3. Server data analysis method

[0254] 4. AI learning methods

[0255] 5. How to Request Reproduction

[0256] 6. Feedback channels

[0257] 7. Emotion Engine

[0258] Program processing

[0259] The system program does the following:

[0260] Acquiring Recording Data

[0261] The system provides a means for users to record the cooking process. Users can use their smartphones or video cameras to record the cooking process of top chefs in detail. After the recording is complete, users can import the recorded data to their devices.

[0262] Uploading data

[0263] The device uploads the recorded data to the server. Data can be easily sent to the server using a dedicated application.

[0264] Data preprocessing and analysis

[0265] The server converts the received recorded data into an analyzable format. First, the video data and audio data are separated and processed separately. The video data is analyzed frame by frame to extract information such as the type of ingredients, how to use cooking utensils, and the cooking process. Meanwhile, the audio data is converted into text to extract the chef's instructions and comments.

[0266] AI Learning

[0267] The server analyzes the data and trains the AI ​​model, which allows it to accurately reproduce cooking techniques and flavors. Specifically, even minute details such as cutting methods, heat levels, and the use of seasonings are reflected in the model.

[0268] Reproduction requests and submissions

[0269] When a user wants to recreate a specific dish or technique, they send a request to the server through their device. The server generates cooking instructions and recipes based on the request and provides them to the device. The user can then follow these instructions to recreate the dish.

[0270] Feedback and model improvement

[0271] The user recreates the dish and sends feedback to the emotion engine via their device. The emotion engine analyzes the user's emotions (for example, satisfaction or difficulties) and sends this information to the server. The server receives the feedback and reflects it in the AI ​​model, further improving the accuracy of future reproductions.

[0272] Emotion Engine Operation

[0273] The emotion engine recognizes the emotions felt by the user regarding the cooking process and the results. For example, it uses facial recognition and voice analysis technology to detect the joy, difficulty, and satisfaction felt by the user in real time, and sends this data to the server. The server then analyzes this information and reflects it in the AI ​​model. This allows the system to provide optimal cooking procedures and recipes that take the user's emotions into account.

[0274] Specific examples

[0275] For example, if a user wants to make "seared bonito", the steps would be as follows:

[0276] 1. A user records a chef "searing bonito."

[0277] 2. The device uploads the recorded data to the server.

[0278] 3. The server analyzes information from the video data, such as the steps for cutting the bonito, how well it is cooked, and how to pound it, and extracts supplementary explanations from the audio data.

[0279] 4. The server trains the AI ​​model on the analysis results.

[0280] 5. The user sends a request to recreate "Seared Bonito."

[0281] 6. The server generates the optimal cooking procedure and necessary recipes and provides them to the terminal.

[0282] 7. The user cooks according to the provided instructions, and the emotion engine analyzes the user's emotions and sends them to the server.

[0283] 8. The server analyzes the emotional feedback and improves the AI ​​model.

[0284] This allows users to efficiently learn first-class cooking techniques and reproduce them in an optimal way, taking into account their emotions during the process. This system will help pass on traditional cooking techniques and improve user satisfaction.

[0285] The processing flow will be explained below.

[0286] Step 1:

[0287] The user records the cooking process. The user uses a smartphone or video camera to record the cooking process of a top chef in detail. The recording is done so that every process is clearly visible.

[0288] Step 2:

[0289] The user imports the recorded data to the device. Once recording is complete, the user transfers the video file to a device such as a smartphone or PC and opens the dedicated application.

[0290] Step 3:

[0291] The device uploads the recorded data to the server. A dedicated application is used to select the recorded data and issue an upload command to the server. The data is then securely transferred to the server.

[0292] Step 4:

[0293] The server receives the recorded data and prepares it for conversion into an analyzable format.

[0294] Step 5:

[0295] The server converts the recorded data into an analyzable format, splits the video file into frames, and separates the video data from the audio data.

[0296] Step 6:

[0297] The server analyzes the video data, analyzing each individual frame to extract each step of the cooking process (e.g., type of ingredient, cutting method, heat level), and even to identify the smallest actions and steps visible in the video.

[0298] Step 7:

[0299] The server analyzes the audio data, converts it into text, and extracts the chef's instructions and comments, providing a detailed textual description of each process.

[0300] Step 8:

[0301] The server trains the AI ​​model with the analysis results, and the extracted text from the video and audio data is input into the AI ​​model, which then learns how to reproduce the cooking techniques and flavors.

[0302] Step 9:

[0303] The user sends a reproduction request to the server. The user sends a request to reproduce a specific dish (e.g., "seared bonito") to the server via the terminal.

[0304] Step 10:

[0305] The server generates cooking instructions and recipes based on the request, calculates the optimal steps based on an AI model, and generates detailed recipes and cooking instructions.

[0306] Step 11:

[0307] The terminal provides the cooking instructions and recipes to the user, who can then view the cooking instructions and recipes obtained from the server through the terminal.

[0308] Step 12:

[0309] The user follows the instructions to cook the food, and then recreates the dish based on the instructions displayed on the device.

[0310] Step 13:

[0311] The emotion engine analyzes the user's emotions during the cooking process. The emotion engine recognizes emotions from the user's facial expressions and voice, and analyzes the satisfaction and difficulties during the cooking process in real time.

[0312] Step 14:

[0313] The emotion engine sends the analysis results to the server, which then transmits the emotion data to the server, which then reflects it in the cooking process.

[0314] Step 15:

[0315] The user provides feedback on the cooking results, sending their evaluation of the cooking results and taste to the server via their device.

[0316] Step 16:

[0317] The server receives and analyzes the feedback sent by users and uses it to improve the AI ​​model.

[0318] Step 17:

[0319] The server improves the AI ​​model, adjusting it based on feedback and emotion data to improve the accuracy of the next replay.

[0320] This process allows users to accurately learn and recreate traditional cooking techniques, and also provides an optimal cooking environment for users by taking into account their emotions during the cooking process.

[0321] Example 2

[0322] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0323] Conventional cooking skill learning systems have difficulty effectively replicating professional cooking techniques, and have been unable to properly utilize feedback to ensure users get emotionally satisfying results. Furthermore, there has been no means to analyze users' emotions in real time and reflect them in the cooking process.

[0324] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for a user to record the cooking process, a means for uploading the recorded cooking process data to the server, a means for the server to convert the received data into an analyzable format, a means for the server to analyze the video data and extract each process, a means for the server to analyze the audio data and extract instructions and comments, a means for the server to train an AI model based on the analysis results, a means for a user to send a reproduction request, a means for the server to generate specific cooking steps and recipes and provide them to the terminal, a means for the user to provide feedback on the cooking results, a means for the server to analyze the feedback and improve the AI ​​model, and an emotion analysis means, which recognizes the user's emotions in real time and transmits the analysis results to the server. This makes it possible to provide optimal cooking instructions that take the user's emotions into consideration, solving problems associated with conventional systems and enabling users to learn and reproduce cooking techniques with greater accuracy.

[0325] A "user" is an individual or entity who uses the system to learn and replicate cooking techniques.

[0326] "Recorded data" refers to video and audio data used by a user to record the cooking process.

[0327] The "server" is a central computer system that receives the recorded data, analyzes it, trains the AI ​​model, and provides the results to the user.

[0328] "Data analysis" refers to the process in which the server identifies the video and audio data it receives and extracts the cooking process, instructions, and comments.

[0329] An "AI model" is an artificial intelligence algorithm or system that learns based on analysis results and provides users with optimal cooking procedures and recipes.

[0330] A "reproduction request" is an act in which a user sends a request to the server that they would like to learn or reproduce a particular dish or technique.

[0331] "Emotion analysis" refers to the emotion analysis means recognizing the user's emotions in real time and sending the analysis results to the server.

[0332] "Feedback" includes the user reporting the results of their cooking reproduction to the system and the results of subsequent sentiment analysis.

[0333] "Cooking instructions" refers to the specific cooking methods and steps generated by the server.

[0334] A "recipe" is a set of detailed instructions detailing the ingredients and quantities needed to prepare a particular dish, as well as the cooking method involved.

[0335] "Emotion analysis means" refers to technology or devices for detecting and analyzing a user's emotions in real time, and uses facial recognition technology and voice analysis technology.

[0336] MODE FOR CARRYING OUT THE INVENTION

[0337] The system of the present invention allows users to record the cooking process, analyze the data, learn cooking techniques, and then reproduce the recipes. Furthermore, by combining it with emotion analysis means that recognizes the user's emotions, the system aims to increase user satisfaction.

[0338] This system consists of a user recording means, a data uploading means, a server data analysis means, an AI learning means, a reproduction request means, a feedback means, and an emotion analysis means.

[0339] Acquiring Recording Data

[0340] Users record the cooking process using a recording device such as a smartphone or video camera. This process includes everything from preparation to completion, and detailed footage is required. For example, when making "seared bonito," users will record in detail how they cut the bonito, how well it is cooked, and how they sear it.

[0341] Uploading data

[0342] Once the recording is complete, the user imports the recording data to their device and then uploads it to the server using a dedicated application. This process can be completed with just a few clicks.

[0343] Data preprocessing and analysis

[0344] The server converts the received video data into an analyzable format. During this process, the video data and audio data are separated and processed separately. The video data is analyzed frame by frame to extract information such as the type of ingredients, the use of cooking utensils, and the cooking process. For example, the way ingredients are cut and cooked can be identified. Meanwhile, the audio data is converted to text to extract the chef's instructions and comments.

[0345] AI Learning

[0346] The server then trains the AI ​​model with the analysis results. This training process includes detailed cooking techniques, including how to cut food, how much heat to use, and how to use seasonings, so that the AI ​​model can accurately reproduce detailed cooking techniques and flavors. This allows the AI ​​model to provide users with optimal cooking procedures and recipes.

[0347] Reproduction requests and submissions

[0348] When a user wants to recreate a specific dish or technique, they send a request to the server via their device. Based on the request, the server generates the optimal cooking steps and necessary recipes and provides them to the device. The user can then recreate the dish based on this information. An example of a specific prompt sentence is, "Please tell me the optimal cooking steps and recipe for making bonito tataki. Please also provide voice instructions."

[0349] Feedback and model improvement

[0350] After the user has finished recreating the dish, they provide feedback to the emotion analysis means via their device. The emotion analysis means analyzes the user's emotions and sends the results to the server. The server receives the feedback and uses it to improve the AI ​​model. For example, the emotion analysis means can analyze the user's satisfaction and difficulties and reflect that information in the AI ​​model.

[0351] Operation of emotion analysis means

[0352] The emotion analysis means uses facial recognition and voice analysis technologies to detect the user's emotions in real time and transmits the data to the server, which enables the system to provide optimal cooking instructions and recipes that take the user's emotions into consideration.

[0353] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0354] Step 1: Obtaining the recording data

[0355] Users can use their smartphones or video cameras to record the cooking process in detail, for example, the entire cooking process of "seared bonito."

[0356] Input: Video and audio data stored on the recording device.

[0357] How it works: The user starts recording and then imports the recording to their device after shooting.

[0358] Output: Recording data file saved on the device.

[0359] Step 2: Upload your data

[0360] The device uploads the recorded data to the server using a dedicated application.

[0361] Input: Recording data files saved on the device.

[0362] How it works: The user launches the dedicated app and clicks the "upload" button to send data to the server.

[0363] Output: Recording data file saved on the server.

[0364] Step 3: Data preprocessing and analysis

[0365] The server converts the recorded data into an analyzable format, separating the video and audio data and processing each separately.

[0366] Input: Recording data files stored on the server.

[0367] How it works: The server separates the video data from the audio data and analyzes the video data frame by frame. Specifically, it extracts information such as the type of ingredients, how to use cooking utensils, and the cooking process. Meanwhile, the audio data is converted into text and the chef's instructions and comments are extracted.

[0368] Output: Analyzable video data and audio data in text format.

[0369] Step 4: AI training

[0370] The data analyzed by the server is trained into an AI model.

[0371] Input: Analyzable video data and audio data in text format.

[0372] How it works: The server inputs the data into the AI ​​model and begins the learning process, incorporating even the smallest details like cutting techniques, heat levels, and seasonings.

[0373] Output: The trained AI model.

[0374] Step 5: Request and provide a reproduction

[0375] When a user wants to recreate a particular dish or technique, they send a request to the server through their device.

[0376] Input: The reproduction request submitted by the user.

[0377] How it works: The server generates the optimal cooking instructions and recipe based on the request and sends it to the device. For example, if the request is "I want to recreate seared bonito," the server will provide the instructions and recipe for that.

[0378] Output: Specific cooking instructions and recipe displayed on the user's device.

[0379] Step 6: Feedback and model improvement

[0380] After the user has completed recreating the dish, they provide feedback to the sentiment analysis means.

[0381] Input: User-provided feedback data.

[0382] Operation: The emotion analysis means analyzes the feedback data and sends it to the server. For example, the satisfaction or difficulties felt during the cooking process are specifically input.

[0383] Output: Feedback analysis data sent to the server.

[0384] Step 7: Sentiment analysis method in action

[0385] The emotion analysis means recognizes the user's emotions in real time and transmits the analysis results to the server.

[0386] Input: Real-time video and audio data of the user cooking.

[0387] How it works: The emotion analysis unit uses face recognition and voice analysis technology to detect the user's emotions and sends the data to the server. For example, it analyzes emotions such as joy or difficulty.

[0388] Output: Real-time emotion data sent to the server.

[0389] (Application example 2)

[0390] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0391] Conventional food delivery systems lacked the means to effectively manage and improve food quality and user satisfaction. In particular, there were many challenges in the reproducibility of cooking processes, the skill acquisition of delivery staff, and the management of customer feedback. The purpose of this invention is to solve these problems, improve the quality of delivery services, and increase the satisfaction of customers and delivery staff.

[0392] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to record the cooking process and upload the data, means for analyzing the uploaded recorded data and extracting each process from the video data, means for extracting instructions and comments from the audio data, means for training the generative AI model based on the analysis results, means for a user to send a reproduction request, generate specific cooking steps and recipes, and provide them to the terminal, means for a user to provide feedback on the cooking results and for the server to analyze the feedback and improve the generative AI model, means for a delivery staff member to record the cooking process, means for uploading the recorded data to a cloud server, means for the cloud server to analyze the uploaded recorded data and extract cooking skills, and means for analyzing the emotions of the delivery staff member and sending feedback to the server. This improves the reproducibility of the cooking process, enabling delivery staff to acquire skills and customer feedback to be managed more effectively.

[0393] "Means for users to record the cooking process" refers to devices or methods that allow users to record the cooking process as video.

[0394] The "means for uploading recorded data of the cooking process to the server" is a means for transmitting video data of the cooking process recorded by the user to the server via the Internet.

[0395] "Means for converting data received by the server into an analyzable format" refers to a method or algorithm for converting data uploaded to the server into a format that can be easily analyzed.

[0396] "Means for the server to analyze video data and extract each process" refers to a method or system for analyzing the content of video data and identifying and extracting each step of food preparation and cooking.

[0397] "Means for the server to analyze the audio data and extract instructions and comments" refers to methods and technologies for identifying cooking instructions and comments from the recorded audio data and extracting them as text.

[0398] "Means for the server to train the generated AI model on the analysis results" refers to a method for training the AI ​​model generated based on the analysis results extracted from video data and audio data on new data.

[0399] The "means by which a user sends a reproduction request" refers to an interface or method by which a user can send a request to the server when they want to reproduce a particular dish or technique.

[0400] "Means for the server to generate specific cooking procedures and recipes and provide them to the terminal" refers to a system that generates optimal cooking procedures and recipes based on the user's reproduction request and provides them to the user's terminal.

[0401] "Means for users to provide feedback on cooking results" refers to methods or tools that allow users to return the results of their recreated dishes to the server as their impressions or evaluations.

[0402] "Means for the server to analyze feedback and improve the generative AI model" refers to a system or method for analyzing feedback from users and improving the generative AI model based on the results.

[0403] "Means for delivery staff to record the cooking process" refers to devices or methods that delivery staff use to record the cooking process of food on video.

[0404] "Means for uploading recorded data to a cloud server" refers to means for saving recorded data to a cloud server via the Internet.

[0405] "Means for the cloud server to analyze the uploaded video data and extract cooking techniques" refers to a method or system for the cloud server to analyze the video data received and identify and extract cooking techniques and processes.

[0406] "Means for analyzing the emotions of delivery staff and sending feedback to a server" refers to a method or technology for analyzing the emotions of delivery staff from video and audio data and sending the results as feedback to a server.

[0407] This invention aims to develop a system that aims to improve customer satisfaction and control the quality of the cooking process in food delivery services. The system consists of the following components:

[0408] 1. User recording method: The user or delivery staff uses a recording device such as a smartphone to record the cooking process. The video data contains detailed information such as the cooking steps, ingredients used, and necessary cooking utensils.

[0409] 2. Data uploading method: Users use a dedicated application to upload recorded data to the cloud server, which allows data to be easily sent to the server via the Internet.

[0410] 3. Server data analysis method: The server converts the received video data into an analyzable format. Specifically, the video data is divided into frames and image recognition technology (e.g., OpenCV) is used to analyze the type of ingredients and cooking process. Additionally, audio data is converted into text using speech recognition technology (e.g., Google Speech-to-Text) to extract cooking comments and instructions.

[0411] 4. AI learning method: The generative AI model learns cooking techniques based on the analyzed data. This model is built using a machine learning library (e.g., TensorFlow). The model reflects detailed cooking techniques, such as how to cut ingredients, how to control heat, and how to use seasonings.

[0412] 5. Reproduction request method: When a user wants to reproduce a specific dish, they send a request to the server through a dedicated application. Based on this request, the server generates the optimal cooking procedure and recipe and provides it to the user's device.

[0413] 6. Feedback method: The user provides feedback on the cooking results to the server through the application. The feedback is analyzed using facial recognition technology (e.g., DeepFace) to quantify the user's satisfaction.

[0414] 7. Emotion Engine: The emotion engine recognizes the user's emotions in real time and transmits the user's joy, difficulty, and satisfaction to the server, which analyzes this data and improves the generative AI model.

[0415] The above system will improve the reproducibility of the cooking process, enable delivery staff to acquire skills, and enable effective management of customer feedback. It will also optimize the recipes and cooking procedures provided to users, which is expected to improve customer satisfaction.

[0416] Examples:

[0417] For example, a delivery staff member serves a highly rated dish to a customer and records the cooking process. The video data is uploaded to a cloud server, which analyzes the data and learns the cooking process and techniques. If another user requests to make the same dish, the server will provide the optimal cooking procedure and recipe, and the user will provide feedback on the results, and the emotion engine will analyze the customer's satisfaction level.

[0418] Example of an input prompt:

[0419] "I want to analyze videos of the cooking process taken by delivery staff and collect emotional feedback. Please generate a report that includes the analysis results and emotional feedback."

[0420] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0421] Step 1:

[0422] The user records the cooking process using a means to record the cooking process. Specifically, the user or delivery staff uses a smartphone or video camera to record the cooking process from start to finish. The input is video data of the cooking process, and the output is the recorded video file.

[0423] Step 2:

[0424] The recorded cooking process data is uploaded to the server. The user uses a dedicated upload application to send the recorded data to the cloud server. The input is a video file, and the output is video data on the cloud server.

[0425] Step 3:

[0426] The server converts the received data into an analyzable format. The cloud server divides the received video file into frames and separates it into image data and audio data. It also converts the audio data into text. The input is video data on the cloud server, and the output is image data and text data.

[0427] Step 4:

[0428] The server analyzes the video data and extracts each process. Using machine learning and image recognition technology (e.g., OpenCV), the server identifies the type of ingredients, cooking utensils, and cooking process from the image data for each frame. The input is the image data for each frame, and the output is the extracted cooking process information.

[0429] Step 5:

[0430] The server analyzes the voice data and extracts instructions and comments. The server uses voice recognition technology (e.g., Google Speech-to-Text) to convert the voice data into text and identify cooking instructions and comments. The input is the voice data, and the output is the extracted text data.

[0431] Step 6:

[0432] The server trains the generative AI model with the analysis results. The server inputs the analyzed cooking process information and text data into the generative AI model (e.g., TensorFlow) and trains the model to learn new cooking techniques. The input is the cooking process information and text data, and the output is an updated generative AI model.

[0433] Step 7:

[0434] The system uses a means for users to send a reproduction request. A user sends a request to reproduce a specific dish or cooking technique through a dedicated application. The input is the reproduction request, and the output is the request data sent to the server.

[0435] Step 8:

[0436] The server generates specific cooking steps and recipes and provides them to the device. The server generates optimal cooking steps and recipes based on the generative AI model and provides this information to the user's device. The input is the generative AI model and the user's reproduction request, and the output is the cooking steps and recipe provided to the user's device.

[0437] Step 9:

[0438] The system uses a means for users to provide feedback on their cooking results. Users cook and provide feedback on the results through a dedicated application. The input is the user's cooking results, and the output is feedback data to the server.

[0439] Step 10:

[0440] The server analyzes the feedback and improves the generative AI model. The server analyzes the user feedback data, evaluates customer satisfaction using an emotion engine, and reflects the results in the generative AI model. The input is the feedback data, and the output is an improved generative AI model.

[0441] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0442] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0443] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0444] [Second embodiment]

[0445] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0446] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0447] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0448] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0449] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0450] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0451] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0452] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0453] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0454] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0455] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0456] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0457] The system of the present invention allows users to easily record and analyze the cooking process, and based on that, learn and reproduce cooking techniques. An embodiment of the system will be described in detail below.

[0458] System configuration

[0459] The system mainly consists of the following elements:

[0460] 1. User Recording Method

[0461] 2. Data upload method

[0462] 3. Server data analysis method

[0463] 4. AI learning methods

[0464] 5. How to Request Reproduction

[0465] 6. Feedback channels

[0466] Program processing

[0467] The system program does the following:

[0468] Acquiring Recording Data

[0469] The user uses a means to record the cooking process. For example, the user uses a smartphone or video camera to record the cooking process. After the recording is finished, the user imports the recorded data into the device.

[0470] Uploading data

[0471] The device uploads the recorded data to the server. Data can be easily sent to the server using a dedicated application.

[0472] Data preprocessing and analysis

[0473] The server converts the received recorded data into an analyzable format. First, the video data and audio data are separated and processed separately. The video data is analyzed frame by frame to extract information such as the type of ingredients, how to use cooking utensils, and the cooking process. Meanwhile, the audio data is converted into text to extract the chef's instructions and comments.

[0474] AI Learning

[0475] The server analyzes the data and trains the AI ​​model, which allows it to accurately reproduce cooking techniques and flavors. Specifically, even minute details such as cutting methods, heat levels, and the use of seasonings are reflected in the model.

[0476] Reproduction requests and submissions

[0477] When a user wants to recreate a specific dish or technique, they send a request to the server through their device. The server generates cooking instructions and recipes based on the request and provides them to the device. The user can then follow these instructions to recreate the dish.

[0478] Feedback and model improvement

[0479] The user recreates the dish and provides feedback on the results. The device then sends the feedback to the server, which analyzes the information and improves the AI ​​model, further improving the accuracy of future attempts.

[0480] Specific examples

[0481] For example, if a user wants to make "seared bonito", the steps would be as follows:

[0482] 1. A user records a chef "searing bonito."

[0483] 2. The device uploads the recorded data to the server.

[0484] 3. The server analyzes information from the video data, such as the steps for cutting the bonito, how well it is cooked, and how to pound it, and extracts supplementary explanations from the audio data.

[0485] 4. The server trains the AI ​​model on the analysis results.

[0486] 5. The user sends a request to recreate "Seared Bonito."

[0487] 6. The server generates the optimal cooking procedure and necessary recipes and provides them to the terminal.

[0488] 7. The user follows the provided instructions to cook the food and provides feedback on the results.

[0489] 8. The server analyzes the feedback and improves the AI ​​model.

[0490] This allows users to efficiently learn and reproduce first-class cooking techniques. This system also enables the efficient and accurate inheritance of traditional cooking techniques.

[0491] The processing flow will be explained below.

[0492] Step 1:

[0493] The user records the cooking process. The user uses a smartphone or video camera to record the cooking process of a top chef in detail. The recording is done so that every process is clearly visible.

[0494] Step 2:

[0495] The user imports the recorded data to the device. Once recording is complete, the user transfers the video file to a device such as a smartphone or PC and opens the dedicated application.

[0496] Step 3:

[0497] The device uploads the recorded data to the server. A dedicated application is used to select the recorded data and issue an upload command to the server. The data is then securely transferred to the server.

[0498] Step 4:

[0499] The server receives the recorded data and prepares it for conversion into an analyzable format.

[0500] Step 5:

[0501] The server converts the recorded data into an analyzable format, splits the video file into frames, and separates the video data from the audio data.

[0502] Step 6:

[0503] The server analyzes the video data, analyzing each individual frame to extract each step of the cooking process (e.g., type of ingredient, cutting method, heat level), and even to identify the smallest actions and steps visible in the video.

[0504] Step 7:

[0505] The server analyzes the audio data, converts it into text, and extracts the chef's instructions and comments, providing a detailed textual description of each process.

[0506] Step 8:

[0507] The server trains the AI ​​model with the analysis results, and the extracted text from the video and audio data is input into the AI ​​model, which then learns how to reproduce the cooking techniques and flavors.

[0508] Step 9:

[0509] The user sends a reproduction request to the server. The user sends a request to reproduce a specific dish (e.g., "seared bonito") to the server via the terminal.

[0510] Step 10:

[0511] The server generates cooking instructions and recipes based on the request, calculates the optimal steps based on an AI model, and generates detailed recipes and cooking instructions.

[0512] Step 11:

[0513] The terminal provides the cooking instructions and recipes to the user, who can then view the cooking instructions and recipes obtained from the server through the terminal.

[0514] Step 12:

[0515] The user follows the instructions to cook the food, and then recreates the dish based on the instructions displayed on the device.

[0516] Step 13:

[0517] The user provides feedback on the cooking results, sending their evaluation of the cooking results and taste to the server via their device.

[0518] Step 14:

[0519] The server receives and analyzes the feedback sent by users and uses it to improve the AI ​​model.

[0520] Step 15:

[0521] The server improves the AI ​​model, adjusting it based on the feedback to improve the accuracy of the next replay.

[0522] This process allows users to accurately learn and recreate traditional cooking techniques.

[0523] Example 1

[0524] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0525] Traditional cooking skill learning methods often rely on watching videos or reading books, making it difficult to fully understand the details of the actual cooking process. It is particularly difficult to accurately reproduce the techniques of professional chefs, making it difficult for learners to improve their skills. Furthermore, there are limited ways to improve one's own cooking skills through feedback, making it difficult to efficiently acquire new cooking skills.

[0526] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0527] In this invention, the server includes means for a user to record the cooking process, means for uploading data of the recorded cooking process to an information processing device, means for the information processing device to convert the received data into an analyzable format, means for the information processing device to analyze the video data and extract each process, means for the information processing device to analyze the audio data and extract instructions and comments, means for the information processing device to train a learning model based on the analysis results, means for a user to send a reproduction request, means for the information processing device to generate specific cooking steps and recipes and provide them to the terminal, means for a user to provide feedback on the cooking results, and means for the information processing device to analyze the feedback and improve the learning model. This enables users to accurately learn and reproduce professional cooking techniques and improve their own skills.

[0528] "User" refers to any individual or legal entity that records the cooking process and uploads the data to the system.

[0529] The "cooking process" refers to the series of steps involved in preparing ingredients, cooking, and finishing.

[0530] "Recording means" refers to a device for recording video and audio, such as a smartphone or video camera.

[0531] "Data" refers to video data and audio data acquired by a recording means.

[0532] "Information processing device" refers to a device that analyzes, processes, and stores data, such as a server or cloud service.

[0533] "Uploading means" refers to the means by which a user sends recorded data to an information processing device, including dedicated applications and web forms.

[0534] The term "analyzable format" refers to a format required for an information processing device to process data appropriately.

[0535] "Video data" refers to video frame information captured by a recording means.

[0536] "Audio data" refers to audio information captured by a recording means.

[0537] The "means for extracting each process" refers to a means for analyzing and identifying each stage of cooking from video data and audio data.

[0538] "Instructions and comments" refers to statements and explanations made by the chef or user regarding the cooking process.

[0539] "Learning model" refers to analytical algorithms and data models that use artificial intelligence to learn cooking techniques and processes.

[0540] A "reproduction request" refers to a user requesting the system to reproduce a specific dish or technique.

[0541] "Cooking instructions and recipes" refers to the specific steps and ingredients needed to recreate a dish.

[0542] "Terminal" refers to a device used by a user, such as a smartphone, tablet, or PC.

[0543] "Feedback" refers to information such as cooking results and impressions that users provide to the system.

[0544] "Analysis means" refers to the algorithms and techniques used by the information processing device to process and analyze data.

[0545] The system of the present invention allows users to easily record and analyze the cooking process, and then learn and reproduce cooking techniques based on the recorded footage. To implement this system, the following hardware and software are required: a user's smartphone, a video camera, a personal computer, a dedicated application, an information processing device (server), and a learning model.

[0546] Hardware and software used

[0547] 1. Smartphone / Video Camera: Used as a recording medium.

[0548] 2. Computer / dedicated application: Used to import and upload data.

[0549] 3. Information processing device (server): Used to analyze data, manage learning models, generate cooking instructions and recipes, and analyze feedback.

[0550] 4. Learning model: A model that uses AI technology and frameworks such as TensorFlow or PyTorch.

[0551] Specific actions and processes

[0552] The user uses a smartphone or video camera to record the cooking process. For example, the process of making "seared bonito" is recorded. The recorded data is then imported into the device and uploaded to an information processing device using a dedicated application. When uploading, the user sends the data by pressing the "upload" button in the application.

[0553] After receiving the data, the server separates the video data from the audio data. The video data is analyzed frame by frame to extract information such as the type of ingredients, how to use cooking utensils, and the cooking process. The audio data is converted to text and instructions and comments are extracted using natural language processing (NLP), for example, using the Google Cloud Speech-to-Text API.

[0554] The server then trains the learning model based on the analysis results. The learning model uses AI technology, such as TensorFlow or PyTorch, to accurately reproduce cooking techniques and flavors.

[0555] When a user wants to recreate a specific dish or technique, they send a request to the server via their device. A request is made by pressing the "reproduce request" button within the application. At this time, by specifying the specific dish name and technique, for example, "I want to recreate seared bonito," the request is sent to the server.

[0556] The server receives requests from users and generates optimal cooking instructions and recipes. The generated information is provided to the device, and the user recreates the dish according to the instructions. For example, the next cooking steps are displayed within the application.

[0557] After the user has finished cooking, they can provide feedback on their results and impressions. By filling out and submitting the feedback form within the dedicated application, the feedback is sent to the server, which then analyzes and improves the model. This improves the accuracy of the reproducibility from the next time onwards.

[0558] Examples and prompts

[0559] For example, a user who wants to make "seared bonito" would follow these steps:

[0560] 1. The user records the cooking process on their smartphone and uploads the recording data to the server using the application.

[0561] 2. The server analyzes the data and processes the video and audio data separately.

[0562] 3. The server trains the AI ​​model on the analysis results.

[0563] 4. The user sends a request to recreate "Seared Bonito" within the app.

[0564] 5. The server generates optimal cooking instructions and recipes and provides them to the device.

[0565] 6. The user follows the provided steps to recreate the dish and receives feedback on the results.

[0566] Prompt Sentence Examples

[0567] "I want to upload video data of the cooking process and have the AI ​​model learn from it. Please tell me the specific steps and tools I need to use."

[0568] "Analyze the cooking steps for seared bonito and generate the optimal recipe."

[0569] "I want to provide feedback to improve the AI ​​model. How do I do that?"

[0570] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0571] Step 1: Obtaining Recording Data

[0572] The user uses a smartphone or video camera to record the cooking process. For example, the process of making "seared bonito" is used as the recording process. The input is the recorded data that includes the entire cooking process. Once the recording is finished, the recorded data is imported to a device such as a smartphone or PC. The output is the recorded data saved on the device.

[0573] Step 2: Upload your data

[0574] The device uses a dedicated application to upload recorded data to the server. The input is the recorded data stored on the device, and the output is the recorded data stored on the server. When the dedicated application is launched on the device and the "upload button" is pressed, the data is sent to the server.

[0575] Step 3: Preprocessing the data

[0576] The recorded data received by the server is converted into an analyzable format. First, the video data and audio data are separated. The input is the recorded data stored on the server, and the output is the separated video and audio data. Specifically, the video data is divided into 30 frames per second, and the audio data is saved as an audio file.

[0577] Step 4: Analyze the data

[0578] The server analyzes the preprocessed data. The input is the separated video and audio data, and the output is the extracted cooking process information and transcribed instructions and comments. Computer vision technology is used to analyze the video data for ingredients, utensil usage, cooking process, etc. Specifically, each frame is analyzed using image recognition technology (e.g., YOLO, OpenCV) to identify ingredients and utensils.

[0579] Step 5: Analyze the audio data

[0580] The server analyzes the voice data and converts it into text. The input is voice data, and the output is text data such as instructions or comments. Voice recognition technology (e.g., Google Cloud Speech-to-Text API) is used to convert the voice data into text format.

[0581] Step 6: Training the AI ​​model

[0582] The server trains the learning model with the analysis results. The input is the extracted cooking process information and text instructions and comments, and the output is the trained model. Specifically, the analysis results are fed into the learning model using TensorFlow or PyTorch to improve the model's reproducibility.

[0583] Step 7: Accepting a reproduction request

[0584] When a user wants to recreate a specific dish or technique, they send a request to the server via their device. The input is the user's request to recreate it, and the output is the server's acceptance of the request. The user presses the "recreate request" button in the dedicated application to send a request to recreate "seared bonito."

[0585] Step 8: Generate cooking instructions and recipes

[0586] The server receives requests from users and generates optimal cooking instructions and the necessary recipes. The input is the reproduction request and the trained model, and the output is the specific cooking instructions and recipes. Specifically, the server uses the trained model to calculate the optimal cooking method for the requested dish and provides it in text format.

[0587] Step 9: Provide and Reproduce to Users

[0588] The server provides the generated cooking instructions and recipes to the device. The input is the cooking instructions and recipe, and the output is the cooking instructions and recipe displayed on the user's device. The user recreates the dish by checking the steps for each step in the application.

[0589] Step 10: Submit your feedback

[0590] The user provides feedback on the cooking results and their impressions. The input is the user's feedback information, and the output is the feedback data sent to the server. After the user finishes cooking, they enter their impressions and suggestions for improvement in the feedback form within the application and submit it.

[0591] Step 11: Improve the model

[0592] The server receives the feedback and analyzes the information to improve the learning model. The input is the feedback data, and the output is an improved learning model. Specifically, the server fine-tunes the model parameters based on the feedback and adds new data for learning, thereby improving the recall accuracy.

[0593] (Application example 1)

[0594] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0595] Traditional methods of learning cooking techniques and recipes require specialized knowledge and numerous manuals, which takes a great deal of time and effort. Furthermore, chefs and cooking staff who want to learn new techniques and recipes have few efficient ways to learn, limiting their ability to improve their skills even when practicing in a brick-and-mortar restaurant. To solve these problems, a system is needed that can easily record and analyze the cooking process, allowing for efficient learning of techniques and recipes.

[0596] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0597] In this invention, the server includes: a means for a user to record the cooking process; a means for uploading the recorded data of the cooking process to the server; a means for the server to convert the received data into an analyzable format; a means for the server to analyze the video data and extract each process; a means for the server to analyze the audio data and extract instructions and comments; a means for the server to train an AI model based on the analysis results; a means for a user to send a reproduction request; a means for the server to generate specific cooking steps and recipes and provide them to a terminal; a means for a user to provide feedback on the cooking results; a means for the server to analyze the feedback and improve the AI ​​model; and a means used by chefs and staff in the restaurant to learn new techniques and recipes. This makes it possible to efficiently and effectively learn cooking techniques and new recipes.

[0598] A "user" is a person or cook who uses the system to record and analyze the cooking process and learn new techniques and recipes.

[0599] "Means for recording" refers to devices or equipment for recording the cooking process on video.

[0600] "Means for uploading data to a server" refers to the function or process of transmitting recorded video data to a server via the Internet.

[0601] "Means of converting into an analyzable format" refers to the process by which the server converts the data it receives into a format that can be processed by AI models or other data analysis tools.

[0602] "Means for analyzing video data" refers to a method for analyzing recorded video data and extracting each cooking process and step.

[0603] The "means for analyzing audio data" is a method for analyzing recorded audio data and extracting verbal instructions and comments.

[0604] "Means for training an AI model" refers to a method of inputting analyzed data into an artificial intelligence model to improve and optimize cooking techniques and recipes.

[0605] The "means for sending a reproduction request" is a function or process by which a user sends a request to the server to reproduce a particular dish or technique.

[0606] "Means for generating specific cooking steps and recipes and providing them to the terminal" refers to a method in which the server generates specific cooking steps and recipes based on the analysis and learning results and provides them to the user's terminal.

[0607] The "means for feeding back cooking results" refers to a function or process that reports the results of cooking performed by the user to the server.

[0608] "Means for analyzing feedback and improving the AI ​​model" refers to the process of analyzing feedback data obtained from users and updating and improving the model.

[0609] "Methods used by in-store chefs and staff to learn new techniques and recipes" refers to methods and tools that allow chefs and culinary staff in physical stores to efficiently learn new techniques and recipes.

[0610] DETAILED DESCRIPTION OF THE INVENTION The following describes an embodiment of the present invention.

[0611] The system's program is designed to allow users to learn and reproduce new techniques and recipes by recording the cooking process and uploading the data to a server for analysis. Specific hardware and software configurations and processing are explained below.

[0612] Hardware and software used

[0613] 1. Hardware:

[0614] Smartphone or head-mounted display (HMD): Used to record and play back the cooking process.

[0615] Server: Stores data, analyzes it, performs AI learning, and processes feedback.

[0616] 2. Software:

[0617] Recording and uploading application: An application installed on a smartphone or HMD to upload recorded data to a server.

[0618] Data analysis software: Video and audio data analysis is performed on the server, using, for example, OpenCV (video processing) and SpeechRecognition (audio processing) libraries.

[0619] AI Model: A generative AI model built using the PyTorch framework.

[0620] User Interface (UI): An application that allows for easy cooking instructions requests and feedback.

[0621] Data processing and data calculation

[0622] The user records the cooking process using a smartphone or HMD. The recorded data is sent to the server via an upload application. The data received by the server is processed as follows:

[0623] 1. Data preprocessing:

[0624] The server splits the video data into frames and extracts detailed information such as the type of ingredients, the use of cooking utensils, the cooking process, etc. It analyzes the video using the OpenCV library.

[0625] At the same time, the voice data is analyzed and instructions and comments are converted into text using the SpeechRecognition library.

[0626] 2. AI Learning:

[0627] The analyzed data is fed into an AI model that learns cooking techniques and recipes. Specifically, PyTorch is used to train and update the model, which reflects subtleties such as cutting, heat control, and seasoning use.

[0628] 3. Providing cooking instructions and recipes:

[0629] When a user requests the reproduction of a specific dish, the server uses an AI model to generate optimal cooking instructions and recipes and provides them to the device.

[0630] 4. Feedback Processing:

[0631] Users provide feedback on their cooking results, and the server analyzes the data to further improve the AI ​​model.

[0632] Specific use cases

[0633] For example, if a user wants to perfectly recreate "seared bonito," they would follow these steps:

[0634] 1. Record the cooking process of seared bonito using a smartphone or HMD.

[0635] 2. Use an upload application to send the recorded data to the server.

[0636] 3. The server analyzes the video and audio data and trains the AI ​​model to learn the necessary cooking techniques and recipe details.

[0637] 4. When a user requests to recreate seared bonito, the server generates the optimal steps and recipe and provides it to the device.

[0638] 5. The user follows the instructions to cook the food and provides feedback on the results.

[0639] Prompt Sentence Examples

[0640] The user can request cooking instructions using prompts such as:

[0641] Instruct the AI ​​to reconstruct the cooking process for "Seared Bonito" and provide detailed step-by-step instructions, including how to cut the bonito, how to cook it, and how to press it into a firm consistency.

[0642] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0643] Step 1:

[0644] The user records the cooking process using a smartphone or head-mounted display (HMD). Video and audio are recorded during the recording. The input is real-time video and audio of the cooking process, and the output is the recorded video and audio files. Specifically, the entire cooking process is recorded by pressing the start recording button.

[0645] Step 2:

[0646] The device uploads the recorded data to the server. After the user finishes recording, they press the upload button and the video and audio files are sent to the server. The input is the recorded video and audio files, and the output is the files saved on the server. Specifically, the upload process runs in the background and the progress is displayed to the user.

[0647] Step 3:

[0648] The server converts the received data into an analyzable format. The server separates the video and audio data and temporarily stores them. The input is the original uploaded video and audio files, and the output is the separated video and audio data. Specifically, the video frames are extracted as continuous still images, and the audio data is stored as a separate audio file.

[0649] Step 4:

[0650] The server analyzes the video data and extracts each process. It uses OpenCV to analyze each frame and automatically extracts the type of ingredients, how to use cooking utensils, and steps in the cooking process. The input is the separated video data, and the output is the analyzed process information. Specifically, it performs object recognition on a frame-by-frame basis to classify ingredients and cooking utensils.

[0651] Step 5:

[0652] The server analyzes the audio data and extracts instructions and comments. It uses the SpeechRecognition library to convert the audio into text and extracts cooking instructions and explanations. The input is separated audio data, and the output is text data. Specifically, the audio file is read and converted to text through a speech recognition API.

[0653] Step 6:

[0654] The server trains the AI ​​model with the analysis results. The analyzed video and audio data is used to train the generative AI model on cooking techniques and recipes. The input is the analyzed process information and instruction text, and the output is an updated AI model. Specifically, the PyTorch framework is used to train the model based on the analyzed data.

[0655] Step 7:

[0656] The user submits a reproduction request. If the user wants to reproduce a specific dish or technique, the request is sent to the server via the device. The input is the reproduction request information, and the output is a notification that the request has been accepted. Specifically, the user enters the name of the dish or a specific technique in the app and submits it.

[0657] Step 8:

[0658] The server generates specific cooking steps and recipes and provides them to the device. Based on the learning results of the AI ​​model, the server generates optimal steps and provides them to the user. The input is the reproduction request and the trained AI model, and the output is specific cooking steps and recipe information. Specifically, a detailed recipe generated based on the prompt text is displayed on the user interface.

[0659] Step 9:

[0660] The user provides feedback on the cooking results. The results of the recreated dish are sent to the server, and the success rate of the cooking and areas for improvement are reported. The input is feedback information (photos, comments), and the output is feedback data stored on the server. Specifically, after cooking, the results are photographed and entered, and sent via the app.

[0661] Step 10:

[0662] The server analyzes the feedback and improves the AI ​​model. The feedback data is analyzed and reflected in the next learning. The input is feedback data from the user, and the output is a further improved AI model. Specifically, the feedback is aggregated and analyzed, and the model's hyperparameters are adjusted and additional learning is performed.

[0663] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0664] The system of the present invention allows users to record the cooking process, analyze the data, learn cooking techniques, and then reproduce the recipes. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system aims to increase user satisfaction. An embodiment of this system will be described in detail below.

[0665] System configuration

[0666] The system mainly consists of the following elements:

[0667] 1. User Recording Method

[0668] 2. Data upload method

[0669] 3. Server data analysis method

[0670] 4. AI learning methods

[0671] 5. How to Request Reproduction

[0672] 6. Feedback channels

[0673] 7. Emotion Engine

[0674] Program processing

[0675] The system program does the following:

[0676] Acquiring Recording Data

[0677] The system provides a means for users to record the cooking process. Users can use their smartphones or video cameras to record the cooking process of top chefs in detail. After the recording is complete, users can import the recorded data to their devices.

[0678] Uploading data

[0679] The device uploads the recorded data to the server. Data can be easily sent to the server using a dedicated application.

[0680] Data preprocessing and analysis

[0681] The server converts the received recorded data into an analyzable format. First, the video data and audio data are separated and processed separately. The video data is analyzed frame by frame to extract information such as the type of ingredients, how to use cooking utensils, and the cooking process. Meanwhile, the audio data is converted into text to extract the chef's instructions and comments.

[0682] AI Learning

[0683] The server analyzes the data and trains the AI ​​model, which allows it to accurately reproduce cooking techniques and flavors. Specifically, even minute details such as cutting methods, heat levels, and the use of seasonings are reflected in the model.

[0684] Reproduction requests and submissions

[0685] When a user wants to recreate a specific dish or technique, they send a request to the server through their device. The server generates cooking instructions and recipes based on the request and provides them to the device. The user can then follow these instructions to recreate the dish.

[0686] Feedback and model improvement

[0687] The user recreates the dish and sends feedback to the emotion engine via their device. The emotion engine analyzes the user's emotions (for example, satisfaction or difficulties) and sends this information to the server. The server receives the feedback and reflects it in the AI ​​model, further improving the accuracy of future reproductions.

[0688] Emotion Engine Operation

[0689] The emotion engine recognizes the emotions felt by the user regarding the cooking process and the results. For example, it uses facial recognition and voice analysis technology to detect the joy, difficulty, and satisfaction felt by the user in real time, and sends this data to the server. The server then analyzes this information and reflects it in the AI ​​model. This allows the system to provide optimal cooking procedures and recipes that take the user's emotions into account.

[0690] Specific examples

[0691] For example, if a user wants to make "seared bonito", the steps would be as follows:

[0692] 1. A user records a chef "searing bonito."

[0693] 2. The device uploads the recorded data to the server.

[0694] 3. The server analyzes information from the video data, such as the steps for cutting the bonito, how well it is cooked, and how to pound it, and extracts supplementary explanations from the audio data.

[0695] 4. The server trains the AI ​​model on the analysis results.

[0696] 5. The user sends a request to recreate "Seared Bonito."

[0697] 6. The server generates the optimal cooking procedure and necessary recipes and provides them to the terminal.

[0698] 7. The user cooks according to the provided instructions, and the emotion engine analyzes the user's emotions and sends them to the server.

[0699] 8. The server analyzes the emotional feedback and improves the AI ​​model.

[0700] This allows users to efficiently learn first-class cooking techniques and reproduce them in an optimal way, taking into account their emotions during the process. This system will help pass on traditional cooking techniques and improve user satisfaction.

[0701] The processing flow will be explained below.

[0702] Step 1:

[0703] The user records the cooking process. The user uses a smartphone or video camera to record the cooking process of a top chef in detail. The recording is done so that every process is clearly visible.

[0704] Step 2:

[0705] The user imports the recorded data to the device. Once recording is complete, the user transfers the video file to a device such as a smartphone or PC and opens the dedicated application.

[0706] Step 3:

[0707] The device uploads the recorded data to the server. A dedicated application is used to select the recorded data and issue an upload command to the server. The data is then securely transferred to the server.

[0708] Step 4:

[0709] The server receives the recorded data and prepares it for conversion into an analyzable format.

[0710] Step 5:

[0711] The server converts the recorded data into an analyzable format, splits the video file into frames, and separates the video data from the audio data.

[0712] Step 6:

[0713] The server analyzes the video data, analyzing each individual frame to extract each step of the cooking process (e.g., type of ingredient, cutting method, heat level), and even to identify the smallest actions and steps visible in the video.

[0714] Step 7:

[0715] The server analyzes the audio data, converts it into text, and extracts the chef's instructions and comments, providing a detailed textual description of each process.

[0716] Step 8:

[0717] The server trains the AI ​​model with the analysis results, and the extracted text from the video and audio data is input into the AI ​​model, which then learns how to reproduce the cooking techniques and flavors.

[0718] Step 9:

[0719] The user sends a reproduction request to the server. The user sends a request to reproduce a specific dish (e.g., "seared bonito") to the server via the terminal.

[0720] Step 10:

[0721] The server generates cooking instructions and recipes based on the request, calculates the optimal steps based on an AI model, and generates detailed recipes and cooking instructions.

[0722] Step 11:

[0723] The terminal provides the cooking instructions and recipes to the user, who can then view the cooking instructions and recipes obtained from the server through the terminal.

[0724] Step 12:

[0725] The user follows the instructions to cook the food, and then recreates the dish based on the instructions displayed on the device.

[0726] Step 13:

[0727] The emotion engine analyzes the user's emotions during the cooking process. The emotion engine recognizes emotions from the user's facial expressions and voice, and analyzes the satisfaction and difficulties during the cooking process in real time.

[0728] Step 14:

[0729] The emotion engine sends the analysis results to the server, which then transmits the emotion data to the server, which then reflects it in the cooking process.

[0730] Step 15:

[0731] The user provides feedback on the cooking results, sending their evaluation of the cooking results and taste to the server via their device.

[0732] Step 16:

[0733] The server receives and analyzes the feedback sent by users and uses it to improve the AI ​​model.

[0734] Step 17:

[0735] The server improves the AI ​​model, adjusting it based on feedback and emotion data to improve the accuracy of the next replay.

[0736] This process allows users to accurately learn and recreate traditional cooking techniques, and also provides an optimal cooking environment for users by taking into account their emotions during the cooking process.

[0737] Example 2

[0738] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0739] Conventional cooking skill learning systems have difficulty effectively replicating professional cooking techniques, and have been unable to properly utilize feedback to ensure users get emotionally satisfying results. Furthermore, there has been no means to analyze users' emotions in real time and reflect them in the cooking process.

[0740] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for a user to record the cooking process, a means for uploading the recorded cooking process data to the server, a means for the server to convert the received data into an analyzable format, a means for the server to analyze the video data and extract each process, a means for the server to analyze the audio data and extract instructions and comments, a means for the server to train an AI model based on the analysis results, a means for a user to send a reproduction request, a means for the server to generate specific cooking steps and recipes and provide them to the terminal, a means for the user to provide feedback on the cooking results, a means for the server to analyze the feedback and improve the AI ​​model, and an emotion analysis means, which recognizes the user's emotions in real time and transmits the analysis results to the server. This makes it possible to provide optimal cooking instructions that take the user's emotions into consideration, solving problems associated with conventional systems and enabling users to learn and reproduce cooking techniques with greater accuracy.

[0741] A "user" is an individual or entity who uses the system to learn and replicate cooking techniques.

[0742] "Recorded data" refers to video and audio data used by a user to record the cooking process.

[0743] The "server" is a central computer system that receives the recorded data, analyzes it, trains the AI ​​model, and provides the results to the user.

[0744] "Data analysis" refers to the process in which the server identifies the video and audio data it receives and extracts the cooking process, instructions, and comments.

[0745] An "AI model" is an artificial intelligence algorithm or system that learns based on analysis results and provides users with optimal cooking procedures and recipes.

[0746] A "reproduction request" is an act in which a user sends a request to the server that they would like to learn or reproduce a particular dish or technique.

[0747] "Emotion analysis" refers to the emotion analysis means recognizing the user's emotions in real time and sending the analysis results to the server.

[0748] "Feedback" includes the user reporting the results of their cooking reproduction to the system and the results of subsequent sentiment analysis.

[0749] "Cooking instructions" refers to the specific cooking methods and steps generated by the server.

[0750] A "recipe" is a set of detailed instructions detailing the ingredients and quantities needed to prepare a particular dish, as well as the cooking method involved.

[0751] "Emotion analysis means" refers to technology or devices for detecting and analyzing a user's emotions in real time, and uses facial recognition technology and voice analysis technology.

[0752] MODE FOR CARRYING OUT THE INVENTION

[0753] The system of the present invention allows users to record the cooking process, analyze the data, learn cooking techniques, and then reproduce the recipes. Furthermore, by combining it with emotion analysis means that recognizes the user's emotions, the system aims to increase user satisfaction.

[0754] This system consists of a user recording means, a data uploading means, a server data analysis means, an AI learning means, a reproduction request means, a feedback means, and an emotion analysis means.

[0755] Acquiring Recording Data

[0756] Users record the cooking process using a recording device such as a smartphone or video camera. This process includes everything from preparation to completion, and detailed footage is required. For example, when making "seared bonito," users will record in detail how they cut the bonito, how well it is cooked, and how they sear it.

[0757] Uploading data

[0758] Once the recording is complete, the user imports the recording data to their device and then uploads it to the server using a dedicated application. This process can be completed with just a few clicks.

[0759] Data preprocessing and analysis

[0760] The server converts the received video data into an analyzable format. During this process, the video data and audio data are separated and processed separately. The video data is analyzed frame by frame to extract information such as the type of ingredients, the use of cooking utensils, and the cooking process. For example, the way ingredients are cut and cooked can be identified. Meanwhile, the audio data is converted to text to extract the chef's instructions and comments.

[0761] AI Learning

[0762] The server then trains the AI ​​model with the analysis results. This training process includes detailed cooking techniques, including how to cut food, how much heat to use, and how to use seasonings, so that the AI ​​model can accurately reproduce detailed cooking techniques and flavors. This allows the AI ​​model to provide users with optimal cooking procedures and recipes.

[0763] Reproduction requests and submissions

[0764] When a user wants to recreate a specific dish or technique, they send a request to the server via their device. Based on the request, the server generates the optimal cooking steps and necessary recipes and provides them to the device. The user can then recreate the dish based on this information. An example of a specific prompt sentence is, "Please tell me the optimal cooking steps and recipe for making bonito tataki. Please also provide voice instructions."

[0765] Feedback and model improvement

[0766] After the user has finished recreating the dish, they provide feedback to the emotion analysis means via their device. The emotion analysis means analyzes the user's emotions and sends the results to the server. The server receives the feedback and uses it to improve the AI ​​model. For example, the emotion analysis means can analyze the user's satisfaction and difficulties and reflect that information in the AI ​​model.

[0767] Operation of emotion analysis means

[0768] The emotion analysis means uses facial recognition and voice analysis technologies to detect the user's emotions in real time and transmits the data to the server, which enables the system to provide optimal cooking instructions and recipes that take the user's emotions into consideration.

[0769] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0770] Step 1: Obtaining the recording data

[0771] Users can use their smartphones or video cameras to record the cooking process in detail, for example, the entire cooking process of "seared bonito."

[0772] Input: Video and audio data stored on the recording device.

[0773] How it works: The user starts recording and then imports the recording to their device after shooting.

[0774] Output: Recording data file saved on the device.

[0775] Step 2: Upload your data

[0776] The device uploads the recorded data to the server using a dedicated application.

[0777] Input: Recording data files saved on the device.

[0778] How it works: The user launches the dedicated app and clicks the "upload" button to send data to the server.

[0779] Output: Recording data file saved on the server.

[0780] Step 3: Data preprocessing and analysis

[0781] The server converts the recorded data into an analyzable format, separating the video and audio data and processing each separately.

[0782] Input: Recording data files stored on the server.

[0783] How it works: The server separates the video data from the audio data and analyzes the video data frame by frame. Specifically, it extracts information such as the type of ingredients, how to use cooking utensils, and the cooking process. Meanwhile, the audio data is converted into text and the chef's instructions and comments are extracted.

[0784] Output: Analyzable video data and audio data in text format.

[0785] Step 4: AI training

[0786] The data analyzed by the server is trained into an AI model.

[0787] Input: Analyzable video data and audio data in text format.

[0788] How it works: The server inputs the data into the AI ​​model and begins the learning process, incorporating even the smallest details like cutting techniques, heat levels, and seasonings.

[0789] Output: The trained AI model.

[0790] Step 5: Request and provide a reproduction

[0791] When a user wants to recreate a particular dish or technique, they send a request to the server through their device.

[0792] Input: The reproduction request submitted by the user.

[0793] How it works: The server generates the optimal cooking instructions and recipe based on the request and sends it to the device. For example, if the request is "I want to recreate seared bonito," the server will provide the instructions and recipe for that.

[0794] Output: Specific cooking instructions and recipe displayed on the user's device.

[0795] Step 6: Feedback and model improvement

[0796] After the user has completed recreating the dish, they provide feedback to the sentiment analysis means.

[0797] Input: User-provided feedback data.

[0798] Operation: The emotion analysis means analyzes the feedback data and sends it to the server. For example, the satisfaction or difficulties felt during the cooking process are specifically input.

[0799] Output: Feedback analysis data sent to the server.

[0800] Step 7: Sentiment analysis method in action

[0801] The emotion analysis means recognizes the user's emotions in real time and transmits the analysis results to the server.

[0802] Input: Real-time video and audio data of the user cooking.

[0803] How it works: The emotion analysis unit uses face recognition and voice analysis technology to detect the user's emotions and sends the data to the server. For example, it analyzes emotions such as joy or difficulty.

[0804] Output: Real-time emotion data sent to the server.

[0805] (Application example 2)

[0806] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0807] Conventional food delivery systems lacked the means to effectively manage and improve food quality and user satisfaction. In particular, there were many challenges in the reproducibility of cooking processes, the skill acquisition of delivery staff, and the management of customer feedback. The purpose of this invention is to solve these problems, improve the quality of delivery services, and increase the satisfaction of customers and delivery staff.

[0808] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to record the cooking process and upload the data, means for analyzing the uploaded recorded data and extracting each process from the video data, means for extracting instructions and comments from the audio data, means for training the generative AI model based on the analysis results, means for a user to send a reproduction request, generate specific cooking steps and recipes, and provide them to the terminal, means for a user to provide feedback on the cooking results and for the server to analyze the feedback and improve the generative AI model, means for a delivery staff member to record the cooking process, means for uploading the recorded data to a cloud server, means for the cloud server to analyze the uploaded recorded data and extract cooking skills, and means for analyzing the emotions of the delivery staff member and sending feedback to the server. This improves the reproducibility of the cooking process, enabling delivery staff to acquire skills and customer feedback to be managed more effectively.

[0809] "Means for users to record the cooking process" refers to devices or methods that allow users to record the cooking process as video.

[0810] The "means for uploading recorded data of the cooking process to the server" is a means for transmitting video data of the cooking process recorded by the user to the server via the Internet.

[0811] "Means for converting data received by the server into an analyzable format" refers to a method or algorithm for converting data uploaded to the server into a format that can be easily analyzed.

[0812] "Means for the server to analyze video data and extract each process" refers to a method or system for analyzing the content of video data and identifying and extracting each step of food preparation and cooking.

[0813] "Means for the server to analyze the audio data and extract instructions and comments" refers to methods and technologies for identifying cooking instructions and comments from the recorded audio data and extracting them as text.

[0814] "Means for the server to train the generated AI model on the analysis results" refers to a method for training the AI ​​model generated based on the analysis results extracted from video data and audio data on new data.

[0815] The "means by which a user sends a reproduction request" refers to an interface or method by which a user can send a request to the server when they want to reproduce a particular dish or technique.

[0816] "Means for the server to generate specific cooking procedures and recipes and provide them to the terminal" refers to a system that generates optimal cooking procedures and recipes based on the user's reproduction request and provides them to the user's terminal.

[0817] "Means for users to provide feedback on cooking results" refers to methods or tools that allow users to return the results of their recreated dishes to the server as their impressions or evaluations.

[0818] "Means for the server to analyze feedback and improve the generative AI model" refers to a system or method for analyzing feedback from users and improving the generative AI model based on the results.

[0819] "Means for delivery staff to record the cooking process" refers to devices or methods that delivery staff use to record the cooking process of food on video.

[0820] "Means for uploading recorded data to a cloud server" refers to means for saving recorded data to a cloud server via the Internet.

[0821] "Means for the cloud server to analyze the uploaded video data and extract cooking techniques" refers to a method or system for the cloud server to analyze the video data received and identify and extract cooking techniques and processes.

[0822] "Means for analyzing the emotions of delivery staff and sending feedback to a server" refers to a method or technology for analyzing the emotions of delivery staff from video and audio data and sending the results as feedback to a server.

[0823] This invention aims to develop a system that aims to improve customer satisfaction and control the quality of the cooking process in food delivery services. The system consists of the following components:

[0824] 1. User recording method: The user or delivery staff uses a recording device such as a smartphone to record the cooking process. The video data contains detailed information such as the cooking steps, ingredients used, and necessary cooking utensils.

[0825] 2. Data uploading method: Users use a dedicated application to upload recorded data to the cloud server, which allows data to be easily sent to the server via the Internet.

[0826] 3. Server data analysis method: The server converts the received video data into an analyzable format. Specifically, the video data is divided into frames and image recognition technology (e.g., OpenCV) is used to analyze the type of ingredients and cooking process. Additionally, audio data is converted into text using speech recognition technology (e.g., Google Speech-to-Text) to extract cooking comments and instructions.

[0827] 4. AI learning method: The generative AI model learns cooking techniques based on the analyzed data. This model is built using a machine learning library (e.g., TensorFlow). The model reflects detailed cooking techniques, such as how to cut ingredients, how to control heat, and how to use seasonings.

[0828] 5. Reproduction request method: When a user wants to reproduce a specific dish, they send a request to the server through a dedicated application. Based on this request, the server generates the optimal cooking procedure and recipe and provides it to the user's device.

[0829] 6. Feedback method: The user provides feedback on the cooking results to the server through the application. The feedback is analyzed using facial recognition technology (e.g., DeepFace) to quantify the user's satisfaction.

[0830] 7. Emotion Engine: The emotion engine recognizes the user's emotions in real time and transmits the user's joy, difficulty, and satisfaction to the server, which analyzes this data and improves the generative AI model.

[0831] The above system will improve the reproducibility of the cooking process, enable delivery staff to acquire skills, and enable effective management of customer feedback. It will also optimize the recipes and cooking procedures provided to users, which is expected to improve customer satisfaction.

[0832] Examples:

[0833] For example, a delivery staff member serves a highly rated dish to a customer and records the cooking process. The video data is uploaded to a cloud server, which analyzes the data and learns the cooking process and techniques. If another user requests to make the same dish, the server will provide the optimal cooking procedure and recipe, and the user will provide feedback on the results, and the emotion engine will analyze the customer's satisfaction level.

[0834] Example of an input prompt:

[0835] "I want to analyze videos of the cooking process taken by delivery staff and collect emotional feedback. Please generate a report that includes the analysis results and emotional feedback."

[0836] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0837] Step 1:

[0838] The user records the cooking process using a means to record the cooking process. Specifically, the user or delivery staff uses a smartphone or video camera to record the cooking process from start to finish. The input is video data of the cooking process, and the output is the recorded video file.

[0839] Step 2:

[0840] The recorded cooking process data is uploaded to the server. The user uses a dedicated upload application to send the recorded data to the cloud server. The input is a video file, and the output is video data on the cloud server.

[0841] Step 3:

[0842] The server converts the received data into an analyzable format. The cloud server divides the received video file into frames and separates it into image data and audio data. It also converts the audio data into text. The input is video data on the cloud server, and the output is image data and text data.

[0843] Step 4:

[0844] The server analyzes the video data and extracts each process. Using machine learning and image recognition technology (e.g., OpenCV), the server identifies the type of ingredients, cooking utensils, and cooking process from the image data for each frame. The input is the image data for each frame, and the output is the extracted cooking process information.

[0845] Step 5:

[0846] The server analyzes the voice data and extracts instructions and comments. The server uses voice recognition technology (e.g., Google Speech-to-Text) to convert the voice data into text and identify cooking instructions and comments. The input is the voice data, and the output is the extracted text data.

[0847] Step 6:

[0848] The server trains the generative AI model with the analysis results. The server inputs the analyzed cooking process information and text data into the generative AI model (e.g., TensorFlow) and trains the model to learn new cooking techniques. The input is the cooking process information and text data, and the output is an updated generative AI model.

[0849] Step 7:

[0850] The system uses a means for users to send a reproduction request. A user sends a request to reproduce a specific dish or cooking technique through a dedicated application. The input is the reproduction request, and the output is the request data sent to the server.

[0851] Step 8:

[0852] The server generates specific cooking steps and recipes and provides them to the device. The server generates optimal cooking steps and recipes based on the generative AI model and provides this information to the user's device. The input is the generative AI model and the user's reproduction request, and the output is the cooking steps and recipe provided to the user's device.

[0853] Step 9:

[0854] The system uses a means for users to provide feedback on their cooking results. Users cook and provide feedback on the results through a dedicated application. The input is the user's cooking results, and the output is feedback data to the server.

[0855] Step 10:

[0856] The server analyzes the feedback and improves the generative AI model. The server analyzes the user feedback data, evaluates customer satisfaction using an emotion engine, and reflects the results in the generative AI model. The input is the feedback data, and the output is an improved generative AI model.

[0857] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0858] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0859] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0860] [Third embodiment]

[0861] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0862] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0863] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0864] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0865] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0866] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0867] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0868] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0869] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0870] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0871] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0872] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0873] The system of the present invention allows users to easily record and analyze the cooking process, and based on that, learn and reproduce cooking techniques. An embodiment of the system will be described in detail below.

[0874] System configuration

[0875] The system mainly consists of the following elements:

[0876] 1. User Recording Method

[0877] 2. Data upload method

[0878] 3. Server data analysis method

[0879] 4. AI learning methods

[0880] 5. How to Request Reproduction

[0881] 6. Feedback channels

[0882] Program processing

[0883] The system program does the following:

[0884] Acquiring Recording Data

[0885] The user uses a means to record the cooking process. For example, the user uses a smartphone or video camera to record the cooking process. After the recording is finished, the user imports the recorded data into the device.

[0886] Uploading data

[0887] The device uploads the recorded data to the server. Data can be easily sent to the server using a dedicated application.

[0888] Data preprocessing and analysis

[0889] The server converts the received recorded data into an analyzable format. First, the video data and audio data are separated and processed separately. The video data is analyzed frame by frame to extract information such as the type of ingredients, how to use cooking utensils, and the cooking process. Meanwhile, the audio data is converted into text to extract the chef's instructions and comments.

[0890] AI Learning

[0891] The server analyzes the data and trains the AI ​​model, which allows it to accurately reproduce cooking techniques and flavors. Specifically, even minute details such as cutting methods, heat levels, and the use of seasonings are reflected in the model.

[0892] Reproduction requests and submissions

[0893] When a user wants to recreate a specific dish or technique, they send a request to the server through their device. The server generates cooking instructions and recipes based on the request and provides them to the device. The user can then follow these instructions to recreate the dish.

[0894] Feedback and model improvement

[0895] The user recreates the dish and provides feedback on the results. The device then sends the feedback to the server, which analyzes the information and improves the AI ​​model, further improving the accuracy of future attempts.

[0896] Specific examples

[0897] For example, if a user wants to make "seared bonito", the steps would be as follows:

[0898] 1. A user records a chef "searing bonito."

[0899] 2. The device uploads the recorded data to the server.

[0900] 3. The server analyzes information from the video data, such as the steps for cutting the bonito, how well it is cooked, and how to pound it, and extracts supplementary explanations from the audio data.

[0901] 4. The server trains the AI ​​model on the analysis results.

[0902] 5. The user sends a request to recreate "Seared Bonito."

[0903] 6. The server generates the optimal cooking procedure and necessary recipes and provides them to the terminal.

[0904] 7. The user follows the provided instructions to cook the food and provides feedback on the results.

[0905] 8. The server analyzes the feedback and improves the AI ​​model.

[0906] This allows users to efficiently learn and reproduce first-class cooking techniques. This system also enables the efficient and accurate inheritance of traditional cooking techniques.

[0907] The processing flow will be explained below.

[0908] Step 1:

[0909] The user records the cooking process. The user uses a smartphone or video camera to record the cooking process of a top chef in detail. The recording is done so that every process is clearly visible.

[0910] Step 2:

[0911] The user imports the recorded data to the device. Once recording is complete, the user transfers the video file to a device such as a smartphone or PC and opens the dedicated application.

[0912] Step 3:

[0913] The device uploads the recorded data to the server. A dedicated application is used to select the recorded data and issue an upload command to the server. The data is then securely transferred to the server.

[0914] Step 4:

[0915] The server receives the recorded data and prepares it for conversion into an analyzable format.

[0916] Step 5:

[0917] The server converts the recorded data into an analyzable format, splits the video file into frames, and separates the video data from the audio data.

[0918] Step 6:

[0919] The server analyzes the video data, analyzing each individual frame to extract each step of the cooking process (e.g., type of ingredient, cutting method, heat level), and even to identify the smallest actions and steps visible in the video.

[0920] Step 7:

[0921] The server analyzes the audio data, converts it into text, and extracts the chef's instructions and comments, providing a detailed textual description of each process.

[0922] Step 8:

[0923] The server trains the AI ​​model with the analysis results, and the extracted text from the video and audio data is input into the AI ​​model, which then learns how to reproduce the cooking techniques and flavors.

[0924] Step 9:

[0925] The user sends a reproduction request to the server. The user sends a request to reproduce a specific dish (e.g., "seared bonito") to the server via the terminal.

[0926] Step 10:

[0927] The server generates cooking instructions and recipes based on the request, calculates the optimal steps based on an AI model, and generates detailed recipes and cooking instructions.

[0928] Step 11:

[0929] The terminal provides the cooking instructions and recipes to the user, who can then view the cooking instructions and recipes obtained from the server through the terminal.

[0930] Step 12:

[0931] The user follows the instructions to cook the food, and then recreates the dish based on the instructions displayed on the device.

[0932] Step 13:

[0933] The user provides feedback on the cooking results, sending their evaluation of the cooking results and taste to the server via their device.

[0934] Step 14:

[0935] The server receives and analyzes the feedback sent by users and uses it to improve the AI ​​model.

[0936] Step 15:

[0937] The server improves the AI ​​model, adjusting it based on the feedback to improve the accuracy of the next replay.

[0938] This process allows users to accurately learn and recreate traditional cooking techniques.

[0939] Example 1

[0940] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0941] Traditional cooking skill learning methods often rely on watching videos or reading books, making it difficult to fully understand the details of the actual cooking process. It is particularly difficult to accurately reproduce the techniques of professional chefs, making it difficult for learners to improve their skills. Furthermore, there are limited ways to improve one's own cooking skills through feedback, making it difficult to efficiently acquire new cooking skills.

[0942] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0943] In this invention, the server includes means for a user to record the cooking process, means for uploading data of the recorded cooking process to an information processing device, means for the information processing device to convert the received data into an analyzable format, means for the information processing device to analyze the video data and extract each process, means for the information processing device to analyze the audio data and extract instructions and comments, means for the information processing device to train a learning model based on the analysis results, means for a user to send a reproduction request, means for the information processing device to generate specific cooking steps and recipes and provide them to the terminal, means for a user to provide feedback on the cooking results, and means for the information processing device to analyze the feedback and improve the learning model. This enables users to accurately learn and reproduce professional cooking techniques and improve their own skills.

[0944] "User" refers to any individual or legal entity that records the cooking process and uploads the data to the system.

[0945] The "cooking process" refers to the series of steps involved in preparing ingredients, cooking, and finishing.

[0946] "Recording means" refers to a device for recording video and audio, such as a smartphone or video camera.

[0947] "Data" refers to video data and audio data acquired by a recording means.

[0948] "Information processing device" refers to a device that analyzes, processes, and stores data, such as a server or cloud service.

[0949] "Uploading means" refers to the means by which a user sends recorded data to an information processing device, including dedicated applications and web forms.

[0950] The term "analyzable format" refers to a format required for an information processing device to process data appropriately.

[0951] "Video data" refers to video frame information captured by a recording means.

[0952] "Audio data" refers to audio information captured by a recording means.

[0953] The "means for extracting each process" refers to a means for analyzing and identifying each stage of cooking from video data and audio data.

[0954] "Instructions and comments" refers to statements and explanations made by the chef or user regarding the cooking process.

[0955] "Learning model" refers to analytical algorithms and data models that use artificial intelligence to learn cooking techniques and processes.

[0956] A "reproduction request" refers to a user requesting the system to reproduce a specific dish or technique.

[0957] "Cooking instructions and recipes" refers to the specific steps and ingredients needed to recreate a dish.

[0958] "Terminal" refers to a device used by a user, such as a smartphone, tablet, or PC.

[0959] "Feedback" refers to information such as cooking results and impressions that users provide to the system.

[0960] "Analysis means" refers to the algorithms and techniques used by the information processing device to process and analyze data.

[0961] The system of the present invention allows users to easily record and analyze the cooking process, and then learn and reproduce cooking techniques based on the recorded footage. To implement this system, the following hardware and software are required: a user's smartphone, a video camera, a personal computer, a dedicated application, an information processing device (server), and a learning model.

[0962] Hardware and software used

[0963] 1. Smartphone / Video Camera: Used as a recording medium.

[0964] 2. Computer / dedicated application: Used to import and upload data.

[0965] 3. Information processing device (server): Used to analyze data, manage learning models, generate cooking instructions and recipes, and analyze feedback.

[0966] 4. Learning model: A model that uses AI technology and frameworks such as TensorFlow or PyTorch.

[0967] Specific actions and processes

[0968] The user uses a smartphone or video camera to record the cooking process. For example, the process of making "seared bonito" is recorded. The recorded data is then imported into the device and uploaded to an information processing device using a dedicated application. When uploading, the user sends the data by pressing the "upload" button in the application.

[0969] After receiving the data, the server separates the video data from the audio data. The video data is analyzed frame by frame to extract information such as the type of ingredients, how to use cooking utensils, and the cooking process. The audio data is converted to text and instructions and comments are extracted using natural language processing (NLP), for example, using the Google Cloud Speech-to-Text API.

[0970] The server then trains the learning model based on the analysis results. The learning model uses AI technology, such as TensorFlow or PyTorch, to accurately reproduce cooking techniques and flavors.

[0971] When a user wants to recreate a specific dish or technique, they send a request to the server via their device. A request is made by pressing the "reproduce request" button within the application. At this time, by specifying the specific dish name and technique, for example, "I want to recreate seared bonito," the request is sent to the server.

[0972] The server receives requests from users and generates optimal cooking instructions and recipes. The generated information is provided to the device, and the user recreates the dish according to the instructions. For example, the next cooking steps are displayed within the application.

[0973] After the user has finished cooking, they can provide feedback on their results and impressions. By filling out and submitting the feedback form within the dedicated application, the feedback is sent to the server, which then analyzes and improves the model. This improves the accuracy of the reproducibility from the next time onwards.

[0974] Examples and prompts

[0975] For example, a user who wants to make "seared bonito" would follow these steps:

[0976] 1. The user records the cooking process on their smartphone and uploads the recording data to the server using the application.

[0977] 2. The server analyzes the data and processes the video and audio data separately.

[0978] 3. The server trains the AI ​​model on the analysis results.

[0979] 4. The user sends a request to recreate "Seared Bonito" within the app.

[0980] 5. The server generates optimal cooking instructions and recipes and provides them to the device.

[0981] 6. The user follows the provided steps to recreate the dish and receives feedback on the results.

[0982] Prompt Sentence Examples

[0983] "I want to upload video data of the cooking process and have the AI ​​model learn from it. Please tell me the specific steps and tools I need to use."

[0984] "Analyze the cooking steps for seared bonito and generate the optimal recipe."

[0985] "I want to provide feedback to improve the AI ​​model. How do I do that?"

[0986] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0987] Step 1: Obtaining Recording Data

[0988] The user uses a smartphone or video camera to record the cooking process. For example, the process of making "seared bonito" is used as the recording process. The input is the recorded data that includes the entire cooking process. Once the recording is finished, the recorded data is imported to a device such as a smartphone or PC. The output is the recorded data saved on the device.

[0989] Step 2: Upload your data

[0990] The device uses a dedicated application to upload recorded data to the server. The input is the recorded data stored on the device, and the output is the recorded data stored on the server. When the dedicated application is launched on the device and the "upload button" is pressed, the data is sent to the server.

[0991] Step 3: Preprocessing the data

[0992] The recorded data received by the server is converted into an analyzable format. First, the video data and audio data are separated. The input is the recorded data stored on the server, and the output is the separated video and audio data. Specifically, the video data is divided into 30 frames per second, and the audio data is saved as an audio file.

[0993] Step 4: Analyze the data

[0994] The server analyzes the preprocessed data. The input is the separated video and audio data, and the output is the extracted cooking process information and transcribed instructions and comments. Computer vision technology is used to analyze the video data for ingredients, utensil usage, cooking process, etc. Specifically, each frame is analyzed using image recognition technology (e.g., YOLO, OpenCV) to identify ingredients and utensils.

[0995] Step 5: Analyze the audio data

[0996] The server analyzes the voice data and converts it into text. The input is voice data, and the output is text data such as instructions or comments. Voice recognition technology (e.g., Google Cloud Speech-to-Text API) is used to convert the voice data into text format.

[0997] Step 6: Training the AI ​​model

[0998] The server trains the learning model with the analysis results. The input is the extracted cooking process information and text instructions and comments, and the output is the trained model. Specifically, the analysis results are fed into the learning model using TensorFlow or PyTorch to improve the model's reproducibility.

[0999] Step 7: Accepting a reproduction request

[1000] When a user wants to recreate a specific dish or technique, they send a request to the server via their device. The input is the user's request to recreate it, and the output is the server's acceptance of the request. The user presses the "recreate request" button in the dedicated application to send a request to recreate "seared bonito."

[1001] Step 8: Generate cooking instructions and recipes

[1002] The server receives requests from users and generates optimal cooking instructions and the necessary recipes. The input is the reproduction request and the trained model, and the output is the specific cooking instructions and recipes. Specifically, the server uses the trained model to calculate the optimal cooking method for the requested dish and provides it in text format.

[1003] Step 9: Provide and Reproduce to Users

[1004] The server provides the generated cooking instructions and recipes to the device. The input is the cooking instructions and recipe, and the output is the cooking instructions and recipe displayed on the user's device. The user recreates the dish by checking the steps for each step in the application.

[1005] Step 10: Submit your feedback

[1006] The user provides feedback on the cooking results and their impressions. The input is the user's feedback information, and the output is the feedback data sent to the server. After the user finishes cooking, they enter their impressions and suggestions for improvement in the feedback form within the application and submit it.

[1007] Step 11: Improve the model

[1008] The server receives the feedback and analyzes the information to improve the learning model. The input is the feedback data, and the output is an improved learning model. Specifically, the server fine-tunes the model parameters based on the feedback and adds new data for learning, thereby improving the recall accuracy.

[1009] (Application example 1)

[1010] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1011] Traditional methods of learning cooking techniques and recipes require specialized knowledge and numerous manuals, which takes a great deal of time and effort. Furthermore, chefs and cooking staff who want to learn new techniques and recipes have few efficient ways to learn, limiting their ability to improve their skills even when practicing in a brick-and-mortar restaurant. To solve these problems, a system is needed that can easily record and analyze the cooking process, allowing for efficient learning of techniques and recipes.

[1012] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1013] In this invention, the server includes: a means for a user to record the cooking process; a means for uploading the recorded data of the cooking process to the server; a means for the server to convert the received data into an analyzable format; a means for the server to analyze the video data and extract each process; a means for the server to analyze the audio data and extract instructions and comments; a means for the server to train an AI model based on the analysis results; a means for a user to send a reproduction request; a means for the server to generate specific cooking steps and recipes and provide them to a terminal; a means for a user to provide feedback on the cooking results; a means for the server to analyze the feedback and improve the AI ​​model; and a means used by chefs and staff in the restaurant to learn new techniques and recipes. This makes it possible to efficiently and effectively learn cooking techniques and new recipes.

[1014] A "user" is a person or cook who uses the system to record and analyze the cooking process and learn new techniques and recipes.

[1015] "Means for recording" refers to devices or equipment for recording the cooking process on video.

[1016] "Means for uploading data to a server" refers to the function or process of transmitting recorded video data to a server via the Internet.

[1017] "Means of converting into an analyzable format" refers to the process by which the server converts the data it receives into a format that can be processed by AI models or other data analysis tools.

[1018] "Means for analyzing video data" refers to a method for analyzing recorded video data and extracting each cooking process and step.

[1019] The "means for analyzing audio data" is a method for analyzing recorded audio data and extracting verbal instructions and comments.

[1020] "Means for training an AI model" refers to a method of inputting analyzed data into an artificial intelligence model to improve and optimize cooking techniques and recipes.

[1021] The "means for sending a reproduction request" is a function or process by which a user sends a request to the server to reproduce a particular dish or technique.

[1022] "Means for generating specific cooking steps and recipes and providing them to the terminal" refers to a method in which the server generates specific cooking steps and recipes based on the analysis and learning results and provides them to the user's terminal.

[1023] The "means for feeding back cooking results" refers to a function or process that reports the results of cooking performed by the user to the server.

[1024] "Means for analyzing feedback and improving the AI ​​model" refers to the process of analyzing feedback data obtained from users and updating and improving the model.

[1025] "Methods used by in-store chefs and staff to learn new techniques and recipes" refers to methods and tools that allow chefs and culinary staff in physical stores to efficiently learn new techniques and recipes.

[1026] DETAILED DESCRIPTION OF THE INVENTION The following describes an embodiment of the present invention.

[1027] The system's program is designed to allow users to learn and reproduce new techniques and recipes by recording the cooking process and uploading the data to a server for analysis. Specific hardware and software configurations and processing are explained below.

[1028] Hardware and software used

[1029] 1. Hardware:

[1030] Smartphone or head-mounted display (HMD): Used to record and play back the cooking process.

[1031] Server: Stores data, analyzes it, performs AI learning, and processes feedback.

[1032] 2. Software:

[1033] Recording and uploading application: An application installed on a smartphone or HMD to upload recorded data to a server.

[1034] Data analysis software: Video and audio data analysis is performed on the server, using, for example, OpenCV (video processing) and SpeechRecognition (audio processing) libraries.

[1035] AI Model: A generative AI model built using the PyTorch framework.

[1036] User Interface (UI): An application that allows for easy cooking instructions requests and feedback.

[1037] Data processing and data calculation

[1038] The user records the cooking process using a smartphone or HMD. The recorded data is sent to the server via an upload application. The data received by the server is processed as follows:

[1039] 1. Data preprocessing:

[1040] The server splits the video data into frames and extracts detailed information such as the type of ingredients, the use of cooking utensils, the cooking process, etc. It analyzes the video using the OpenCV library.

[1041] At the same time, the voice data is analyzed and instructions and comments are converted into text using the SpeechRecognition library.

[1042] 2. AI Learning:

[1043] The analyzed data is fed into an AI model that learns cooking techniques and recipes. Specifically, PyTorch is used to train and update the model, which reflects subtleties such as cutting, heat control, and seasoning use.

[1044] 3. Providing cooking instructions and recipes:

[1045] When a user requests the reproduction of a specific dish, the server uses an AI model to generate optimal cooking instructions and recipes and provides them to the device.

[1046] 4. Feedback Processing:

[1047] Users provide feedback on their cooking results, and the server analyzes the data to further improve the AI ​​model.

[1048] Specific use cases

[1049] For example, if a user wants to perfectly recreate "seared bonito," they would follow these steps:

[1050] 1. Record the cooking process of seared bonito using a smartphone or HMD.

[1051] 2. Use an upload application to send the recorded data to the server.

[1052] 3. The server analyzes the video and audio data and trains the AI ​​model to learn the necessary cooking techniques and recipe details.

[1053] 4. When a user requests to recreate seared bonito, the server generates the optimal steps and recipe and provides it to the device.

[1054] 5. The user follows the instructions to cook the food and provides feedback on the results.

[1055] Prompt Sentence Examples

[1056] The user can request cooking instructions using prompts such as:

[1057] Instruct the AI ​​to reconstruct the cooking process for "Seared Bonito" and provide detailed step-by-step instructions, including how to cut the bonito, how to cook it, and how to press it into a firm consistency.

[1058] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1059] Step 1:

[1060] The user records the cooking process using a smartphone or head-mounted display (HMD). Video and audio are recorded during the recording. The input is real-time video and audio of the cooking process, and the output is the recorded video and audio files. Specifically, the entire cooking process is recorded by pressing the start recording button.

[1061] Step 2:

[1062] The device uploads the recorded data to the server. After the user finishes recording, they press the upload button and the video and audio files are sent to the server. The input is the recorded video and audio files, and the output is the files saved on the server. Specifically, the upload process runs in the background and the progress is displayed to the user.

[1063] Step 3:

[1064] The server converts the received data into an analyzable format. The server separates the video and audio data and temporarily stores them. The input is the original uploaded video and audio files, and the output is the separated video and audio data. Specifically, the video frames are extracted as continuous still images, and the audio data is stored as a separate audio file.

[1065] Step 4:

[1066] The server analyzes the video data and extracts each process. It uses OpenCV to analyze each frame and automatically extracts the type of ingredients, how to use cooking utensils, and steps in the cooking process. The input is the separated video data, and the output is the analyzed process information. Specifically, it performs object recognition on a frame-by-frame basis to classify ingredients and cooking utensils.

[1067] Step 5:

[1068] The server analyzes the audio data and extracts instructions and comments. It uses the SpeechRecognition library to convert the audio into text and extracts cooking instructions and explanations. The input is separated audio data, and the output is text data. Specifically, the audio file is read and converted to text through a speech recognition API.

[1069] Step 6:

[1070] The server trains the AI ​​model with the analysis results. The analyzed video and audio data is used to train the generative AI model on cooking techniques and recipes. The input is the analyzed process information and instruction text, and the output is an updated AI model. Specifically, the PyTorch framework is used to train the model based on the analyzed data.

[1071] Step 7:

[1072] The user submits a reproduction request. If the user wants to reproduce a specific dish or technique, the request is sent to the server via the device. The input is the reproduction request information, and the output is a notification that the request has been accepted. Specifically, the user enters the name of the dish or a specific technique in the app and submits it.

[1073] Step 8:

[1074] The server generates specific cooking steps and recipes and provides them to the device. Based on the learning results of the AI ​​model, the server generates optimal steps and provides them to the user. The input is the reproduction request and the trained AI model, and the output is specific cooking steps and recipe information. Specifically, a detailed recipe generated based on the prompt text is displayed on the user interface.

[1075] Step 9:

[1076] The user provides feedback on the cooking results. The results of the recreated dish are sent to the server, and the success rate of the cooking and areas for improvement are reported. The input is feedback information (photos, comments), and the output is feedback data stored on the server. Specifically, after cooking, the results are photographed and entered, and sent via the app.

[1077] Step 10:

[1078] The server analyzes the feedback and improves the AI ​​model. The feedback data is analyzed and reflected in the next learning. The input is feedback data from the user, and the output is a further improved AI model. Specifically, the feedback is aggregated and analyzed, and the model's hyperparameters are adjusted and additional learning is performed.

[1079] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1080] The system of the present invention allows users to record the cooking process, analyze the data, learn cooking techniques, and then reproduce the recipes. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system aims to increase user satisfaction. An embodiment of this system will be described in detail below.

[1081] System configuration

[1082] The system mainly consists of the following elements:

[1083] 1. User Recording Method

[1084] 2. Data upload method

[1085] 3. Server data analysis method

[1086] 4. AI learning methods

[1087] 5. How to Request Reproduction

[1088] 6. Feedback channels

[1089] 7. Emotion Engine

[1090] Program processing

[1091] The system program does the following:

[1092] Acquiring Recording Data

[1093] The system provides a means for users to record the cooking process. Users can use their smartphones or video cameras to record the cooking process of top chefs in detail. After the recording is complete, users can import the recorded data to their devices.

[1094] Uploading data

[1095] The device uploads the recorded data to the server. Data can be easily sent to the server using a dedicated application.

[1096] Data preprocessing and analysis

[1097] The server converts the received recorded data into an analyzable format. First, the video data and audio data are separated and processed separately. The video data is analyzed frame by frame to extract information such as the type of ingredients, how to use cooking utensils, and the cooking process. Meanwhile, the audio data is converted into text to extract the chef's instructions and comments.

[1098] AI Learning

[1099] The server analyzes the data and trains the AI ​​model, which allows it to accurately reproduce cooking techniques and flavors. Specifically, even minute details such as cutting methods, heat levels, and the use of seasonings are reflected in the model.

[1100] Reproduction requests and submissions

[1101] When a user wants to recreate a specific dish or technique, they send a request to the server through their device. The server generates cooking instructions and recipes based on the request and provides them to the device. The user can then follow these instructions to recreate the dish.

[1102] Feedback and model improvement

[1103] The user recreates the dish and sends feedback to the emotion engine via their device. The emotion engine analyzes the user's emotions (for example, satisfaction or difficulties) and sends this information to the server. The server receives the feedback and reflects it in the AI ​​model, further improving the accuracy of future reproductions.

[1104] Emotion Engine Operation

[1105] The emotion engine recognizes the emotions felt by the user regarding the cooking process and the results. For example, it uses facial recognition and voice analysis technology to detect the joy, difficulty, and satisfaction felt by the user in real time, and sends this data to the server. The server then analyzes this information and reflects it in the AI ​​model. This allows the system to provide optimal cooking procedures and recipes that take the user's emotions into account.

[1106] Specific examples

[1107] For example, if a user wants to make "seared bonito", the steps would be as follows:

[1108] 1. A user records a chef "searing bonito."

[1109] 2. The device uploads the recorded data to the server.

[1110] 3. The server analyzes information from the video data, such as the steps for cutting the bonito, how well it is cooked, and how to pound it, and extracts supplementary explanations from the audio data.

[1111] 4. The server trains the AI ​​model on the analysis results.

[1112] 5. The user sends a request to recreate "Seared Bonito."

[1113] 6. The server generates the optimal cooking procedure and necessary recipes and provides them to the terminal.

[1114] 7. The user cooks according to the provided instructions, and the emotion engine analyzes the user's emotions and sends them to the server.

[1115] 8. The server analyzes the emotional feedback and improves the AI ​​model.

[1116] This allows users to efficiently learn first-class cooking techniques and reproduce them in an optimal way, taking into account their emotions during the process. This system will help pass on traditional cooking techniques and improve user satisfaction.

[1117] The processing flow will be explained below.

[1118] Step 1:

[1119] The user records the cooking process. The user uses a smartphone or video camera to record the cooking process of a top chef in detail. The recording is done so that every process is clearly visible.

[1120] Step 2:

[1121] The user imports the recorded data to the device. Once recording is complete, the user transfers the video file to a device such as a smartphone or PC and opens the dedicated application.

[1122] Step 3:

[1123] The device uploads the recorded data to the server. A dedicated application is used to select the recorded data and issue an upload command to the server. The data is then securely transferred to the server.

[1124] Step 4:

[1125] The server receives the recorded data and prepares it for conversion into an analyzable format.

[1126] Step 5:

[1127] The server converts the recorded data into an analyzable format, splits the video file into frames, and separates the video data from the audio data.

[1128] Step 6:

[1129] The server analyzes the video data, analyzing each individual frame to extract each step of the cooking process (e.g., type of ingredient, cutting method, heat level), and even to identify the smallest actions and steps visible in the video.

[1130] Step 7:

[1131] The server analyzes the audio data, converts it into text, and extracts the chef's instructions and comments, providing a detailed textual description of each process.

[1132] Step 8:

[1133] The server trains the AI ​​model with the analysis results, and the extracted text from the video and audio data is input into the AI ​​model, which then learns how to reproduce the cooking techniques and flavors.

[1134] Step 9:

[1135] The user sends a reproduction request to the server. The user sends a request to reproduce a specific dish (e.g., "seared bonito") to the server via the terminal.

[1136] Step 10:

[1137] The server generates cooking instructions and recipes based on the request, calculates the optimal steps based on an AI model, and generates detailed recipes and cooking instructions.

[1138] Step 11:

[1139] The terminal provides the cooking instructions and recipes to the user, who can then view the cooking instructions and recipes obtained from the server through the terminal.

[1140] Step 12:

[1141] The user follows the instructions to cook the food, and then recreates the dish based on the instructions displayed on the device.

[1142] Step 13:

[1143] The emotion engine analyzes the user's emotions during the cooking process. The emotion engine recognizes emotions from the user's facial expressions and voice, and analyzes the satisfaction and difficulties during the cooking process in real time.

[1144] Step 14:

[1145] The emotion engine sends the analysis results to the server, which then transmits the emotion data to the server, which then reflects it in the cooking process.

[1146] Step 15:

[1147] The user provides feedback on the cooking results, sending their evaluation of the cooking results and taste to the server via their device.

[1148] Step 16:

[1149] The server receives and analyzes the feedback sent by users and uses it to improve the AI ​​model.

[1150] Step 17:

[1151] The server improves the AI ​​model, adjusting it based on feedback and emotion data to improve the accuracy of the next replay.

[1152] This process allows users to accurately learn and recreate traditional cooking techniques, and also provides an optimal cooking environment for users by taking into account their emotions during the cooking process.

[1153] Example 2

[1154] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1155] Conventional cooking skill learning systems have difficulty effectively replicating professional cooking techniques, and have been unable to properly utilize feedback to ensure users get emotionally satisfying results. Furthermore, there has been no means to analyze users' emotions in real time and reflect them in the cooking process.

[1156] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for a user to record the cooking process, a means for uploading the recorded cooking process data to the server, a means for the server to convert the received data into an analyzable format, a means for the server to analyze the video data and extract each process, a means for the server to analyze the audio data and extract instructions and comments, a means for the server to train an AI model based on the analysis results, a means for a user to send a reproduction request, a means for the server to generate specific cooking steps and recipes and provide them to the terminal, a means for the user to provide feedback on the cooking results, a means for the server to analyze the feedback and improve the AI ​​model, and an emotion analysis means, which recognizes the user's emotions in real time and transmits the analysis results to the server. This makes it possible to provide optimal cooking instructions that take the user's emotions into consideration, solving problems associated with conventional systems and enabling users to learn and reproduce cooking techniques with greater accuracy.

[1157] A "user" is an individual or entity who uses the system to learn and replicate cooking techniques.

[1158] "Recorded data" refers to video and audio data used by a user to record the cooking process.

[1159] The "server" is a central computer system that receives the recorded data, analyzes it, trains the AI ​​model, and provides the results to the user.

[1160] "Data analysis" refers to the process in which the server identifies the video and audio data it receives and extracts the cooking process, instructions, and comments.

[1161] An "AI model" is an artificial intelligence algorithm or system that learns based on analysis results and provides users with optimal cooking procedures and recipes.

[1162] A "reproduction request" is an act in which a user sends a request to the server that they would like to learn or reproduce a particular dish or technique.

[1163] "Emotion analysis" refers to the emotion analysis means recognizing the user's emotions in real time and sending the analysis results to the server.

[1164] "Feedback" includes the user reporting the results of their cooking reproduction to the system and the results of subsequent sentiment analysis.

[1165] "Cooking instructions" refers to the specific cooking methods and steps generated by the server.

[1166] A "recipe" is a set of detailed instructions detailing the ingredients and quantities needed to prepare a particular dish, as well as the cooking method involved.

[1167] "Emotion analysis means" refers to technology or devices for detecting and analyzing a user's emotions in real time, and uses facial recognition technology and voice analysis technology.

[1168] MODE FOR CARRYING OUT THE INVENTION

[1169] The system of the present invention allows users to record the cooking process, analyze the data, learn cooking techniques, and then reproduce the recipes. Furthermore, by combining it with emotion analysis means that recognizes the user's emotions, the system aims to increase user satisfaction.

[1170] This system consists of a user recording means, a data uploading means, a server data analysis means, an AI learning means, a reproduction request means, a feedback means, and an emotion analysis means.

[1171] Acquiring Recording Data

[1172] Users record the cooking process using a recording device such as a smartphone or video camera. This process includes everything from preparation to completion, and detailed footage is required. For example, when making "seared bonito," users will record in detail how they cut the bonito, how well it is cooked, and how they sear it.

[1173] Uploading data

[1174] Once the recording is complete, the user imports the recording data to their device and then uploads it to the server using a dedicated application. This process can be completed with just a few clicks.

[1175] Data preprocessing and analysis

[1176] The server converts the received video data into an analyzable format. During this process, the video data and audio data are separated and processed separately. The video data is analyzed frame by frame to extract information such as the type of ingredients, the use of cooking utensils, and the cooking process. For example, the way ingredients are cut and cooked can be identified. Meanwhile, the audio data is converted to text to extract the chef's instructions and comments.

[1177] AI Learning

[1178] The server then trains the AI ​​model with the analysis results. This training process includes detailed cooking techniques, including how to cut food, how much heat to use, and how to use seasonings, so that the AI ​​model can accurately reproduce detailed cooking techniques and flavors. This allows the AI ​​model to provide users with optimal cooking procedures and recipes.

[1179] Reproduction requests and submissions

[1180] When a user wants to recreate a specific dish or technique, they send a request to the server via their device. Based on the request, the server generates the optimal cooking steps and necessary recipes and provides them to the device. The user can then recreate the dish based on this information. An example of a specific prompt sentence is, "Please tell me the optimal cooking steps and recipe for making bonito tataki. Please also provide voice instructions."

[1181] Feedback and model improvement

[1182] After the user has finished recreating the dish, they provide feedback to the emotion analysis means via their device. The emotion analysis means analyzes the user's emotions and sends the results to the server. The server receives the feedback and uses it to improve the AI ​​model. For example, the emotion analysis means can analyze the user's satisfaction and difficulties and reflect that information in the AI ​​model.

[1183] Operation of emotion analysis means

[1184] The emotion analysis means uses facial recognition and voice analysis technologies to detect the user's emotions in real time and transmits the data to the server, which enables the system to provide optimal cooking instructions and recipes that take the user's emotions into consideration.

[1185] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1186] Step 1: Obtaining the recording data

[1187] Users can use their smartphones or video cameras to record the cooking process in detail, for example, the entire cooking process of "seared bonito."

[1188] Input: Video and audio data stored on the recording device.

[1189] How it works: The user starts recording and then imports the recording to their device after shooting.

[1190] Output: Recording data file saved on the device.

[1191] Step 2: Upload your data

[1192] The device uploads the recorded data to the server using a dedicated application.

[1193] Input: Recording data files saved on the device.

[1194] How it works: The user launches the dedicated app and clicks the "upload" button to send data to the server.

[1195] Output: Recording data file saved on the server.

[1196] Step 3: Data preprocessing and analysis

[1197] The server converts the recorded data into an analyzable format, separating the video and audio data and processing each separately.

[1198] Input: Recording data files stored on the server.

[1199] How it works: The server separates the video data from the audio data and analyzes the video data frame by frame. Specifically, it extracts information such as the type of ingredients, how to use cooking utensils, and the cooking process. Meanwhile, the audio data is converted into text and the chef's instructions and comments are extracted.

[1200] Output: Analyzable video data and audio data in text format.

[1201] Step 4: AI training

[1202] The data analyzed by the server is trained into an AI model.

[1203] Input: Analyzable video data and audio data in text format.

[1204] How it works: The server inputs the data into the AI ​​model and begins the learning process, incorporating even the smallest details like cutting techniques, heat levels, and seasonings.

[1205] Output: The trained AI model.

[1206] Step 5: Request and provide a reproduction

[1207] When a user wants to recreate a particular dish or technique, they send a request to the server through their device.

[1208] Input: The reproduction request submitted by the user.

[1209] How it works: The server generates the optimal cooking instructions and recipe based on the request and sends it to the device. For example, if the request is "I want to recreate seared bonito," the server will provide the instructions and recipe for that.

[1210] Output: Specific cooking instructions and recipe displayed on the user's device.

[1211] Step 6: Feedback and model improvement

[1212] After the user has completed recreating the dish, they provide feedback to the sentiment analysis means.

[1213] Input: User-provided feedback data.

[1214] Operation: The emotion analysis means analyzes the feedback data and sends it to the server. For example, the satisfaction or difficulties felt during the cooking process are specifically input.

[1215] Output: Feedback analysis data sent to the server.

[1216] Step 7: Sentiment analysis method in action

[1217] The emotion analysis means recognizes the user's emotions in real time and transmits the analysis results to the server.

[1218] Input: Real-time video and audio data of the user cooking.

[1219] How it works: The emotion analysis unit uses face recognition and voice analysis technology to detect the user's emotions and sends the data to the server. For example, it analyzes emotions such as joy or difficulty.

[1220] Output: Real-time emotion data sent to the server.

[1221] (Application example 2)

[1222] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1223] Conventional food delivery systems lacked the means to effectively manage and improve food quality and user satisfaction. In particular, there were many challenges in the reproducibility of cooking processes, the skill acquisition of delivery staff, and the management of customer feedback. The purpose of this invention is to solve these problems, improve the quality of delivery services, and increase the satisfaction of customers and delivery staff.

[1224] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to record the cooking process and upload the data, means for analyzing the uploaded recorded data and extracting each process from the video data, means for extracting instructions and comments from the audio data, means for training the generative AI model based on the analysis results, means for a user to send a reproduction request, generate specific cooking steps and recipes, and provide them to the terminal, means for a user to provide feedback on the cooking results and for the server to analyze the feedback and improve the generative AI model, means for a delivery staff member to record the cooking process, means for uploading the recorded data to a cloud server, means for the cloud server to analyze the uploaded recorded data and extract cooking skills, and means for analyzing the emotions of the delivery staff member and sending feedback to the server. This improves the reproducibility of the cooking process, enabling delivery staff to acquire skills and customer feedback to be managed more effectively.

[1225] "Means for users to record the cooking process" refers to devices or methods that allow users to record the cooking process as video.

[1226] The "means for uploading recorded data of the cooking process to the server" is a means for transmitting video data of the cooking process recorded by the user to the server via the Internet.

[1227] "Means for converting data received by the server into an analyzable format" refers to a method or algorithm for converting data uploaded to the server into a format that can be easily analyzed.

[1228] "Means for the server to analyze video data and extract each process" refers to a method or system for analyzing the content of video data and identifying and extracting each step of food preparation and cooking.

[1229] "Means for the server to analyze the audio data and extract instructions and comments" refers to methods and technologies for identifying cooking instructions and comments from the recorded audio data and extracting them as text.

[1230] "Means for the server to train the generated AI model on the analysis results" refers to a method for training the AI ​​model generated based on the analysis results extracted from video data and audio data on new data.

[1231] The "means by which a user sends a reproduction request" refers to an interface or method by which a user can send a request to the server when they want to reproduce a particular dish or technique.

[1232] "Means for the server to generate specific cooking procedures and recipes and provide them to the terminal" refers to a system that generates optimal cooking procedures and recipes based on the user's reproduction request and provides them to the user's terminal.

[1233] "Means for users to provide feedback on cooking results" refers to methods or tools that allow users to return the results of their recreated dishes to the server as their impressions or evaluations.

[1234] "Means for the server to analyze feedback and improve the generative AI model" refers to a system or method for analyzing feedback from users and improving the generative AI model based on the results.

[1235] "Means for delivery staff to record the cooking process" refers to devices or methods that delivery staff use to record the cooking process of food on video.

[1236] "Means for uploading recorded data to a cloud server" refers to means for saving recorded data to a cloud server via the Internet.

[1237] "Means for the cloud server to analyze the uploaded video data and extract cooking techniques" refers to a method or system for the cloud server to analyze the video data received and identify and extract cooking techniques and processes.

[1238] "Means for analyzing the emotions of delivery staff and sending feedback to a server" refers to a method or technology for analyzing the emotions of delivery staff from video and audio data and sending the results as feedback to a server.

[1239] This invention aims to develop a system that aims to improve customer satisfaction and control the quality of the cooking process in food delivery services. The system consists of the following components:

[1240] 1. User recording method: The user or delivery staff uses a recording device such as a smartphone to record the cooking process. The video data contains detailed information such as the cooking steps, ingredients used, and necessary cooking utensils.

[1241] 2. Data uploading method: Users use a dedicated application to upload recorded data to the cloud server, which allows data to be easily sent to the server via the Internet.

[1242] 3. Server data analysis method: The server converts the received video data into an analyzable format. Specifically, the video data is divided into frames and image recognition technology (e.g., OpenCV) is used to analyze the type of ingredients and cooking process. Additionally, audio data is converted into text using speech recognition technology (e.g., Google Speech-to-Text) to extract cooking comments and instructions.

[1243] 4. AI learning method: The generative AI model learns cooking techniques based on the analyzed data. This model is built using a machine learning library (e.g., TensorFlow). The model reflects detailed cooking techniques, such as how to cut ingredients, how to control heat, and how to use seasonings.

[1244] 5. Reproduction request method: When a user wants to reproduce a specific dish, they send a request to the server through a dedicated application. Based on this request, the server generates the optimal cooking procedure and recipe and provides it to the user's device.

[1245] 6. Feedback method: The user provides feedback on the cooking results to the server through the application. The feedback is analyzed using facial recognition technology (e.g., DeepFace) to quantify the user's satisfaction.

[1246] 7. Emotion Engine: The emotion engine recognizes the user's emotions in real time and transmits the user's joy, difficulty, and satisfaction to the server, which analyzes this data and improves the generative AI model.

[1247] The above system will improve the reproducibility of the cooking process, enable delivery staff to acquire skills, and enable effective management of customer feedback. It will also optimize the recipes and cooking procedures provided to users, which is expected to improve customer satisfaction.

[1248] Examples:

[1249] For example, a delivery staff member serves a highly rated dish to a customer and records the cooking process. The video data is uploaded to a cloud server, which analyzes the data and learns the cooking process and techniques. If another user requests to make the same dish, the server will provide the optimal cooking procedure and recipe, and the user will provide feedback on the results, and the emotion engine will analyze the customer's satisfaction level.

[1250] Example of an input prompt:

[1251] "I want to analyze videos of the cooking process taken by delivery staff and collect emotional feedback. Please generate a report that includes the analysis results and emotional feedback."

[1252] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1253] Step 1:

[1254] The user records the cooking process using a means to record the cooking process. Specifically, the user or delivery staff uses a smartphone or video camera to record the cooking process from start to finish. The input is video data of the cooking process, and the output is the recorded video file.

[1255] Step 2:

[1256] The recorded cooking process data is uploaded to the server. The user uses a dedicated upload application to send the recorded data to the cloud server. The input is a video file, and the output is video data on the cloud server.

[1257] Step 3:

[1258] The server converts the received data into an analyzable format. The cloud server divides the received video file into frames and separates it into image data and audio data. It also converts the audio data into text. The input is video data on the cloud server, and the output is image data and text data.

[1259] Step 4:

[1260] The server analyzes the video data and extracts each process. Using machine learning and image recognition technology (e.g., OpenCV), the server identifies the type of ingredients, cooking utensils, and cooking process from the image data for each frame. The input is the image data for each frame, and the output is the extracted cooking process information.

[1261] Step 5:

[1262] The server analyzes the voice data and extracts instructions and comments. The server uses voice recognition technology (e.g., Google Speech-to-Text) to convert the voice data into text and identify cooking instructions and comments. The input is the voice data, and the output is the extracted text data.

[1263] Step 6:

[1264] The server trains the generative AI model with the analysis results. The server inputs the analyzed cooking process information and text data into the generative AI model (e.g., TensorFlow) and trains the model to learn new cooking techniques. The input is the cooking process information and text data, and the output is an updated generative AI model.

[1265] Step 7:

[1266] The system uses a means for users to send a reproduction request. A user sends a request to reproduce a specific dish or cooking technique through a dedicated application. The input is the reproduction request, and the output is the request data sent to the server.

[1267] Step 8:

[1268] The server generates specific cooking steps and recipes and provides them to the device. The server generates optimal cooking steps and recipes based on the generative AI model and provides this information to the user's device. The input is the generative AI model and the user's reproduction request, and the output is the cooking steps and recipe provided to the user's device.

[1269] Step 9:

[1270] The system uses a means for users to provide feedback on their cooking results. Users cook and provide feedback on the results through a dedicated application. The input is the user's cooking results, and the output is feedback data to the server.

[1271] Step 10:

[1272] The server analyzes the feedback and improves the generative AI model. The server analyzes the user feedback data, evaluates customer satisfaction using an emotion engine, and reflects the results in the generative AI model. The input is the feedback data, and the output is an improved generative AI model.

[1273] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1274] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1275] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1276] [Fourth embodiment]

[1277] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1278] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1279] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1280] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1281] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1282] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1283] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1284] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1285] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1286] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1287] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1288] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1289] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1290] The system of the present invention allows users to easily record and analyze the cooking process, and based on that, learn and reproduce cooking techniques. An embodiment of the system will be described in detail below.

[1291] System configuration

[1292] The system mainly consists of the following elements:

[1293] 1. User Recording Method

[1294] 2. Data upload method

[1295] 3. Server data analysis method

[1296] 4. AI learning methods

[1297] 5. How to Request Reproduction

[1298] 6. Feedback channels

[1299] Program processing

[1300] The system program does the following:

[1301] Acquiring Recording Data

[1302] The user uses a means to record the cooking process. For example, the user uses a smartphone or video camera to record the cooking process. After the recording is finished, the user imports the recorded data into the device.

[1303] Uploading data

[1304] The device uploads the recorded data to the server. Data can be easily sent to the server using a dedicated application.

[1305] Data preprocessing and analysis

[1306] The server converts the received recorded data into an analyzable format. First, the video data and audio data are separated and processed separately. The video data is analyzed frame by frame to extract information such as the type of ingredients, how to use cooking utensils, and the cooking process. Meanwhile, the audio data is converted into text to extract the chef's instructions and comments.

[1307] AI Learning

[1308] The server analyzes the data and trains the AI ​​model, which allows it to accurately reproduce cooking techniques and flavors. Specifically, even minute details such as cutting methods, heat levels, and the use of seasonings are reflected in the model.

[1309] Reproduction requests and submissions

[1310] When a user wants to recreate a specific dish or technique, they send a request to the server through their device. The server generates cooking instructions and recipes based on the request and provides them to the device. The user can then follow these instructions to recreate the dish.

[1311] Feedback and model improvement

[1312] The user recreates the dish and provides feedback on the results. The device then sends the feedback to the server, which analyzes the information and improves the AI ​​model, further improving the accuracy of future attempts.

[1313] Specific examples

[1314] For example, if a user wants to make "seared bonito", the steps would be as follows:

[1315] 1. A user records a chef "searing bonito."

[1316] 2. The device uploads the recorded data to the server.

[1317] 3. The server analyzes information from the video data, such as the steps for cutting the bonito, how well it is cooked, and how to pound it, and extracts supplementary explanations from the audio data.

[1318] 4. The server trains the AI ​​model on the analysis results.

[1319] 5. The user sends a request to recreate "Seared Bonito."

[1320] 6. The server generates the optimal cooking procedure and necessary recipes and provides them to the terminal.

[1321] 7. The user follows the provided instructions to cook the food and provides feedback on the results.

[1322] 8. The server analyzes the feedback and improves the AI ​​model.

[1323] This allows users to efficiently learn and reproduce first-class cooking techniques. This system also enables the efficient and accurate inheritance of traditional cooking techniques.

[1324] The processing flow will be explained below.

[1325] Step 1:

[1326] The user records the cooking process. The user uses a smartphone or video camera to record the cooking process of a top chef in detail. The recording is done so that every process is clearly visible.

[1327] Step 2:

[1328] The user imports the recorded data to the device. Once recording is complete, the user transfers the video file to a device such as a smartphone or PC and opens the dedicated application.

[1329] Step 3:

[1330] The device uploads the recorded data to the server. A dedicated application is used to select the recorded data and issue an upload command to the server. The data is then securely transferred to the server.

[1331] Step 4:

[1332] The server receives the recorded data and prepares it for conversion into an analyzable format.

[1333] Step 5:

[1334] The server converts the recorded data into an analyzable format, splits the video file into frames, and separates the video data from the audio data.

[1335] Step 6:

[1336] The server analyzes the video data, analyzing each individual frame to extract each step of the cooking process (e.g., type of ingredient, cutting method, heat level), and even to identify the smallest actions and steps visible in the video.

[1337] Step 7:

[1338] The server analyzes the audio data, converts it into text, and extracts the chef's instructions and comments, providing a detailed textual description of each process.

[1339] Step 8:

[1340] The server trains the AI ​​model with the analysis results, and the extracted text from the video and audio data is input into the AI ​​model, which then learns how to reproduce the cooking techniques and flavors.

[1341] Step 9:

[1342] The user sends a reproduction request to the server. The user sends a request to reproduce a specific dish (e.g., "seared bonito") to the server via the terminal.

[1343] Step 10:

[1344] The server generates cooking instructions and recipes based on the request, calculates the optimal steps based on an AI model, and generates detailed recipes and cooking instructions.

[1345] Step 11:

[1346] The terminal provides the cooking instructions and recipes to the user, who can then view the cooking instructions and recipes obtained from the server through the terminal.

[1347] Step 12:

[1348] The user follows the instructions to cook the food, and then recreates the dish based on the instructions displayed on the device.

[1349] Step 13:

[1350] The user provides feedback on the cooking results, sending their evaluation of the cooking results and taste to the server via their device.

[1351] Step 14:

[1352] The server receives and analyzes the feedback sent by users and uses it to improve the AI ​​model.

[1353] Step 15:

[1354] The server improves the AI ​​model, adjusting it based on the feedback to improve the accuracy of the next replay.

[1355] This process allows users to accurately learn and recreate traditional cooking techniques.

[1356] Example 1

[1357] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1358] Traditional cooking skill learning methods often rely on watching videos or reading books, making it difficult to fully understand the details of the actual cooking process. It is particularly difficult to accurately reproduce the techniques of professional chefs, making it difficult for learners to improve their skills. Furthermore, there are limited ways to improve one's own cooking skills through feedback, making it difficult to efficiently acquire new cooking skills.

[1359] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1360] In this invention, the server includes means for a user to record the cooking process, means for uploading data of the recorded cooking process to an information processing device, means for the information processing device to convert the received data into an analyzable format, means for the information processing device to analyze the video data and extract each process, means for the information processing device to analyze the audio data and extract instructions and comments, means for the information processing device to train a learning model based on the analysis results, means for a user to send a reproduction request, means for the information processing device to generate specific cooking steps and recipes and provide them to the terminal, means for a user to provide feedback on the cooking results, and means for the information processing device to analyze the feedback and improve the learning model. This enables users to accurately learn and reproduce professional cooking techniques and improve their own skills.

[1361] "User" refers to any individual or legal entity that records the cooking process and uploads the data to the system.

[1362] The "cooking process" refers to the series of steps involved in preparing ingredients, cooking, and finishing.

[1363] "Recording means" refers to a device for recording video and audio, such as a smartphone or video camera.

[1364] "Data" refers to video data and audio data acquired by a recording means.

[1365] "Information processing device" refers to a device that analyzes, processes, and stores data, such as a server or cloud service.

[1366] "Uploading means" refers to the means by which a user sends recorded data to an information processing device, including dedicated applications and web forms.

[1367] The term "analyzable format" refers to a format required for an information processing device to process data appropriately.

[1368] "Video data" refers to video frame information captured by a recording means.

[1369] "Audio data" refers to audio information captured by a recording means.

[1370] The "means for extracting each process" refers to a means for analyzing and identifying each stage of cooking from video data and audio data.

[1371] "Instructions and comments" refers to statements and explanations made by the chef or user regarding the cooking process.

[1372] "Learning model" refers to analytical algorithms and data models that use artificial intelligence to learn cooking techniques and processes.

[1373] A "reproduction request" refers to a user requesting the system to reproduce a specific dish or technique.

[1374] "Cooking instructions and recipes" refers to the specific steps and ingredients needed to recreate a dish.

[1375] "Terminal" refers to a device used by a user, such as a smartphone, tablet, or PC.

[1376] "Feedback" refers to information such as cooking results and impressions that users provide to the system.

[1377] "Analysis means" refers to the algorithms and techniques used by the information processing device to process and analyze data.

[1378] The system of the present invention allows users to easily record and analyze the cooking process, and then learn and reproduce cooking techniques based on the recorded footage. To implement this system, the following hardware and software are required: a user's smartphone, a video camera, a personal computer, a dedicated application, an information processing device (server), and a learning model.

[1379] Hardware and software used

[1380] 1. Smartphone / Video Camera: Used as a recording medium.

[1381] 2. Computer / dedicated application: Used to import and upload data.

[1382] 3. Information processing device (server): Used to analyze data, manage learning models, generate cooking instructions and recipes, and analyze feedback.

[1383] 4. Learning model: A model that uses AI technology and frameworks such as TensorFlow or PyTorch.

[1384] Specific actions and processes

[1385] The user uses a smartphone or video camera to record the cooking process. For example, the process of making "seared bonito" is recorded. The recorded data is then imported into the device and uploaded to an information processing device using a dedicated application. When uploading, the user sends the data by pressing the "upload" button in the application.

[1386] After receiving the data, the server separates the video data from the audio data. The video data is analyzed frame by frame to extract information such as the type of ingredients, how to use cooking utensils, and the cooking process. The audio data is converted to text and instructions and comments are extracted using natural language processing (NLP), for example, using the Google Cloud Speech-to-Text API.

[1387] The server then trains the learning model based on the analysis results. The learning model uses AI technology, such as TensorFlow or PyTorch, to accurately reproduce cooking techniques and flavors.

[1388] When a user wants to recreate a specific dish or technique, they send a request to the server via their device. A request is made by pressing the "reproduce request" button within the application. At this time, by specifying the specific dish name and technique, for example, "I want to recreate seared bonito," the request is sent to the server.

[1389] The server receives requests from users and generates optimal cooking instructions and recipes. The generated information is provided to the device, and the user recreates the dish according to the instructions. For example, the next cooking steps are displayed within the application.

[1390] After the user has finished cooking, they can provide feedback on their results and impressions. By filling out and submitting the feedback form within the dedicated application, the feedback is sent to the server, which then analyzes and improves the model. This improves the accuracy of the reproducibility from the next time onwards.

[1391] Examples and prompts

[1392] For example, a user who wants to make "seared bonito" would follow these steps:

[1393] 1. The user records the cooking process on their smartphone and uploads the recording data to the server using the application.

[1394] 2. The server analyzes the data and processes the video and audio data separately.

[1395] 3. The server trains the AI ​​model on the analysis results.

[1396] 4. The user sends a request to recreate "Seared Bonito" within the app.

[1397] 5. The server generates optimal cooking instructions and recipes and provides them to the device.

[1398] 6. The user follows the provided steps to recreate the dish and receives feedback on the results.

[1399] Prompt Sentence Examples

[1400] "I want to upload video data of the cooking process and have the AI ​​model learn from it. Please tell me the specific steps and tools I need to use."

[1401] "Analyze the cooking steps for seared bonito and generate the optimal recipe."

[1402] "I want to provide feedback to improve the AI ​​model. How do I do that?"

[1403] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1404] Step 1: Obtaining Recording Data

[1405] The user uses a smartphone or video camera to record the cooking process. For example, the process of making "seared bonito" is used as the recording process. The input is the recorded data that includes the entire cooking process. Once the recording is finished, the recorded data is imported to a device such as a smartphone or PC. The output is the recorded data saved on the device.

[1406] Step 2: Upload your data

[1407] The device uses a dedicated application to upload recorded data to the server. The input is the recorded data stored on the device, and the output is the recorded data stored on the server. When the dedicated application is launched on the device and the "upload button" is pressed, the data is sent to the server.

[1408] Step 3: Preprocessing the data

[1409] The recorded data received by the server is converted into an analyzable format. First, the video data and audio data are separated. The input is the recorded data stored on the server, and the output is the separated video and audio data. Specifically, the video data is divided into 30 frames per second, and the audio data is saved as an audio file.

[1410] Step 4: Analyze the data

[1411] The server analyzes the preprocessed data. The input is the separated video and audio data, and the output is the extracted cooking process information and transcribed instructions and comments. Computer vision technology is used to analyze the video data for ingredients, utensil usage, cooking process, etc. Specifically, each frame is analyzed using image recognition technology (e.g., YOLO, OpenCV) to identify ingredients and utensils.

[1412] Step 5: Analyze the audio data

[1413] The server analyzes the voice data and converts it into text. The input is voice data, and the output is text data such as instructions or comments. Voice recognition technology (e.g., Google Cloud Speech-to-Text API) is used to convert the voice data into text format.

[1414] Step 6: Training the AI ​​model

[1415] The server trains the learning model with the analysis results. The input is the extracted cooking process information and text instructions and comments, and the output is the trained model. Specifically, the analysis results are fed into the learning model using TensorFlow or PyTorch to improve the model's reproducibility.

[1416] Step 7: Accepting a reproduction request

[1417] When a user wants to recreate a specific dish or technique, they send a request to the server via their device. The input is the user's request to recreate it, and the output is the server's acceptance of the request. The user presses the "recreate request" button in the dedicated application to send a request to recreate "seared bonito."

[1418] Step 8: Generate cooking instructions and recipes

[1419] The server receives requests from users and generates optimal cooking instructions and the necessary recipes. The input is the reproduction request and the trained model, and the output is the specific cooking instructions and recipes. Specifically, the server uses the trained model to calculate the optimal cooking method for the requested dish and provides it in text format.

[1420] Step 9: Provide and Reproduce to Users

[1421] The server provides the generated cooking instructions and recipes to the device. The input is the cooking instructions and recipe, and the output is the cooking instructions and recipe displayed on the user's device. The user recreates the dish by checking the steps for each step in the application.

[1422] Step 10: Submit your feedback

[1423] The user provides feedback on the cooking results and their impressions. The input is the user's feedback information, and the output is the feedback data sent to the server. After the user finishes cooking, they enter their impressions and suggestions for improvement in the feedback form within the application and submit it.

[1424] Step 11: Improve the model

[1425] The server receives the feedback and analyzes the information to improve the learning model. The input is the feedback data, and the output is an improved learning model. Specifically, the server fine-tunes the model parameters based on the feedback and adds new data for learning, thereby improving the recall accuracy.

[1426] (Application example 1)

[1427] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1428] Traditional methods of learning cooking techniques and recipes require specialized knowledge and numerous manuals, which takes a great deal of time and effort. Furthermore, chefs and cooking staff who want to learn new techniques and recipes have few efficient ways to learn, limiting their ability to improve their skills even when practicing in a brick-and-mortar restaurant. To solve these problems, a system is needed that can easily record and analyze the cooking process, allowing for efficient learning of techniques and recipes.

[1429] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1430] In this invention, the server includes: a means for a user to record the cooking process; a means for uploading the recorded data of the cooking process to the server; a means for the server to convert the received data into an analyzable format; a means for the server to analyze the video data and extract each process; a means for the server to analyze the audio data and extract instructions and comments; a means for the server to train an AI model based on the analysis results; a means for a user to send a reproduction request; a means for the server to generate specific cooking steps and recipes and provide them to a terminal; a means for a user to provide feedback on the cooking results; a means for the server to analyze the feedback and improve the AI ​​model; and a means used by chefs and staff in the restaurant to learn new techniques and recipes. This makes it possible to efficiently and effectively learn cooking techniques and new recipes.

[1431] A "user" is a person or cook who uses the system to record and analyze the cooking process and learn new techniques and recipes.

[1432] "Means for recording" refers to devices or equipment for recording the cooking process on video.

[1433] "Means for uploading data to a server" refers to the function or process of transmitting recorded video data to a server via the Internet.

[1434] "Means of converting into an analyzable format" refers to the process by which the server converts the data it receives into a format that can be processed by AI models or other data analysis tools.

[1435] "Means for analyzing video data" refers to a method for analyzing recorded video data and extracting each cooking process and step.

[1436] The "means for analyzing audio data" is a method for analyzing recorded audio data and extracting verbal instructions and comments.

[1437] "Means for training an AI model" refers to a method of inputting analyzed data into an artificial intelligence model to improve and optimize cooking techniques and recipes.

[1438] The "means for sending a reproduction request" is a function or process by which a user sends a request to the server to reproduce a particular dish or technique.

[1439] "Means for generating specific cooking steps and recipes and providing them to the terminal" refers to a method in which the server generates specific cooking steps and recipes based on the analysis and learning results and provides them to the user's terminal.

[1440] The "means for feeding back cooking results" refers to a function or process that reports the results of cooking performed by the user to the server.

[1441] "Means for analyzing feedback and improving the AI ​​model" refers to the process of analyzing feedback data obtained from users and updating and improving the model.

[1442] "Methods used by in-store chefs and staff to learn new techniques and recipes" refers to methods and tools that allow chefs and culinary staff in physical stores to efficiently learn new techniques and recipes.

[1443] DETAILED DESCRIPTION OF THE INVENTION The following describes an embodiment of the present invention.

[1444] The system's program is designed to allow users to learn and reproduce new techniques and recipes by recording the cooking process and uploading the data to a server for analysis. Specific hardware and software configurations and processing are explained below.

[1445] Hardware and software used

[1446] 1. Hardware:

[1447] Smartphone or head-mounted display (HMD): Used to record and play back the cooking process.

[1448] Server: Stores data, analyzes it, performs AI learning, and processes feedback.

[1449] 2. Software:

[1450] Recording and uploading application: An application installed on a smartphone or HMD to upload recorded data to a server.

[1451] Data analysis software: Video and audio data analysis is performed on the server, using, for example, OpenCV (video processing) and SpeechRecognition (audio processing) libraries.

[1452] AI Model: A generative AI model built using the PyTorch framework.

[1453] User Interface (UI): An application that allows for easy cooking instructions requests and feedback.

[1454] Data processing and data calculation

[1455] The user records the cooking process using a smartphone or HMD. The recorded data is sent to the server via an upload application. The data received by the server is processed as follows:

[1456] 1. Data preprocessing:

[1457] The server splits the video data into frames and extracts detailed information such as the type of ingredients, the use of cooking utensils, the cooking process, etc. It analyzes the video using the OpenCV library.

[1458] At the same time, the voice data is analyzed and instructions and comments are converted into text using the SpeechRecognition library.

[1459] 2. AI Learning:

[1460] The analyzed data is fed into an AI model that learns cooking techniques and recipes. Specifically, PyTorch is used to train and update the model, which reflects subtleties such as cutting, heat control, and seasoning use.

[1461] 3. Providing cooking instructions and recipes:

[1462] When a user requests the reproduction of a specific dish, the server uses an AI model to generate optimal cooking instructions and recipes and provides them to the device.

[1463] 4. Feedback Processing:

[1464] Users provide feedback on their cooking results, and the server analyzes the data to further improve the AI ​​model.

[1465] Specific use cases

[1466] For example, if a user wants to perfectly recreate "seared bonito," they would follow these steps:

[1467] 1. Record the cooking process of seared bonito using a smartphone or HMD.

[1468] 2. Use an upload application to send the recorded data to the server.

[1469] 3. The server analyzes the video and audio data and trains the AI ​​model to learn the necessary cooking techniques and recipe details.

[1470] 4. When a user requests to recreate seared bonito, the server generates the optimal steps and recipe and provides it to the device.

[1471] 5. The user follows the instructions to cook the food and provides feedback on the results.

[1472] Prompt Sentence Examples

[1473] The user can request cooking instructions using prompts such as:

[1474] Instruct the AI ​​to reconstruct the cooking process for "Seared Bonito" and provide detailed step-by-step instructions, including how to cut the bonito, how to cook it, and how to press it into a firm consistency.

[1475] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1476] Step 1:

[1477] The user records the cooking process using a smartphone or head-mounted display (HMD). Video and audio are recorded during the recording. The input is real-time video and audio of the cooking process, and the output is the recorded video and audio files. Specifically, the entire cooking process is recorded by pressing the start recording button.

[1478] Step 2:

[1479] The device uploads the recorded data to the server. After the user finishes recording, they press the upload button and the video and audio files are sent to the server. The input is the recorded video and audio files, and the output is the files saved on the server. Specifically, the upload process runs in the background and the progress is displayed to the user.

[1480] Step 3:

[1481] The server converts the received data into an analyzable format. The server separates the video and audio data and temporarily stores them. The input is the original uploaded video and audio files, and the output is the separated video and audio data. Specifically, the video frames are extracted as continuous still images, and the audio data is stored as a separate audio file.

[1482] Step 4:

[1483] The server analyzes the video data and extracts each process. It uses OpenCV to analyze each frame and automatically extracts the type of ingredients, how to use cooking utensils, and steps in the cooking process. The input is the separated video data, and the output is the analyzed process information. Specifically, it performs object recognition on a frame-by-frame basis to classify ingredients and cooking utensils.

[1484] Step 5:

[1485] The server analyzes the audio data and extracts instructions and comments. It uses the SpeechRecognition library to convert the audio into text and extracts cooking instructions and explanations. The input is separated audio data, and the output is text data. Specifically, the audio file is read and converted to text through a speech recognition API.

[1486] Step 6:

[1487] The server trains the AI ​​model with the analysis results. The analyzed video and audio data is used to train the generative AI model on cooking techniques and recipes. The input is the analyzed process information and instruction text, and the output is an updated AI model. Specifically, the PyTorch framework is used to train the model based on the analyzed data.

[1488] Step 7:

[1489] The user submits a reproduction request. If the user wants to reproduce a specific dish or technique, the request is sent to the server via the device. The input is the reproduction request information, and the output is a notification that the request has been accepted. Specifically, the user enters the name of the dish or a specific technique in the app and submits it.

[1490] Step 8:

[1491] The server generates specific cooking steps and recipes and provides them to the device. Based on the learning results of the AI ​​model, the server generates optimal steps and provides them to the user. The input is the reproduction request and the trained AI model, and the output is specific cooking steps and recipe information. Specifically, a detailed recipe generated based on the prompt text is displayed on the user interface.

[1492] Step 9:

[1493] The user provides feedback on the cooking results. The results of the recreated dish are sent to the server, and the success rate of the cooking and areas for improvement are reported. The input is feedback information (photos, comments), and the output is feedback data stored on the server. Specifically, after cooking, the results are photographed and entered, and sent via the app.

[1494] Step 10:

[1495] The server analyzes the feedback and improves the AI ​​model. The feedback data is analyzed and reflected in the next learning. The input is feedback data from the user, and the output is a further improved AI model. Specifically, the feedback is aggregated and analyzed, and the model's hyperparameters are adjusted and additional learning is performed.

[1496] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1497] The system of the present invention allows users to record the cooking process, analyze the data, learn cooking techniques, and then reproduce the recipes. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system aims to increase user satisfaction. An embodiment of this system will be described in detail below.

[1498] System configuration

[1499] The system mainly consists of the following elements:

[1500] 1. User Recording Method

[1501] 2. Data upload method

[1502] 3. Server data analysis method

[1503] 4. AI learning methods

[1504] 5. How to Request Reproduction

[1505] 6. Feedback channels

[1506] 7. Emotion Engine

[1507] Program processing

[1508] The system program does the following:

[1509] Acquiring Recording Data

[1510] The system provides a means for users to record the cooking process. Users can use their smartphones or video cameras to record the cooking process of top chefs in detail. After the recording is complete, users can import the recorded data to their devices.

[1511] Uploading data

[1512] The device uploads the recorded data to the server. Data can be easily sent to the server using a dedicated application.

[1513] Data preprocessing and analysis

[1514] The server converts the received recorded data into an analyzable format. First, the video data and audio data are separated and processed separately. The video data is analyzed frame by frame to extract information such as the type of ingredients, how to use cooking utensils, and the cooking process. Meanwhile, the audio data is converted into text to extract the chef's instructions and comments.

[1515] AI Learning

[1516] The server analyzes the data and trains the AI ​​model, which allows it to accurately reproduce cooking techniques and flavors. Specifically, even minute details such as cutting methods, heat levels, and the use of seasonings are reflected in the model.

[1517] Reproduction requests and submissions

[1518] When a user wants to recreate a specific dish or technique, they send a request to the server through their device. The server generates cooking instructions and recipes based on the request and provides them to the device. The user can then follow these instructions to recreate the dish.

[1519] Feedback and model improvement

[1520] The user recreates the dish and sends feedback to the emotion engine via their device. The emotion engine analyzes the user's emotions (for example, satisfaction or difficulties) and sends this information to the server. The server receives the feedback and reflects it in the AI ​​model, further improving the accuracy of future reproductions.

[1521] Emotion Engine Operation

[1522] The emotion engine recognizes the emotions felt by the user regarding the cooking process and the results. For example, it uses facial recognition and voice analysis technology to detect the joy, difficulty, and satisfaction felt by the user in real time, and sends this data to the server. The server then analyzes this information and reflects it in the AI ​​model. This allows the system to provide optimal cooking procedures and recipes that take the user's emotions into account.

[1523] Specific examples

[1524] For example, if a user wants to make "seared bonito", the steps would be as follows:

[1525] 1. A user records a chef "searing bonito."

[1526] 2. The device uploads the recorded data to the server.

[1527] 3. The server analyzes information from the video data, such as the steps for cutting the bonito, how well it is cooked, and how to pound it, and extracts supplementary explanations from the audio data.

[1528] 4. The server trains the AI ​​model on the analysis results.

[1529] 5. The user sends a request to recreate "Seared Bonito."

[1530] 6. The server generates the optimal cooking procedure and necessary recipes and provides them to the terminal.

[1531] 7. The user cooks according to the provided instructions, and the emotion engine analyzes the user's emotions and sends them to the server.

[1532] 8. The server analyzes the emotional feedback and improves the AI ​​model.

[1533] This allows users to efficiently learn first-class cooking techniques and reproduce them in an optimal way, taking into account their emotions during the process. This system will help pass on traditional cooking techniques and improve user satisfaction.

[1534] The processing flow will be explained below.

[1535] Step 1:

[1536] The user records the cooking process. The user uses a smartphone or video camera to record the cooking process of a top chef in detail. The recording is done so that every process is clearly visible.

[1537] Step 2:

[1538] The user imports the recorded data to the device. Once recording is complete, the user transfers the video file to a device such as a smartphone or PC and opens the dedicated application.

[1539] Step 3:

[1540] The device uploads the recorded data to the server. A dedicated application is used to select the recorded data and issue an upload command to the server. The data is then securely transferred to the server.

[1541] Step 4:

[1542] The server receives the recorded data and prepares it for conversion into an analyzable format.

[1543] Step 5:

[1544] The server converts the recorded data into an analyzable format, splits the video file into frames, and separates the video data from the audio data.

[1545] Step 6:

[1546] The server analyzes the video data, analyzing each individual frame to extract each step of the cooking process (e.g., type of ingredient, cutting method, heat level), and even to identify the smallest actions and steps visible in the video.

[1547] Step 7:

[1548] The server analyzes the audio data, converts it into text, and extracts the chef's instructions and comments, providing a detailed textual description of each process.

[1549] Step 8:

[1550] The server trains the AI ​​model with the analysis results, and the extracted text from the video and audio data is input into the AI ​​model, which then learns how to reproduce the cooking techniques and flavors.

[1551] Step 9:

[1552] The user sends a reproduction request to the server. The user sends a request to reproduce a specific dish (e.g., "seared bonito") to the server via the terminal.

[1553] Step 10:

[1554] The server generates cooking instructions and recipes based on the request, calculates the optimal steps based on an AI model, and generates detailed recipes and cooking instructions.

[1555] Step 11:

[1556] The terminal provides the cooking instructions and recipes to the user, who can then view the cooking instructions and recipes obtained from the server through the terminal.

[1557] Step 12:

[1558] The user follows the instructions to cook the food, and then recreates the dish based on the instructions displayed on the device.

[1559] Step 13:

[1560] The emotion engine analyzes the user's emotions during the cooking process. The emotion engine recognizes emotions from the user's facial expressions and voice, and analyzes the satisfaction and difficulties during the cooking process in real time.

[1561] Step 14:

[1562] The emotion engine sends the analysis results to the server, which then transmits the emotion data to the server, which then reflects it in the cooking process.

[1563] Step 15:

[1564] The user provides feedback on the cooking results, sending their evaluation of the cooking results and taste to the server via their device.

[1565] Step 16:

[1566] The server receives and analyzes the feedback sent by users and uses it to improve the AI ​​model.

[1567] Step 17:

[1568] The server improves the AI ​​model, adjusting it based on feedback and emotion data to improve the accuracy of the next replay.

[1569] This process allows users to accurately learn and recreate traditional cooking techniques, and also provides an optimal cooking environment for users by taking into account their emotions during the cooking process.

[1570] Example 2

[1571] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1572] Conventional cooking skill learning systems have difficulty effectively replicating professional cooking techniques, and have been unable to properly utilize feedback to ensure users get emotionally satisfying results. Furthermore, there has been no means to analyze users' emotions in real time and reflect them in the cooking process.

[1573] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for a user to record the cooking process, a means for uploading the recorded cooking process data to the server, a means for the server to convert the received data into an analyzable format, a means for the server to analyze the video data and extract each process, a means for the server to analyze the audio data and extract instructions and comments, a means for the server to train an AI model based on the analysis results, a means for a user to send a reproduction request, a means for the server to generate specific cooking steps and recipes and provide them to the terminal, a means for the user to provide feedback on the cooking results, a means for the server to analyze the feedback and improve the AI ​​model, and an emotion analysis means, which recognizes the user's emotions in real time and transmits the analysis results to the server. This makes it possible to provide optimal cooking instructions that take the user's emotions into consideration, solving problems associated with conventional systems and enabling users to learn and reproduce cooking techniques with greater accuracy.

[1574] A "user" is an individual or entity who uses the system to learn and replicate cooking techniques.

[1575] "Recorded data" refers to video and audio data used by a user to record the cooking process.

[1576] The "server" is a central computer system that receives the recorded data, analyzes it, trains the AI ​​model, and provides the results to the user.

[1577] "Data analysis" refers to the process in which the server identifies the video and audio data it receives and extracts the cooking process, instructions, and comments.

[1578] An "AI model" is an artificial intelligence algorithm or system that learns based on analysis results and provides users with optimal cooking procedures and recipes.

[1579] A "reproduction request" is an act in which a user sends a request to the server that they would like to learn or reproduce a particular dish or technique.

[1580] "Emotion analysis" refers to the emotion analysis means recognizing the user's emotions in real time and sending the analysis results to the server.

[1581] "Feedback" includes the user reporting the results of their cooking reproduction to the system and the results of subsequent sentiment analysis.

[1582] "Cooking instructions" refers to the specific cooking methods and steps generated by the server.

[1583] A "recipe" is a set of detailed instructions detailing the ingredients and quantities needed to prepare a particular dish, as well as the cooking method involved.

[1584] "Emotion analysis means" refers to technology or devices for detecting and analyzing a user's emotions in real time, and uses facial recognition technology and voice analysis technology.

[1585] MODE FOR CARRYING OUT THE INVENTION

[1586] The system of the present invention allows users to record the cooking process, analyze the data, learn cooking techniques, and then reproduce the recipes. Furthermore, by combining it with emotion analysis means that recognizes the user's emotions, the system aims to increase user satisfaction.

[1587] This system consists of a user recording means, a data uploading means, a server data analysis means, an AI learning means, a reproduction request means, a feedback means, and an emotion analysis means.

[1588] Acquiring Recording Data

[1589] Users record the cooking process using a recording device such as a smartphone or video camera. This process includes everything from preparation to completion, and detailed footage is required. For example, when making "seared bonito," users will record in detail how they cut the bonito, how well it is cooked, and how they sear it.

[1590] Uploading data

[1591] Once the recording is complete, the user imports the recording data to their device and then uploads it to the server using a dedicated application. This process can be completed with just a few clicks.

[1592] Data preprocessing and analysis

[1593] The server converts the received video data into an analyzable format. During this process, the video data and audio data are separated and processed separately. The video data is analyzed frame by frame to extract information such as the type of ingredients, the use of cooking utensils, and the cooking process. For example, the way ingredients are cut and cooked can be identified. Meanwhile, the audio data is converted to text to extract the chef's instructions and comments.

[1594] AI Learning

[1595] The server then trains the AI ​​model with the analysis results. This training process includes detailed cooking techniques, including how to cut food, how much heat to use, and how to use seasonings, so that the AI ​​model can accurately reproduce detailed cooking techniques and flavors. This allows the AI ​​model to provide users with optimal cooking procedures and recipes.

[1596] Reproduction requests and submissions

[1597] When a user wants to recreate a specific dish or technique, they send a request to the server via their device. Based on the request, the server generates the optimal cooking steps and necessary recipes and provides them to the device. The user can then recreate the dish based on this information. An example of a specific prompt sentence is, "Please tell me the optimal cooking steps and recipe for making bonito tataki. Please also provide voice instructions."

[1598] Feedback and model improvement

[1599] After the user has finished recreating the dish, they provide feedback to the emotion analysis means via their device. The emotion analysis means analyzes the user's emotions and sends the results to the server. The server receives the feedback and uses it to improve the AI ​​model. For example, the emotion analysis means can analyze the user's satisfaction and difficulties and reflect that information in the AI ​​model.

[1600] Operation of emotion analysis means

[1601] The emotion analysis means uses facial recognition and voice analysis technologies to detect the user's emotions in real time and transmits the data to the server, which enables the system to provide optimal cooking instructions and recipes that take the user's emotions into consideration.

[1602] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1603] Step 1: Obtaining the recording data

[1604] Users can use their smartphones or video cameras to record the cooking process in detail, for example, the entire cooking process of "seared bonito."

[1605] Input: Video and audio data stored on the recording device.

[1606] How it works: The user starts recording and then imports the recording to their device after shooting.

[1607] Output: Recording data file saved on the device.

[1608] Step 2: Upload your data

[1609] The device uploads the recorded data to the server using a dedicated application.

[1610] Input: Recording data files saved on the device.

[1611] How it works: The user launches the dedicated app and clicks the "upload" button to send data to the server.

[1612] Output: Recording data file saved on the server.

[1613] Step 3: Data preprocessing and analysis

[1614] The server converts the recorded data into an analyzable format, separating the video and audio data and processing each separately.

[1615] Input: Recording data files stored on the server.

[1616] How it works: The server separates the video data from the audio data and analyzes the video data frame by frame. Specifically, it extracts information such as the type of ingredients, how to use cooking utensils, and the cooking process. Meanwhile, the audio data is converted into text and the chef's instructions and comments are extracted.

[1617] Output: Analyzable video data and audio data in text format.

[1618] Step 4: AI training

[1619] The data analyzed by the server is trained into an AI model.

[1620] Input: Analyzable video data and audio data in text format.

[1621] How it works: The server inputs the data into the AI ​​model and begins the learning process, incorporating even the smallest details like cutting techniques, heat levels, and seasonings.

[1622] Output: The trained AI model.

[1623] Step 5: Request and provide a reproduction

[1624] When a user wants to recreate a particular dish or technique, they send a request to the server through their device.

[1625] Input: The reproduction request submitted by the user.

[1626] How it works: The server generates the optimal cooking instructions and recipe based on the request and sends it to the device. For example, if the request is "I want to recreate seared bonito," the server will provide the instructions and recipe for that.

[1627] Output: Specific cooking instructions and recipe displayed on the user's device.

[1628] Step 6: Feedback and model improvement

[1629] After the user has completed recreating the dish, they provide feedback to the sentiment analysis means.

[1630] Input: User-provided feedback data.

[1631] Operation: The emotion analysis means analyzes the feedback data and sends it to the server. For example, the satisfaction or difficulties felt during the cooking process are specifically input.

[1632] Output: Feedback analysis data sent to the server.

[1633] Step 7: Sentiment analysis method in action

[1634] The emotion analysis means recognizes the user's emotions in real time and transmits the analysis results to the server.

[1635] Input: Real-time video and audio data of the user cooking.

[1636] How it works: The emotion analysis unit uses face recognition and voice analysis technology to detect the user's emotions and sends the data to the server. For example, it analyzes emotions such as joy or difficulty.

[1637] Output: Real-time emotion data sent to the server.

[1638] (Application example 2)

[1639] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1640] Conventional food delivery systems lacked the means to effectively manage and improve food quality and user satisfaction. In particular, there were many challenges in the reproducibility of cooking processes, the skill acquisition of delivery staff, and the management of customer feedback. The purpose of this invention is to solve these problems, improve the quality of delivery services, and increase the satisfaction of customers and delivery staff.

[1641] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to record the cooking process and upload the data, means for analyzing the uploaded recorded data and extracting each process from the video data, means for extracting instructions and comments from the audio data, means for training the generative AI model based on the analysis results, means for a user to send a reproduction request, generate specific cooking steps and recipes, and provide them to the terminal, means for a user to provide feedback on the cooking results and for the server to analyze the feedback and improve the generative AI model, means for a delivery staff member to record the cooking process, means for uploading the recorded data to a cloud server, means for the cloud server to analyze the uploaded recorded data and extract cooking skills, and means for analyzing the emotions of the delivery staff member and sending feedback to the server. This improves the reproducibility of the cooking process, enabling delivery staff to acquire skills and customer feedback to be managed more effectively.

[1642] "Means for users to record the cooking process" refers to devices or methods that allow users to record the cooking process as video.

[1643] The "means for uploading recorded data of the cooking process to the server" is a means for transmitting video data of the cooking process recorded by the user to the server via the Internet.

[1644] "Means for converting data received by the server into an analyzable format" refers to a method or algorithm for converting data uploaded to the server into a format that can be easily analyzed.

[1645] "Means for the server to analyze video data and extract each process" refers to a method or system for analyzing the content of video data and identifying and extracting each step of food preparation and cooking.

[1646] "Means for the server to analyze the audio data and extract instructions and comments" refers to methods and technologies for identifying cooking instructions and comments from the recorded audio data and extracting them as text.

[1647] "Means for the server to train the generated AI model on the analysis results" refers to a method for training the AI ​​model generated based on the analysis results extracted from video data and audio data on new data.

[1648] The "means by which a user sends a reproduction request" refers to an interface or method by which a user can send a request to the server when they want to reproduce a particular dish or technique.

[1649] "Means for the server to generate specific cooking procedures and recipes and provide them to the terminal" refers to a system that generates optimal cooking procedures and recipes based on the user's reproduction request and provides them to the user's terminal.

[1650] "Means for users to provide feedback on cooking results" refers to methods or tools that allow users to return the results of their recreated dishes to the server as their impressions or evaluations.

[1651] "Means for the server to analyze feedback and improve the generative AI model" refers to a system or method for analyzing feedback from users and improving the generative AI model based on the results.

[1652] "Means for delivery staff to record the cooking process" refers to devices or methods that delivery staff use to record the cooking process of food on video.

[1653] "Means for uploading recorded data to a cloud server" refers to means for saving recorded data to a cloud server via the Internet.

[1654] "Means for the cloud server to analyze the uploaded video data and extract cooking techniques" refers to a method or system for the cloud server to analyze the video data received and identify and extract cooking techniques and processes.

[1655] "Means for analyzing the emotions of delivery staff and sending feedback to a server" refers to a method or technology for analyzing the emotions of delivery staff from video and audio data and sending the results as feedback to a server.

[1656] This invention aims to develop a system that aims to improve customer satisfaction and control the quality of the cooking process in food delivery services. The system consists of the following components:

[1657] 1. User recording method: The user or delivery staff uses a recording device such as a smartphone to record the cooking process. The video data contains detailed information such as the cooking steps, ingredients used, and necessary cooking utensils.

[1658] 2. Data uploading method: Users use a dedicated application to upload recorded data to the cloud server, which allows data to be easily sent to the server via the Internet.

[1659] 3. Server data analysis method: The server converts the received video data into an analyzable format. Specifically, the video data is divided into frames and image recognition technology (e.g., OpenCV) is used to analyze the type of ingredients and cooking process. Additionally, audio data is converted into text using speech recognition technology (e.g., Google Speech-to-Text) to extract cooking comments and instructions.

[1660] 4. AI learning method: The generative AI model learns cooking techniques based on the analyzed data. This model is built using a machine learning library (e.g., TensorFlow). The model reflects detailed cooking techniques, such as how to cut ingredients, how to control heat, and how to use seasonings.

[1661] 5. Reproduction request method: When a user wants to reproduce a specific dish, they send a request to the server through a dedicated application. Based on this request, the server generates the optimal cooking procedure and recipe and provides it to the user's device.

[1662] 6. Feedback method: The user provides feedback on the cooking results to the server through the application. The feedback is analyzed using facial recognition technology (e.g., DeepFace) to quantify the user's satisfaction.

[1663] 7. Emotion Engine: The emotion engine recognizes the user's emotions in real time and transmits the user's joy, difficulty, and satisfaction to the server, which analyzes this data and improves the generative AI model.

[1664] The above system will improve the reproducibility of the cooking process, enable delivery staff to acquire skills, and enable effective management of customer feedback. It will also optimize the recipes and cooking procedures provided to users, which is expected to improve customer satisfaction.

[1665] Examples:

[1666] For example, a delivery staff member serves a highly rated dish to a customer and records the cooking process. The video data is uploaded to a cloud server, which analyzes the data and learns the cooking process and techniques. If another user requests to make the same dish, the server will provide the optimal cooking procedure and recipe, and the user will provide feedback on the results, and the emotion engine will analyze the customer's satisfaction level.

[1667] Example of an input prompt:

[1668] "I want to analyze videos of the cooking process taken by delivery staff and collect emotional feedback. Please generate a report that includes the analysis results and emotional feedback."

[1669] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1670] Step 1:

[1671] The user records the cooking process using a means to record the cooking process. Specifically, the user or delivery staff uses a smartphone or video camera to record the cooking process from start to finish. The input is video data of the cooking process, and the output is the recorded video file.

[1672] Step 2:

[1673] The recorded cooking process data is uploaded to the server. The user uses a dedicated upload application to send the recorded data to the cloud server. The input is a video file, and the output is video data on the cloud server.

[1674] Step 3:

[1675] The server converts the received data into an analyzable format. The cloud server divides the received video file into frames and separates it into image data and audio data. It also converts the audio data into text. The input is video data on the cloud server, and the output is image data and text data.

[1676] Step 4:

[1677] The server analyzes the video data and extracts each process. Using machine learning and image recognition technology (e.g., OpenCV), the server identifies the type of ingredients, cooking utensils, and cooking process from the image data for each frame. The input is the image data for each frame, and the output is the extracted cooking process information.

[1678] Step 5:

[1679] The server analyzes the voice data and extracts instructions and comments. The server uses voice recognition technology (e.g., Google Speech-to-Text) to convert the voice data into text and identify cooking instructions and comments. The input is the voice data, and the output is the extracted text data.

[1680] Step 6:

[1681] The server trains the generative AI model with the analysis results. The server inputs the analyzed cooking process information and text data into the generative AI model (e.g., TensorFlow) and trains the model to learn new cooking techniques. The input is the cooking process information and text data, and the output is an updated generative AI model.

[1682] Step 7:

[1683] The system uses a means for users to send a reproduction request. A user sends a request to reproduce a specific dish or cooking technique through a dedicated application. The input is the reproduction request, and the output is the request data sent to the server.

[1684] Step 8:

[1685] The server generates specific cooking steps and recipes and provides them to the device. The server generates optimal cooking steps and recipes based on the generative AI model and provides this information to the user's device. The input is the generative AI model and the user's reproduction request, and the output is the cooking steps and recipe provided to the user's device.

[1686] Step 9:

[1687] The system uses a means for users to provide feedback on their cooking results. Users cook and provide feedback on the results through a dedicated application. The input is the user's cooking results, and the output is feedback data to the server.

[1688] Step 10:

[1689] The server analyzes the feedback and improves the generative AI model. The server analyzes the user feedback data, evaluates customer satisfaction using an emotion engine, and reflects the results in the generative AI model. The input is the feedback data, and the output is an improved generative AI model.

[1690] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1691] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1692] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1693] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1694] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1695] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1696] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1697] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1698] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1699] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1700] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1701] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1702] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1703] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1704] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1705] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1706] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1707] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1708] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1709] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1710] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1711] The following is further disclosed regarding the above embodiment.

[1712] (Claim 1)

[1713] a means for a user to record the cooking process;

[1714] A means for uploading the recorded cooking process data to a server;

[1715] a means for converting the data received by the server into a parsable format;

[1716] A means for the server to analyze the video data and extract each process;

[1717] A means for the server to analyze the voice data and extract instructions and comments;

[1718] The server uses the analysis results to train the AI ​​model,

[1719] a means for a user to submit a reproduction request;

[1720] A means for the server to generate specific cooking steps and recipes and provide them to the terminal;

[1721] A means for the user to provide feedback on the cooking results;

[1722] A means for the server to analyze the feedback and improve the AI ​​model;

[1723] A system including:

[1724] (Claim 2)

[1725] 10. The system of claim 1, further comprising means for analyzing the video data frame by frame.

[1726] (Claim 3)

[1727] 10. The system of claim 1, further comprising means for converting voice data into text.

[1728] "Example 1"

[1729] (Claim 1)

[1730] a means for a user to record the cooking process;

[1731] means for uploading the recorded cooking process data to an information processing device;

[1732] means for converting data received by the information processing device into an analyzable format;

[1733] A means for an information processing device to analyze video data and extract each process;

[1734] A means for an information processing device to analyze the voice data and extract instructions and comments;

[1735] A means for the information processing device to train a learning model based on the analysis results;

[1736] a means for a user to submit a reproduction request;

[1737] A means for an information processing device to generate specific cooking steps and recipes and provide them to a terminal;

[1738] A means for the user to provide feedback on the cooking results;

[1739] means for the information processing device to analyze the feedback and improve the learning model;

[1740] A system including:

[1741] (Claim 2)

[1742] 10. The system of claim 1, further comprising means for analyzing the video data frame by frame.

[1743] (Claim 3)

[1744] 10. The system of claim 1, further comprising means for converting voice data into text.

[1745] "Application Example 1"

[1746] (Claim 1)

[1747] a means for a user to record the cooking process;

[1748] A means for uploading the recorded cooking process data to a server;

[1749] a means for converting the data received by the server into a parsable format;

[1750] A means for the server to analyze the video data and extract each process;

[1751] A means for the server to analyze the voice data and extract instructions and comments;

[1752] The server uses the analysis results to train the AI ​​model,

[1753] a means for a user to submit a reproduction request;

[1754] A means for the server to generate specific cooking steps and recipes and provide them to the terminal;

[1755] A means for the user to provide feedback on the cooking results;

[1756] A means for the server to analyze the feedback and improve the AI ​​model;

[1757] A means for chefs and staff to learn new techniques and recipes in-store; 【...

Claims

1. a means for a user to record the cooking process; A means for uploading the recorded cooking process data to a server; a means for converting the data received by the server into a parsable format; A means for the server to analyze the video data and extract each process; A means for the server to analyze the voice data and extract instructions and comments; The server uses the analysis results to train the AI ​​model, a means for a user to submit a reproduction request; A means for the server to generate specific cooking steps and recipes and provide them to the terminal; A means for the user to provide feedback on the cooking results; A means for the server to analyze the feedback and improve the AI ​​model; A system including:

2. 10. The system of claim 1, further comprising means for analyzing the video data frame by frame.

3. 2. The system of claim 1, further comprising means for converting voice data into text.

Images