Information processing device, information processing method, and program

Abstract

An information processing device (IP) has a processor (PR). The processor (PR) links an utterance of a recorded user during cooking to sensor data measured at the time of the utterance, and records the same in cooking data.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Techniques for sensing, recording, and reproducing cooking have been proposed. For example, in Patent Document 1, a system for recording information related to cooking (types of ingredients, amounts used, cooking timing, cooking time, movement trajectories of cooking utensils, heat adjustment timing, heat duration) using various sensors (camera, electronic scale, pot temperature sensor, angle sensor of stove switch, infrared sensor, piezoelectric sensor) has been proposed. Techniques for generating and reproducing the movement of a robot from such information have also been proposed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, these techniques mainly assume that the reproducer is a robot. Therefore, while reproduction by a robot in a controlled cooking environment with the same measuring instruments is possible, there is a problem that reproduction by a person, such as in various restaurants or homes, is difficult.

[0005] Therefore, the present disclosure proposes an information processing apparatus, an information processing method, and a program capable of intuitively reproducing the recorded cooking.

Means for Solving the Problems

[0006] According to this disclosure, an information processing device is provided, which has a processor that records statements made by a recording user during cooking, linking them with sensor data measured at the time of the statement, and records them in cooking data. Furthermore, according to this disclosure, an information processing method is provided in which the information processing of the information processing device is performed by a computer, and a program is provided in which the computer implements the information processing of the information processing device. [Brief explanation of the drawing]

[0007] [Figure 1] This diagram shows an overview of the cooking data recording and reproduction system. [Figure 2] This is a functional block diagram of the cooking data recording and reproduction system. [Figure 3] This diagram shows the processing flow in recording mode. [Figure 4] This figure shows examples of subjective information contained in the statements of recording users. [Figure 5] This diagram illustrates the tacit knowledge of recording users. [Figure 6] This is an explanatory diagram for recognizing the cooking process. [Figure 7] This figure shows examples of statements made during recording regarding the start and end of the cooking process. [Figure 8] This figure shows an example of the hardware configuration of an information processing device. [Modes for carrying out the invention]

[0008] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0009] The explanation will proceed in the following order. [1. Configuration of the Cooking Data Recording and Reproduction System] [2. Processing Flow] [2-1. Recording Mode] [2-2. Playback Mode] [3. Examples of subjective information contained in the statements of recording users] [4. Learning tacit knowledge] [5. Cooking process recognition] [6. Hardware Configuration Examples] [7. Effects]

[0010] [1. Configuration of the Cooking Data Recording and Reproduction System] Figure 1 shows an overview of the cooking data recording and reproduction system CS. Figure 2 is a functional block diagram of the cooking data recording and reproduction system CS.

[0011] The Cooking Data Recording and Reproduction System CS is a type of smart kitchen that supports cooking operations through the coordination of cooking equipment with built-in sensors and information terminals. For example, the Cooking Data Recording and Reproduction System CS supports cooking using heating appliances KW. In the example in Figure 1, an IH (Induction Heating) heater HT and a pot PT are used as heating appliances KW.

[0012] The cooking data recording and reproduction system CS generates cooking data CDs for each dish using sensor data acquired from the sensor device SE. The cooking data CD is data that shows the cooking profile performed by the cook US from the start to the end of the dish. The cooking data CD includes, for example, time-series information (TTW information) regarding the heating temperature, heating time, and weight of the food being cooked CO. Information regarding the heat output of the cooking appliance KW is also included in the cooking data CD in relation to the heating temperature. The cooking data CD can also include video (cooking video) of the cooking process captured by the camera sensor CM.

[0013] The web server SV stores numerous recorded cooking data CDs. The cook at the time of recording can be the cook US themselves or another cook. The cooking data recording and reproduction system CS obtains the recorded cooking data CDs from the web server SV as reference data RF. Cook US performs cooking while referring to the reference data RF. This allows the recorded dishes to be reproduced.

[0014] Hereinafter, a cook US who uses the cooking data recording and reproduction system CS for the purpose of recording their own cooking is referred to as a recording user. Also, a cook US who uses the cooking data recording and reproduction system CS for the purpose of reproducing the cooking of the recording user is referred to as a reproduction user.

[0015] The cooking data recording and reproduction system CS generates reproduction data RP based on the cooking profile of the food being cooked during cooking. The cooking profile means time-series information representing the operation history on the cooking object CO during the cooking process and the state change of the cooking object CO. The reproduction data RP is cooking data CD generated using the sensor data obtained during reproduction. The cooking data recording and reproduction system CS monitors the cooking of the reproduction user and continues to present information regarding the heating temperature, heating timing, etc. to the reproduction user based on the reference data RF. Thereby, an appropriate cooking operation according to the cooking situation is prompted to the reproduction user.

[0016] The cooking data recording and reproduction system CS has, for example, an IH heater HT, a pot PT, a display DP, a speaker SP, a gaze sensor SC, a camera sensor CM, a microphone MC, lighting LT, a web server SV, a network router RT, an input UI device IND, and an information processing device IP.

[0017] The camera sensor CM captures an image of the cooking object CO in the pot PT. The gaze sensor SC detects the gaze of the cook US. The microphone MC detects the sound during cooking. The detected sound includes the sound generated by cooking and the voice of the cook US. The microphone MC is not limited to a stationary microphone and may be a microphone built into a portable device such as an earphone or a headset. The microphone MC may include all devices capable of voice input.

[0018] The display DP presents various types of information for cooking support (support information) by video and voice. The input UI (User Interface) device IND supplies the user input information input by the cook US to the information processing device IP.

[0019] The Input UI Device IND includes all user interface elements with the Cook (US) that correspond to input devices. Examples of Input UI Device INDs include information terminals such as smartphones and touch panels built into displays (DP). Furthermore, voice input via microphone (MC) and gesture input via camera sensor (CM) can also be included in the input information using the Input UI Device IND. Therefore, microphones (MC) and camera sensors (CM) function as Input UI Device INDs.

[0020] Output devices (OTDs) include all output UI devices that present information to the cook (US), such as displays (DPs) and speakers (SPs). Examples of displays (DPs) include stationary displays such as LCD monitors and projectors, as well as wearable displays such as AR (Augmented Reality) glasses. Output devices (OTDs) may also include information presentation devices other than visual and auditory ones, such as haptic devices.

[0021] The information processing device IP controls the entire cooking data recording and reproduction system CS based on various information detected by the cooking data recording and reproduction system CS. The information processing device IP includes, for example, a processor PR, a temperature sensor TS, a weight sensor WS, and a storage ST. All components of the information processing device IP may be built into the IH heater HT, or some or all of the information processing device IP may be mounted on an external device that communicates with the IH heater HT.

[0022] The temperature sensor TS measures the temperature of the pot PT or the food being cooked CO inside the pot PT. The weight sensor WS measures the weight of the food being cooked CO inside the pot PT. For example, a radiation thermometer that can measure quickly and without contact can be used as the temperature sensor TS. A load cell can be used as the weight sensor WS. In the example in Figure 1, the temperature sensor TS is built into the IH heater HT, which is the heating control unit. However, the temperature sensor TS may also be built into the pot PT, which is the heating medium.

[0023] The temperature sensor TS and weight sensor WS, along with the camera sensor CM, gaze sensor SC, and microphone MC, constitute a sensor device SE that detects various information within the cooking data recording and reproduction system CS. The state of the object being cooked CO is detected based on the sensor data acquired by the sensor device SE. In addition to the sensors mentioned above, the sensor device SE may include other sensors capable of detecting the state of the object being cooked CO, such as odor sensors, taste sensors, moisture content sensors, and elasticity meters. The sensor data acquired by the sensor device SE is supplied to the information processing device IP in real time.

[0024] Processor PR generates cooking data CD based on sensor data acquired from sensor device SE. Processor PR records the statements made by the recording user during cooking on the cooking data CD, linking them with the sensor data measured at the time of the statement. "Time of statement" refers to the start time of the statement, or a predetermined time period centered around the start time of the statement (around the start time of the statement).

[0025] For example, the recording user's statements include subjective information perceived by the recording user. Examples of subjective information include cooking information such as the state and movement of the cooking object CO and cooking utensils. The processor PR records sensor data that quantitatively represents the subjective information, linked to the recording user's statements, onto the cooking data CD.

[0026] The processor PR presents information related to the recording user's statements to the user reproducing the cooking data CD at the moment the same cooking conditions as those at the time of the recording user's statements are detected. This information is presented, for example, by playing back the recording user's statements as they were recorded at the time of recording. However, the method of presenting the information is not limited to this. The content of the recording user's statements may be displayed as text information on the display DP, or processed video and audio of the recording user's statements may be presented via the display DP and speaker SP.

[0027] If the object being spoken about is not identifiable from the content of the statement, the object can be identified based on the recording user's gaze information. For example, the processor PR identifies the object being spoken about by the recording user based on the recording user's gaze detected by the gaze sensor SC.

[0028] This system assumes that the recording user will be speaking about the cooking process as they cook. However, there may be cases where the recording user switches cooking steps without speaking, or where the object being referred to is unclear from the spoken content (textual information) alone. For example, the user may slightly reduce the heat without saying anything, or the textual information may include demonstrative pronouns such as "when this gets bigger" or "when this temperature reaches 80°C." In such cases, by using the gaze sensor SC to measure where the recording user is looking, it is possible to infer the sensory information the recording user used to make decisions from their gaze.

[0029] As eye-tracking sensors (SCs), wearable devices such as AR glasses and eye trackers (manufactured by Tobii Technology Co., Ltd.) can be used. However, the methods for detecting eye movements are not limited to these. For example, methods such as eye-attached tracking, which involves attaching special contact lenses to the eyes and measuring their movement; optical tracking, which measures using optical technology without directly touching the eyes with an object; and electric potential measurement, which involves placing electrodes around the eyes to measure the electrical potential generated by the muscles that move the eyeballs, can also be employed.

[0030] The processor PR adjusts the lighting LT to control the shooting conditions of the camera sensor CM. The lighting LT is part of the linked equipment LD that controls the cooking environment. The linked equipment LD may include other equipment such as air conditioning units. The processor PR controls the operation of the linked equipment LD based on sensor data.

[0031] The processor PR communicates with external devices and web servers such as SV via the network router RT. Based on external information, sensor data, and user input information acquired through communication, the processor PR controls the operation of each device within the cooking data recording and reproduction system CS.

[0032] [2. Processing Flow] The information processing of the cooking data recording and reproduction system CS can be broadly divided into two types. One is the process of recording the cooking performed by the recording user (recording mode). The other is the process of supporting the reproduction user's cooking based on the recorded data (reproduction mode). The processing flow of each mode is explained below.

[0033] [2-1. Recording Mode] Figure 3 shows the processing flow in recording mode.

[0034] <Step S1: Recipe Setup> The recording user sets the recipe information, which will be the cooking conditions, via the input UI device IND. For example, the recipe information may include some or all of the following:

[0035] (i) Type and name of the dish (ii) The type, quantity, cutting method, and mixing method of the ingredients used (iii) Cooking utensils to use (iv) Cooking process (timing and quantity of ingredients added, heat setting, cooking time) (v) Finished quantity

[0036] For example, when a recording user selects their preferred recipe from a menu in an app displayed on the input UI device IND, the recipe information is automatically set. Recording users can also fine-tune individual information within the app.

[0037] <Step S2: Start cooking> The processor PR detects the start of cooking based on some trigger event caused by the recording user's operation. When the start of cooking is detected, the processor PR performs initialization processing. The initialization processing is linked to the recipe information and registered with the web server SV. In the subsequent flow, the process from sensor data input (step S3) to data recording (step S7) becomes a processing loop that is executed repeatedly in real time. The timing of processing execution and the method of parallelization will differ depending on the specific processing system.

[0038] <Step S3: Sensor data input> The processor PR receives sensor data from the sensor device SE, which detects the cooking status. Generally, the frame rates of each sensor differ and are not necessarily synchronized. All received data is stored with a timestamp based on a clock device managed by the processor PR. While the specific use of the clock device is not limited, the timestamps of all sensor data are managed in a way that allows them to be compared with each other.

[0039] <Step S4: UI Data Entry> The recording user detects cooking timing and other details using their five senses. The recording user verbally describes the sensory cooking information detected by their five senses. The processor PR receives the recording user's statements as user input information via the microphone MC.

[0040] <Step S5: Recognition of the cooking process> The processor PR recognizes the current cooking process based on recipe information and previous input data (sensor data, user input information). In other words, the processor PR identifies which stage of the cooking process is currently being executed. Real-time execution of this process is optional in recording mode; offline processing is also possible by analyzing the recorded cooking data CD after cooking is complete.

[0041] <Step S6: Information Presentation> The processor PR generates support information based on recipe information and sensor data. The processor PR presents the generated support information to the recording user via the output device OTD.

[0042] <Step S7: Data Recording> The processor PR records all the data required in playback mode in the appropriate format and generates a cooking data CD. While a non-volatile storage device ST is typically assumed as the recording destination, it is not limited to this. Any storage device accessible by the processor PR can be used as the recording destination. It is not necessarily limited to recording; streaming data to an external device is also envisioned.

[0043] <Step S8: Cooking complete> The processor PR detects the end of cooking based on a trigger event caused by the recording user's actions. When the processor PR detects the end of cooking, it performs a termination process. This termination process is linked to the recipe information and registered with the web server SV.

[0044] [2-2. Playback Mode] The following explains the contents of each process in playback mode, focusing on the differences in operation compared to recording mode. Steps S2, S3, S4, S7, and S8 are the same as in recording mode, so their explanation is omitted.

[0045] <Step S1: Recipe Setup> The user attempting to recreate the dish specifies a reference, recorded cooking data CD (reference data RF) via the input UI device IND. The reference data RF contains the cooking process that the user should recreate. The user then performs the cooking according to the instructions in the reference data RF. This recreates the dish prepared by the original user.

[0046] The processor PR reads the specified reference data RF. The processor PR sets the recipe information contained in the reference data RF as cooking conditions. The user can start cooking with the same conditions as before, or they can change the conditions via the input UI device IND, similar to the recording mode.

[0047] <Step S5: Recognition of the cooking process> The processor PR recognizes the current cooking process based on recipe information and previous input data (sensor data, user input information). In playback mode, cooking is performed according to the cooking procedure recorded in the reference data RF, so it is important to identify which stage of the cooking process is being executed.

[0048] <Step S6: Information Presentation> The processor PR generates support information based on recipe information, reference data RF, and sensor data. The processor PR presents the generated support information to the reproduction user via the output device OTD.

[0049] [3. Examples of subjective information contained in the statements of recording users] Figure 4 shows an example of subjective information contained in the statements of recording users.

[0050] In recording mode, the recording user cooks while verbally describing the cooking process. The processor PR records the recording user's statements, linking them to quantitative data (sensor data indicating temperature, time, weight, color, etc.) that shows the cooking conditions at the time of the statements. In playback mode, the processor PR determines, based on the quantitative data detected by the sensor device SE, whether the same cooking conditions as when the recording user spoke occurred. If the cooking conditions at the time of the statements occurred, the processor PR presents the recorded statements to the playback user via the output device OTD.

[0051] For example, if the recording user says, "The onions have turned amber, so I'm turning off the heat," and then turns off the heat, the processor PR records the "color of the onions" (RGB data) at the time of the statement using the camera sensor CM. Then, during playback, the processor PR senses the color of the onions being cooked in real time using the camera sensor CM, and when it matches the recorded "color of the onions" at the time of the statement, it notifies the playback user via voice or video, "The onions have turned amber, so I'm turning off the heat." This allows the recording user to record their cooking with greater accuracy, and the playback user can intuitively cook by following the recording user's voice instructions.

[0052] In the example above, the information that the onion turned amber is subjective information perceived by the recording user, while the RGB data indicating amber color is sensor data (quantitative data) that quantitatively represents the subjective information.

[0053] Examples of cooking information to be recorded are not limited to those mentioned above. For example, subjective information may include "when you hear a crackling sound," "when the pan is heated up," "when the bubbles get bigger," and "when you move the pan in a circular motion." This information is linked to quantitative data related to sound, temperature, size, and movement, respectively. For example, sound is detected by a microphone (MC). Temperature is detected by a thermographic camera or temperature sensor. Size is detected by a camera sensor (CM). Movement is detected by an accelerometer or infrared camera.

[0054] The spoken content in Figure 4 is the result of recognizing the recording user's voice input into the microphone MC and converting it into text information. The processor PR recognizes subjective information related to the cooking state, such as color, sound, temperature, and size, from the text information obtained through the conversion. The processor PR records the sensor data at the time of speaking, which quantitatively shows the recognized information, in conjunction with the spoken content and records it on the cooking data CD. This ensures that the recording user's sensory information is recorded at a more accurate timing.

[0055] Quantitative data is measured, for example, at the time the recording user makes a statement. However, quantitative data can also be measured and recorded continuously, and desired data can be extracted from the recorded time-series data and recorded on the cooking data CD. This is because statements may contain information about time intervals, such as "stir-fry over high heat for 2 minutes." For example, the temperature sensor at the bottom of the pot can be kept measuring continuously from the start to the end of cooking, and when a statement about temperature is made, data corresponding to the time of the statement or the content of the statement can be extracted and recorded on the cooking data CD along with the statement.

[0056] The system design allows for manual configuration of which sensor's quantitative data is assigned to subjective information spoken by the recording user. However, it is also possible to automate this assignment using methods such as machine learning. For example, if a comment is made about color, the system can be configured to always record both the color information from the camera sensor (CM) and the frying pan temperature, regardless of the recipe. The frying pan temperature is recorded because its temperature can cause color changes.

[0057] The recording user's statements can be linked not just to one quantitative data point, but to multiple quantitative data points. This is because the cooking conditions indicated by the statements may be defined by multiple parameters. For example, if the recording user makes a statement about sound, both the sound from the microphone (MC) and the temperature of the food being cooked (CO) can be recorded as quantitative data. The temperature is recorded because sound is likely to be produced by high temperatures. It is also possible to configure the system to record both the thermographic camera and the pot bottom temperature sensor as quantitative data if the recording user makes a statement about temperature.

[0058] Furthermore, if the quantitative data indicated by the statement cannot be directly measured by the sensors included in the system, the measurement data from one or more sensors related to the statement may be recorded as quantitative data. For information that is difficult to measure directly, such as smell or taste, sensor data closely related to the smell or taste will be recorded as quantitative data. For example, if there is a statement about smell, the system may be configured to record heating time, temperature, and weight as quantitative data. This allows for the recording of sensory information that cannot be directly measured by the sensors included in the system, based on multiple data points.

[0059] [4. Learning tacit knowledge] Figure 5 illustrates the tacit knowledge of the recording user.

[0060] Subjective elements such as color and sound may be perceived subtly differently by each recording user. For example, the upper part of Figure 5 shows the state of onion cooking that recording user A perceives as "burnt." The lower part of Figure 5 shows the state of onion cooking that recording user B perceives as "burnt." The color that recording user A perceives as "burnt" is darker than the color that recording user B perceives as "burnt." Recording user B perceives "burnt" when the cooked food CO is sautéed to a brown color, whereas recording user A does not perceive "burnt" until it is sautéed to a fairly black color.

[0061] Recording users A and B each have tacit knowledge that they implicitly understand. In the example in Figure 5, each recording user has an implicit understanding of "burnt color." However, the tacit knowledge of one recording user is not necessarily the same for other recording users. Therefore, the processor PR links objective measurement data (quantitative data) to subjective information that indicates tacit knowledge. However, even without such linking, if the processor learns the tacit knowledge of each recording user, it can directly infer quantitative data from what the recording users say.

[0062] For example, the relationship between tacit knowledge and quantitative data is accumulated as training data, and the analytical model is trained using machine learning. The processor PR uses the analytical model, which has learned the sensations of the recording user, to estimate the sensor data indicated by subjective information. The processor PR records the estimated sensor data on the cooking data CD, linking it to the recording user's statements. This makes it possible to quantitatively represent tacit knowledge.

[0063] In the example in Figure 5, the color information of the onions from the camera footage at the point when recording user A says, "I'll turn off the heat when it starts to turn a burnt color," is accumulated and used to train the analysis model. This allows the model to determine the color of the onions that recording user A refers to as "burnt." The learning of tacit knowledge from recording user B can be done in a similar manner. In this way, by representing each recording user's perception as quantitative data, it is possible to generate quantitative recipes solely from the content of the recording users' statements (textual information).

[0064] In the example above, one quantitative data point, color information from the camera image, is linked to the tacit knowledge of "burnt color." However, the number of quantitative data points that can be linked to tacit knowledge is not limited to one. Smells and tastes are complex sensations, and by learning from multiple quantitative data points as training data, it may be possible to explain the smells and tastes that the recording user perceives. Therefore, tacit knowledge representing complex sensations can be learned by linking it to multiple quantitative data points. This allows machine learning to infer the recording user's judgments, even for sensory information that cannot be directly measured by individual sensors.

[0065] [5. Cooking process recognition] Figure 6 is an explanatory diagram for recognizing the cooking process.

[0066] Cooking process recognition is the process of recognizing the start and end timings of each step in the cooking process for each sample, in order to compare data on the time axis between different samples. Here, "different samples" refer to time-series data sets recorded from different cooking methods based on the same recipe, typically meaning reference data RF and re-recorded data RP.

[0067] Figure 6 shows the entire cooking process from start to finish. The horizontal axis of Figure 6 represents the time from the start of cooking. The time-series data includes, for example, a cooking video V captured by a camera sensor CM inside the pot PT, the heat output of the induction heater HT, the temperature T measured by a temperature sensor TS, and the weight W measured by a weight sensor WS, all recorded in time series. The weight Ws represents data generated by removing external noise such as the addition of ingredients and stirring from the weight W.

[0068] The heating power of an IH heater (HT) is expressed in levels, for example, from 1 to 7. The higher the level, the higher the heating power setting. For example, "Level 7," "Level 5," and "Level 3" represent 200°C, 160°C, and 120°C, respectively. "OFF" indicates that heating has stopped.

[0069] In Figure 6, "#0" through "#10" each represent different cooking processes. The boundaries between cooking processes are characterized by the timing of operations such as those listed below. While these operations don't always define the boundaries of a cooking process, they often serve as effective criteria for division.

[0070] • Switching between different heat settings • Adding ingredients to the hot pot • Removing ingredients from the pot (including skimming off scum, etc.) • Starting / ending operations such as stirring ingredients or tossing the pan. • Removing the pot itself and the pot lid, etc.

[0071] This system assumes that the recording user will manually input the cooking process offline after cooking is complete. For example, the recording user will perform five cooking steps during recording, such as "cut, heat, add ingredient A, stir, add ingredient B, turn off the heat," and after cooking is complete, manually divide the cooking process into steps, such as "cut" from this time to this time, and "heat" from this time to this time. However, the task of manually setting the cooking process by the user can feel cumbersome.

[0072] Therefore, the recording user uses the microphone MC to make statements while cooking, indicating the content of the cooking process and the start / end of each cooking step. This allows the cooking process to be recorded automatically. For example, by making statements to indicate the start and end of cooking, such as "I'm going to mix now" and "I'll stop mixing here," the time from the start utterance to the end utterance can be recorded as a single cooking step. This not only allows for highly accurate recording of cooking step transitions but also reduces the hassle of manually setting cooking steps.

[0073] Figure 7 shows examples of statements made during recording regarding the start and end of the cooking process.

[0074] The Processor PR detects the timing of cooking process transitions (start and end times) based on the recorded user's statements that notify the user of the transition in the cooking process. The Processor PR records the detected information regarding the timing of cooking process transitions on the cooking data CD, linking it with the sensor data measured at the time of notification. When making statements regarding the start and end of a cooking process, a pre-set keyword is uttered. The Processor PR detects the timing when the recorded user utters the pre-set keyword as the timing of the cooking process transition. Figure 7 shows an example where the cooking process of chicken curry is divided using the keyword "Hi".

[0075] For example, the recording user's statement, "Okay, I'm turning on the fire," notifies the start of cooking process #0. The processor PR records the data of the heat setting at the time of the statement on the cooking data CD, linking it with the statement "Okay, I'm turning on the fire" and information indicating that it is the start of cooking process #0.

[0076] The recording user's statement, "Okay, I'll add the onions when the pan is hot," signals the start of cooking process #1 (the end of cooking process #0). Processor PR records the temperature sensor TS's measurement data at the time of the statement, linking it to the statement "Okay, I'll add the onions when the pan is hot" and information indicating the start of cooking process #1 (the end of cooking process #0), and records it on the cooking data CD.

[0077] The recording user announces the start of cooking process #2 (end of cooking process #1) by saying, "Okay, I'll turn off the heat when I hear a crackling sound." The processor PR records the measurement data from the microphone MC at the time of the announcement, linking it with the announcement "Okay, I'll turn off the heat when I hear a crackling sound" and information indicating the start of cooking process #2 (end of cooking process #1), onto the cooking data CD.

[0078] The recording user announces the start of cooking process #3 (end of cooking process #2) by saying, "Okay, I'll relight the fire when the sound stops." The processor PR records the measurement data from the microphone MC at the time of the announcement onto the cooking data CD, linking it with the announcement "Okay, I'll relight the fire when the sound stops" and information indicating the start of cooking process #3 (end of cooking process #2).

[0079] The recording user announces the start of cooking step #4 (end of cooking step #3) by saying, "Okay, when the onions are caramelized, add the chopped tomatoes and spices." The processor PR records the color data of the camera footage at the time of the announcement onto the cooking data CD, linking it with the announcement "Okay, when the onions are caramelized, add the chopped tomatoes and spices" and information indicating the start of cooking step #4 (end of cooking step #3).

[0080] The recording user announces the start of cooking step #5 (end of cooking step #4) by saying, "Okay, I'll increase the heat once the chopped tomatoes are crushed." The processor PR records the shape data of the camera image at the time of the announcement onto the cooking data CD, linking it with the announcement "Okay, I'll increase the heat once the chopped tomatoes are crushed" and information indicating the start of cooking step #5 (end of cooking step #4).

[0081] The recording user announces the start of cooking step #6 (end of cooking step #5) by saying, "Okay, when it becomes sticky, reduce the heat and add the water and chicken." The processor PR records the elasticity sensor measurement data at the time of the announcement, linking it with the announcement "Okay, when it becomes sticky, reduce the heat and add the water and chicken," and information indicating the start of cooking step #6 (end of cooking step #5), onto the cooking data CD.

[0082] The recording user's statement, "Okay, I'll put the lid on once I've added the ingredients," signals the start of cooking process #7 (the end of cooking process #6). The processor PR records the measurement data from the weight sensor WS at the time of the statement, linking it to the statement "Okay, I'll put the lid on once I've added the ingredients" and information indicating the start of cooking process #7 (the end of cooking process #6), and records it on the cooking data CD.

[0083] When the recording user says, "Okay, I'll remove the lid in 30 minutes," the start of cooking step #8 (the end of cooking step #7) is notified. Processor PR records the timer measurement data at the time of the statement, linking it with the statement "Okay, I'll remove the lid in 30 minutes" and information indicating the start of cooking step #8 (the end of cooking step #7), onto the cooking data CD.

[0084] The recording user announces the start of cooking step #9 (end of cooking step #8) by saying, "Okay, I'll increase the heat once the chicken is cooked through." The processor PR records the color data of the camera image at the time of the announcement onto the cooking data CD, linking it with the announcement "Okay, I'll increase the heat once the chicken is cooked through" and information indicating the start of cooking step #9 (end of cooking step #8).

[0085] The recording user announces the start of cooking process #10 (end of cooking process #9) by saying, "Okay, I'll turn off the heat when the bubbles get bigger." The processor PR records the shape data of the camera image at the time of the announcement onto the cooking data CD, linking it with the announcement "Okay, I'll turn off the heat when the bubbles get bigger" and information indicating the start of cooking process #10 (end of cooking process #9).

[0086] The recording user announces the end of cooking step #10 by saying, "Okay, it's done when the liquid is gone." The processor PR records the shape data from the camera image and the measurement data from the elasticity sensor at the time of the announcement onto the cooking data CD, linking it with the announcement "Okay, it's done when the liquid is gone" and information indicating that cooking step #10 has ended.

[0087] In the example in Figure 7, "yes" was used as the keyword, but the keyword is not limited to "yes." Any word such as "starting now" or "here" can be used as a keyword. Multiple keywords can also be used within a single cooking process without needing to differentiate between them. Furthermore, the declaration of the start / end of a cooking process using keywords can be replaced by the operation of physical buttons, etc. Even if a keyword is not recognized, the recording user's statements will always be recorded and can be manually corrected after cooking is complete.

[0088] The reproducing user recreates the recording user's dish using reference data RF, which includes information about the timing of cooking process transitions. The processor PR monitors the reproducing user's cooking status based on sensor data. When the processor PR detects a cooking status indicating a transition in the cooking process, it presents the reproducing user with information spoken by the recording user at the time of the transition.

[0089] The user can modify quantitative data indicating the timing of changes in the cooking process during the cooking process. For example, the user can make statements such as, "I'll fry it a little longer until it's slightly burnt," in accordance with the timing of when the information is presented.

[0090] The processor PR modifies the sensor data indicating the timing of cooking process transitions in the reference data RF based on the reproducing user's statements regarding the adjustments made to the timing of cooking process transitions during the reproduction of the reference data RF. The processor PR records the modified quantitative data in the reconstructed data RP as sensor data indicating the timing of cooking process transitions. This allows the recording user to make adjustments to the cooking process, for example, based on the preferences of the reproducing user or the person to whom the reproducing user is serving the food, such as "I want to add a little more salt" or "I want to heat it a little longer."

[0091] [6. Hardware Configuration Examples] Figure 8 shows an example of the hardware configuration of an information processing device IP.

[0092] The information processing device IP comprises a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, and a host bus 904a. The information processing device IP also includes a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 911, a communication device 913, and a sensor 915. The information processing device IP may have a processing circuit such as a DSP or ASIC in place of, or in conjunction with, the CPU 901.

[0093] The CPU 901 functions as an arithmetic processing unit and control unit, controlling the overall operation of the cooking data recording and reproduction system CS according to various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs and calculation parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change as needed during its execution. The CPU 901 functions, for example, as a processor PR.

[0094] The CPU 901, ROM 902, and RAM 903 are interconnected by a host bus 904a, which includes the CPU bus. The host bus 904a is connected to an external bus 904b, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 904. It is not necessary to configure the host bus 904a, bridge 904, and external bus 904b separately; these functions may be implemented on a single bus.

[0095] The input device 906 is implemented by a device into which information is input by the user, such as a mouse, keyboard, touch panel, button, microphone, switch, and lever. The input device 906 may also be a remote control device using infrared or other radio waves, or an external device such as a mobile phone or PDA that is compatible with the operation of the information processing device IP. Furthermore, the input device 906 may include, for example, an input control circuit that generates an input signal based on the information input by the user using the above-mentioned input means and outputs it to the CPU 901. The user of the information processing device IP can input various data or instruct processing operations to the information processing device IP by operating this input device 906. The input device 906 can, for example, form an input UI device IND.

[0096] The output device 907 is formed of a device capable of notifying the user of acquired information visually or audibly. Examples of such devices include display devices such as CRT displays, liquid crystal displays, plasma displays, EL displays, and lamps, as well as audio output devices such as speakers and headphones, and printers. The output device 907 outputs, for example, the results obtained from various processes performed by the information processing device IP. Specifically, the display device visually displays the results obtained from various processes performed by the information processing device IP in various formats such as text, images, tables, and graphs. On the other hand, the audio output device converts the audio signal, consisting of reproduced audio data or sound data, into an analog signal and outputs it audibly. The output device 907 can form, for example, an output device OTD.

[0097] The storage device 908 is a data storage device formed as an example of a storage unit of the information processing device IP. The storage device 908 can be implemented by, for example, a magnetic storage device such as an HDD, a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 908 may also include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded on the storage medium. This storage device 908 stores programs executed by the CPU 901, various data, and various data acquired from external sources. The storage device 908 can, for example, form a storage ST.

[0098] Drive 909 is a reader / writer for storage media, and is either built into or external to the information processing unit IP. Drive 909 reads information recorded on removable storage media such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory, and outputs it to RAM 903. Drive 909 can also write information to removable storage media.

[0099] Connection port 911 is an interface for connecting to external devices, and is a connection port for external devices that can transmit data via, for example, USB (Universal Serial Bus).

[0100] The communication device 913 is, for example, a communication interface formed by a communication device for connecting to the network 920. The communication device 913 is, for example, a communication card for wired or wireless LAN (Local Area Network), LTE (Long Term Evolution), Bluetooth (registered trademark), or WUSB (Wireless USB). Alternatively, the communication device 913 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication. This communication device 913 can, for example, send and receive signals to and from the Internet or other communication devices (such as a server SV) in accordance with a predetermined protocol such as TCP / IP.

[0101] Sensor 915 is a variety of sensors, such as a temperature sensor, weight sensor (force sensor), camera sensor, distance sensor, sound sensor, acceleration sensor, gyroscope sensor, and geomagnetic sensor. Sensor 915 acquires information about the cooking state of the food being cooked, such as the heating temperature, heating time, and weight of the food being cooked, as well as information about the surrounding environment of the information processing device IP, such as the brightness and noise level around the information processing device IP. Sensor 915 can form, for example, a sensor device SE.

[0102] Network 920 is a wired or wireless transmission path for information transmitted from devices connected to Network 920. For example, Network 920 may include public networks such as the Internet, telephone networks, and satellite communication networks, as well as various LANs (Local Area Networks) and WANs (Wide Area Networks), including Ethernet®. Network 920 may also include dedicated network lines such as IP-VPN (Internet Protocol-Virtual Private Network).

[0103] [7. Effects] The information processing device IP has a processor PR. The processor PR records the statements made by the recording user during cooking, linking them with sensor data measured at the time of the statements, and records them on a cooking data CD. In the information processing method of this disclosure, the processing of the information processing device IP is executed by a computer. The program of this disclosure causes the computer to implement the processing of the information processing device IP.

[0104] With this configuration, the recorded dishes are intuitively recreated based on the spoken content recorded on the cooking data CD.

[0105] The processor PR presents information related to the statements to the user reproducing the cooking data CD at the moment it detects the same cooking conditions as when the recording user made the statement.

[0106] With this configuration, information spoken by the recording user is presented at the appropriate time.

[0107] The PR processor identifies the object that the recording user is speaking about, based on the recording user's gaze.

[0108] This configuration ensures that the objects from which sensor data is acquired are properly identified.

[0109] The processor PR detects the timing of a cooking process change based on the recorded user's statements that notify it of the change in cooking process. The processor PR records the detected information regarding the timing of the cooking process change on the cooking data CD, linking it with the sensor data measured at the time of notification.

[0110] This configuration ensures that the timing for switching between cooking processes is set appropriately. Furthermore, it eliminates the hassle of manually setting the switching timing.

[0111] The processor PR detects the timing when the recording user utters a pre-set keyword as the timing of a change in the cooking process.

[0112] According to this configuration, the cooking process is appropriately partitioned based on keywords.

[0113] The processor PR modifies the sensor data indicating the timing of cooking process transitions in the cooking data, based on the user's comments regarding the correction of cooking process transition timings made during the reproduction of the cooking data CD.

[0114] This configuration allows for customization of the recipe to suit the user's preferences.

[0115] The recording user's statements include subjective information perceived by the recording user. The processor PR records sensor data that quantitatively represents this subjective information, linked to the recording user's statements, onto the cooking data CD.

[0116] This configuration allows for the objective definition of the cooking status as perceived by the recording user, based on sensor data.

[0117] The Processor PR uses an analytical model that has learned the sensations of the recording user to estimate sensor data indicated by subjective information contained in the recording user's statements. The Processor PR records the estimated sensor data in cooking data, linking it to the recording user's statements.

[0118] This configuration allows for the generation of quantitative recipes solely from the recorded user's statements (textual information).

[0119] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0120] [Note] Furthermore, this technology can also be configured as follows. (1) An information processing device having a processor that records statements made by a recording user during cooking, linking them with sensor data measured at the time of the statement, and recording them in cooking data. (2) The processor presents information related to the statement to the user reproducing the cooking data at the time it detects that the same cooking conditions as those at the time of the recording user's statement are present. The information processing device described in (1) above. (3) The processor identifies the object that is the subject of the recording user's statement, based on the recording user's gaze. The information processing device described in (2) above. (4) The processor detects the timing of the cooking process change based on the recorded user's statement notifying the change in the cooking process, and records the detected information regarding the timing of the cooking process change in the cooking data, linked with the sensor data measured at the time of notification. The information processing device described in (2) or (3) above. (5) The processor detects the timing at which the recording user utters a pre-set keyword as the timing of the switchover in the cooking process. The information processing device described in (4) above. (6) The processor modifies the sensor data indicating the timing of the cooking process transitions in the cooking data, based on the statements made by the user during the reproduction of the cooking data regarding the correction of the timing of the cooking process transitions. The information processing device described in (4) or (5) above. (7) The aforementioned statement includes subjective information perceived by the recording user, The processor records the sensor data, which quantitatively represents the subjective information, in the cooking data, linking it to the statement. An information processing device as described in any one of (1) through (6) above. (8) The processor uses an analysis model that has learned the sensations of the recording user to estimate the sensor data indicated by the subjective information, and records the estimated sensor data in the cooking data, linking it to the utterance. An information processing device as described in any one of (1) through (6) above. (9) A computer-based information processing method that records statements made by a recording user during cooking, linking them with sensor data measured at the time of the statement, and recording them in cooking data. (10) This program enables a computer to record cooking data by linking the user's statements during cooking with sensor data measured at the time of the statements. [Explanation of symbols]

[0121] CD Cooking Data IP Information Processing Device PR Processor

Claims

1. It has a processor that records the user's statements made during cooking, linking them with sensor data measured at the time of the statement, and records them in the cooking data. The processor modifies the sensor data indicating the timing of the cooking process transition in the cooking data based on a statement regarding the correction of the timing of the cooking process transition. Information processing device.

2. The processor presents information related to the statement to the user reproducing the cooking data at the time it detects that the same cooking conditions as those at the time of the recording user's statement are present. The information processing apparatus according to claim 1.

3. The processor identifies the object that is the subject of the recording user's statement, based on the recording user's gaze. The information processing apparatus according to claim 2.

4. The processor detects the timing of the cooking process change based on the recorded user's statement notifying the change in the cooking process, and records the detected information regarding the timing of the cooking process change in the cooking data, linked with the sensor data measured at the time of notification. The information processing apparatus according to claim 2.

5. The processor detects the timing at which the recording user utters a pre-set keyword as the timing of the switchover in the cooking process. The information processing apparatus according to claim 4.

6. The aforementioned statement includes subjective information perceived by the recording user, The processor records the sensor data, which quantitatively represents the subjective information, in the cooking data, linking it to the statement. The information processing apparatus according to claim 2.

7. The processor uses an analysis model that has learned the sensations of the recording user to estimate the sensor data indicated by the subjective information, and records the estimated sensor data in the cooking data, linking it to the statement. The information processing apparatus according to claim 6.

8. The system records the user's statements made during cooking, linking them to sensor data measured at the time of the statement, and includes them in the cooking data. Based on statements regarding the correction of the timing of the cooking process transitions, the sensor data indicating the timing of the cooking process transitions in the cooking data is corrected. An information processing method performed by a computer, which includes the ability to perform the following actions.

9. The system records the user's statements made during cooking, linking them to sensor data measured at the time of the statement, and includes them in the cooking data. Based on statements regarding the correction of the timing of the cooking process transitions, the sensor data indicating the timing of the cooking process transitions in the cooking data is corrected. A program that allows a computer to accomplish something.

Images