Information processing apparatus, information processing method, and information processing program

Abstract

To provide an information processing apparatus, an information processing method, and an information processing program that can improve a customer satisfaction.SOLUTION: An information processing apparatus includes: a storage unit that stores correspondence information in which ingredients containing meat are associated with a cooking procedure for meat-containing dishes in which the ingredients are used; a first acquisition unit that acquires image information relating to an image of a cooking area where the dish containing the meat is cooked; a specification unit that specifies the ingredient based on the image information acquired by the first acquisition unit; a second acquisition unit that acquires the cooking procedure according to the ingredient specified by the specification unit based on the correspondence information stored in the storage unit; and an output control unit that outputs cooking instruction information relating to a cooking instruction based on the cooking procedure acquired by the second acquisition unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, there has been an apparatus (support apparatus) that supports the work procedures of employees in a store (store operation). The support apparatus uses a camera to capture the actions of employees and the like who act according to the situation of customers who come to the store. The support apparatus accumulates the captured image information and conducts evaluations of employees and education for employees, etc., to support store operation (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The support apparatus described in Patent Document 1 supports store operation for employees. In stores such as restaurants, the cooking method of the dishes provided in the restaurant may be defined. Generally, it is common for cooking staff to confirm the cooking procedure and cook according to that cooking procedure. Currently, improvement of customer satisfaction is required, and it was necessary to confirm whether the cooking staff was cooking according to the cooking procedure. Note that the support apparatus described in Patent Document 1 does not improve customer satisfaction related to the cooking procedure.

[0005] The present disclosure provides an information processing apparatus, an information processing method, and an information processing program capable of improving customer satisfaction.

Means for Solving the Problems

[0006] One embodiment of the information processing device includes: a storage unit that stores correspondence information associated with ingredients including meat and cooking procedures for dishes including meat using those ingredients; a first acquisition unit that acquires image information relating to an image of a cooking area where a dish including meat is cooked; an identification unit that identifies ingredients based on the image information acquired by the first acquisition unit; a second acquisition unit that acquires cooking procedures corresponding to the ingredients identified by the identification unit based on the correspondence information stored in the storage unit; and an output control unit that outputs cooking instruction information relating to cooking instructions based on the cooking procedures acquired by the second acquisition unit. [Effects of the Invention]

[0007] According to one embodiment, ingredients are identified based on image information relating to an image of a cooking area where a dish containing meat is prepared, cooking procedures corresponding to the identified ingredients are obtained based on corresponding information, and cooking instruction information relating to cooking instructions is output based on the cooking procedure, thereby improving customer satisfaction. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram illustrating an information processing device according to one embodiment. [Figure 2] This is a block diagram illustrating an information processing device according to one embodiment. [Figure 3] This is a flowchart illustrating an information processing method according to one embodiment. [Modes for carrying out the invention]

[0009] One embodiment will be described below.

[0010] [Overview of Information Processing Device 1] First, an overview of the information processing device 1 according to one embodiment will be described. Figure 1 is a diagram illustrating an information processing device 1 according to one embodiment.

[0011] The information processing device 1 may be configured, for example, as an instruction device that provides instructions on the cooking procedure for dish 103. Alternatively, the information processing device 1 may be configured, for example, as a warning device or output device that outputs a warning if the cooking procedure differs from a predetermined cooking procedure. The information processing device 1 may be a device related to the cooking procedure of a dish using various ingredients including meat, or it may be a device related to the cooking procedure of a dish using various ingredients that do not include meat. A dish using ingredients including meat may be, for example, a dish cooked by boiling the ingredients including meat, or it may be a dish cooked by other methods. In other words, the information processing device 1 may be a device related to the cooking procedure of a dish 103 using various ingredients 102. The information processing device 1 may be a computer such as a server, desktop, laptop, or tablet. The following is an example of a cooking procedure for a dish that uses meat as an ingredient.

[0012] The storage unit 23 of the information processing device 1 (see Figure 2) may store correspondence information that associates ingredients including meat with cooking procedures for meat dishes that utilize those ingredients. The imaging unit 21 of the information processing device 1 captures an image of the cooking area 101 when preparing the dish 103 and generates image information. The cooking area 101 may be, for example, the kitchen of various facilities such as restaurants. In facilities such as restaurants, if the kitchen where the dish 103 is prepared is different from the area (fountain area) where fountains for parfaits, ice cream, and beverages such as cream soda are made, the cooking area 101 may be a concept that includes both the kitchen and the fountain area.

[0013] The information processing device 1 identifies the food ingredient 102 by recognizing the object recorded in the image information generated by the imaging unit 21. The information processing device 1 refers to the corresponding information and obtains the cooking procedure for the dish 103 that uses the identified food ingredient 102. Based on that cooking procedure, the information processing device 1 outputs cooking instruction information related to the cooking instructions.

[0014] [Details of Information Processing Device 1] Next, we will describe the details of the information processing device 1 according to one embodiment. Figure 2 is a block diagram illustrating an information processing device 1 according to one embodiment.

[0015] The information processing device 1 includes, for example, an imaging unit 21, a communication unit 22, a storage unit 23, a display unit 24, and a control unit 11. The communication unit 22, the storage unit 23, and the display unit 24 may be embodiments of the output unit. The control unit 11 includes, for example, a first acquisition unit 12, a specific unit 13, a second acquisition unit 14, a first estimation unit 15, a second estimation unit 16, a timing unit 17, and an output control unit 18. The control unit 11 may be configured by, for example, the arithmetic processing unit of the information processing device 1. The control unit 11 (for example, the arithmetic processing unit) may realize the functions of each unit (for example, the first acquisition unit 12, the specific unit 13, the second acquisition unit 14, the first estimation unit 15, the second estimation unit 16, the timing unit 17, and the output control unit 18) by appropriately reading and executing various programs stored in the storage unit 23, etc.

[0016] The imaging unit 21 may be a camera placed in the cooking area. The imaging unit 21 may consist of, for example, one or more units. The imaging unit 21 images the cooking area and generates image information in which the cooking area is recorded. The imaging unit 21 may capture images in the form of, for example, video or still images. When the imaging unit 21 captures still images, for example, it may capture images at predetermined intervals. These predetermined intervals may include various timings, such as every predetermined time or every time there is any change within the imaging range of the imaging unit 21. When there are changes, for example, various changes such as when an object moves and when a color changes may occur. When the object moves, for example, it may be when the food moves in order to put the food into the pot. When the color changes, for example, it may be when the color of the food in the pot changes because the food is being cooked. As a specific example, the imaging unit 21 continuously captures images and performs image recognition on the images by the control unit 11. When there is any change as exemplified above as a result of the recognition, the imaging unit 21 may capture the current cooking area and generate image information. The control unit 11 may perform image recognition, for example, by using a learned model that has learned an object and the color of the object, etc. The cooking area may be a concept including, for example, the kitchen and the fountain area as described above.

[0017] The communication unit 22 can transmit and receive various information to and from, for example, a device (external device) (not shown) outside the information processing apparatus 1. The external device may be, for example, a server and a user terminal used by the user of the information processing apparatus 1. The user terminal may be, for example, a desktop, a laptop, a tablet, and a smartphone, etc.

[0018] The storage unit 23 may store various information and programs, for example. An example of the storage unit 23 may be a memory, a solid state drive, and a hard disk drive, etc. Note that the storage unit 23 may be, for example, a storage area and a server on the cloud, etc.

[0019] The storage unit 23 stores correspondence information associating food ingredients with the cooking procedures of the dishes using those food ingredients. The correspondence information may be, for example, information associating food ingredients containing meat with the cooking procedures of dishes containing meat using those food ingredients. The cooking procedures define, for example, the timing of cooking the food ingredients, the cooking details, and the cooking time, etc., and may define the cooking details of one or more food ingredients according to the passage of time. The cooking procedures may be referred to as, for example, cooking steps and cooking manuals, etc. The cooking procedures may be, for example, various contents such as images, voices, and texts in which the cooking steps are recorded, may be various information indicating the cooking steps, and may be various information for controlling cooking equipment. For example, in the case of commercial use in restaurants, etc., the dishes to be cooked and the amount of food ingredients vary depending on demand forecasting and order details, etc. On the other hand, for example, in the case of household use, the dishes to be cooked and the amount of food ingredients are determined by the food ingredients to be cooked, the dishes to be eaten, and the family composition. The "food ingredient containing meat" in the present embodiment may be, for example, a food ingredient used for commercial purposes. Therefore, the amount of dishes and food ingredients to be cooked for the "food ingredient containing meat" in the present embodiment may vary according to demand forecasting and order details, etc. in the case of commercial use.

[0020] The display unit 24 is, for example, a display capable of displaying various characters, symbols, images, etc.

[0021] The first acquisition unit 12 acquires image information regarding an image of a cooking area for cooking a dish containing meat. The first acquisition unit 12 may acquire the image information from the imaging unit 21, for example. Or, when the image information generated by the imaging unit 21 is stored in a server (not shown), the first acquisition unit 12 may acquire the image information from that server, for example. Also, the first acquisition unit 12 may acquire the image information via a communication cable and a communication line (communication network), for example, and when the image information is recorded in an external memory (not shown), etc., when the external memory is inserted into an interface (not shown) provided in the information processing apparatus 1, the first acquisition unit 12 may acquire the image information from that external memory.

[0022] The identification unit 13 identifies the food ingredient based on the image information acquired by the first acquisition unit 12. The identification unit 13 may also identify the food ingredient by performing image recognition based on the image information. For example, the identification unit 13 may identify the type of food ingredient recorded in the image information by using a first trained model that has learned about food ingredients and types of food ingredients. As a more specific example, the identification unit 13 may identify meat as a food ingredient if meat is recorded in the image information by using images of various patterns of meat as an example of food ingredients (for example, various patterns such as shape and color) and a first trained model that has learned about meat (type of food ingredient). In addition to learning about meat as in the example above, the first trained model may be a model that has learned about food ingredients used in dishes cooked in the cooking area where the imaging unit 21 is located.

[0023] The first trained model may be generated by training performed by, for example, a learning unit (learning device) (not shown) of the information processing device 1. Alternatively, the first trained model may be generated by training performed outside the information processing device 1 (for example, a learning device (not shown)). Alternatively, the first trained model may be stored in, for example, a server (not shown). Alternatively, the first trained model may be stored in, for example, the storage unit 23. The information processing device 1 may receive the first trained model from an external learning device or server via the communication unit 22 and external memory (not shown), or from an internal learning unit or storage unit 23, through a receiving unit (not shown) constituting the control unit 11. The identification unit 13 acquires the first trained model received by the receiving unit, for example.

[0024] The second acquisition unit 14 acquires cooking procedures corresponding to the ingredients identified by the identification unit 13, based on the correspondence information stored in the memory unit 23. In other words, the second acquisition unit 14 refers to the correspondence information and acquires cooking procedures corresponding to the identified ingredients. For example, if the identified ingredient is meat, the second acquisition unit 14 refers to the correspondence information and acquires cooking procedures for dishes that use meat.

[0025] In this case, the second acquisition unit 14 may, for example, acquire cooking procedures corresponding to ingredients identified over time. For example, if meat is identified first by the identification unit 13 based on image information, the second acquisition unit 14 may acquire the cooking procedures for the dish in which the meat is first cooked.

[0026] Furthermore, the second acquisition unit 14 may acquire cooking procedures corresponding to multiple ingredients identified over time. For example, if the identification unit 13 identifies meat first based on image information, and then identifies onions, the second acquisition unit 14 may acquire cooking procedures for a dish in which the meat is cooked first, and then the onions are cooked. Note that the identified ingredients are not limited to two as in the example above, but may be three or more ingredients.

[0027] The first estimation unit 15 may estimate whether the food is cooked according to the cooking procedure, based on the food ingredients recorded in the image information acquired by the first acquisition unit 12 and appearing in chronological order, and the cooking procedure acquired by the second acquisition unit 14. For example, the first estimation unit 15 may use a second trained model that has learned the food ingredients appearing in chronological order, the cooking content of those food ingredients, and the cooking procedure using those food ingredients to estimate whether the food ingredients identified in chronological order by the identification unit 13 are cooked according to the cooking procedure. Here, the cooking content may include, for example, how the food is cut, cooking methods such as boiling, grilling, simmering, and steaming, and various cooking-related details such as the cooking time for each cooking method. The cooking procedure may include, for example, cooking content according to the chronological order.

[0028] The second trained model may be generated by training using various learning units (learning devices) (not shown), similar to the first trained model used in the specific unit 13 described above. Similarly, the first estimation unit 15 may acquire the second trained model from the receiving unit (not shown) of the information processing device 1, for example, if the receiving unit receives the second trained model.

[0029] In other words, the first estimation unit 15, for example, uses the second trained model to estimate whether the ingredients recorded in the image information (ingredients appearing in a time series) are cooked according to the cooking content and cooking time. Furthermore, the first estimation unit 15 may, for example, estimate that cooking is being carried out according to the cooking procedure based on image information, and may also estimate that all of the cooking procedures have been completed. That is, the first estimation unit 15 may, for example, use the second trained model and image information to estimate that all of the cooking procedures have been completed.

[0030] The first estimation unit 15 may, for example, estimate whether the food is being cooked according to the cooking procedure, based on the food ingredients recorded in the image information based on real-time imaging by the imaging unit 21 (food ingredients identified in real-time by the identification unit 13) and the cooking procedure using those food ingredients acquired by the second acquisition unit 14. Alternatively, the first estimation unit 15 may estimate whether the food has been cooked according to the cooking procedure, for example, based on the food ingredients recorded in the image information previously captured by the imaging unit 21 (food ingredients identified by the identification unit 13) and the cooking procedure using those food ingredients acquired by the second acquisition unit 14.

[0031] The second estimation unit 16 may estimate the amount of food remaining in the cooking utensils based on the image information acquired by the first acquisition unit 12. For example, after cooking is finished, the food may be stored in a large pot (cooking utensil), and each time a food order is made, the food is transferred from the large pot to a serving dish for the customer and then served to that customer. The food stored in the cooking utensils mentioned above may also include, for example, food that has been transferred from the cooking utensils to a storage container. The second estimation unit 16 may, for example, estimate the amount of food remaining in the cooking utensils based on image information of the cooking utensils themselves. The amount of food remaining may be set in multiple stages, such as a lot, a normal amount, and a little amount, or it may be any other amount. As an example in this case, the second estimation unit 16 may use a third trained model that has learned about cooking utensils and the amount of food in them to estimate the amount of food remaining in the cooking utensils recorded in the image information.

[0032] The third trained model may be generated by training using various learning units (learning devices) (not shown), similar to the first trained model used in the specific unit 13 described above. Similarly, the second estimation unit 16 may acquire the third trained model from the receiving unit (not shown) of the information processing device 1, for example, if the receiving unit receives the third trained model.

[0033] The second estimation unit 16 may estimate the time it takes for the food to run out based on the change in the amount of food remaining over time. That is, the second estimation unit 16 may, for example, acquire the change in the amount of food remaining over time based on temporally different image information. In this case, the second estimation unit 16 may, for example, use various known estimation methods to estimate the time it takes for the food to run out from the change in the amount of food remaining over time. For example, if the change in the amount of food remaining over time can be approximated by a linear or quadratic equation, the second estimation unit 16 may use that approximation to estimate the time it takes for the food to run out, or it may use any other estimation method to estimate the time it takes for the food to run out.

[0034] Furthermore, the second estimation unit 16 may obtain the status of food orders, for example, by using a POS (Point of Sale) system that takes food orders in a restaurant or the like. The second estimation unit 16 may estimate that if there are relatively many food orders, the remaining amount of food will decrease relatively quickly. Similarly, the second estimation unit 16 may estimate that if there are relatively few food orders, the remaining amount of food will decrease relatively slowly. In other words, the second estimation unit 16 may estimate the time it takes for the food to run out based on the number of food orders. In this case, the second estimation unit 16 may estimate the time it takes for the food to run out based, for example, on at least one of the current number of food orders and the past number of food orders. The past number of food orders may be at least one selected from, for example, the group of past food orders for the same time period as the current time of estimation and the group of past food orders for the same day of the week as the current time of estimation.

[0035] Furthermore, the second estimation unit 16 may, for example, estimate the time (first time) when the food will run out based on the change in the amount of food remaining over time, and then estimate a second time by correcting the first time to be later or earlier based on at least one of the current number of food orders and the past number of food orders using the POS system, and set the second time as the time when the food will run out.

[0036] Furthermore, the second estimation unit 16 may, for example, obtain the amount of food consumed based on the change in the amount of food remaining over time. Similarly, the second estimation unit 16 may, for example, obtain the order quantity using a POS system. In this case, the second estimation unit 16 may, for example, obtain the amount of food consumed or the order quantity per unit time based on the amount of food consumed or the order quantity of food, or it may obtain the amount of food consumed or the order quantity within the most recent predetermined time (or the amount of food consumed or the order quantity per unit time within the most recent predetermined time, etc.). The most recent predetermined time may be various times, such as 1 hour, 2 hours, and 3 hours. In this case, the predetermined time may be, for example, a time shorter than the expiration date for the food that has been prepared. The second estimation unit 16 may, for example, estimate the amount of food to be cooked again based on the decrease or order quantity exemplified above. For example, if the decrease or order quantity is relatively large, the second estimation unit 16 may relatively increase the amount of food to be cooked again. Also, for example, if the decrease or order quantity is relatively small, the second estimation unit 16 may relatively decrease the amount of food to be cooked again. In this case, the information processing device 1 can estimate the amount of food to be prepared that is more in line with the current amount of food consumed or the amount of food ordered, for example, by considering the amount of food consumed or the amount of food ordered most recently. Based on this amount of food prepared, the risk of food being discarded without being consumed (risk of food loss) can be reduced.

[0037] Furthermore, the second estimation unit 16 may estimate the amount of food to be cooked again, for example, based on the remaining amount of food stored in the cooking utensils and the amount of decrease or order amount as exemplified above.

[0038] Furthermore, the second estimation unit 16 may estimate the order volume of dishes to be sold in the future based, for example, on the order volume of dishes in the past using the POS system. The order volume of dishes in the past using the POS system may be, for example, the order volume from the same day of the week and time of day as the current date and time. If the day of the week and time of day are the same, the order volume of dishes often shows roughly the same trend. The second estimation unit 16 may, for example, estimate the amount of food to be cooked again based on the past order quantities exemplified above. In this case, the second estimation unit 16 may estimate the amount of food to be cooked again based on the remaining amount of food stored in the cooking equipment used at the present time and the past order quantities. That is, the second estimation unit 16 may estimate the amount of food to be cooked again by subtracting the remaining amount of food from the amount of food to be cooked estimated based on past order quantities.

[0039] Furthermore, the timing unit 17, described later, starts timing from the moment cooking is completed and counts the time until the expiration date of the cooked food. Any food remaining when the expiration date is reached will be discarded. As described above, the second estimation unit 16 estimates the remaining amount of food based on image information. In this case, the second estimation unit 16 may estimate the amount of food remaining when the expiration date is reached, for example, based on the change in the remaining amount of food over time, the amount of food ordered recently or in the past using the POS system, and the timing time of the timing unit 17, which will be described later. The second estimation unit 16 may estimate the amount of food to be cooked again, for example, based on the remaining amount of food stored in the cooking equipment at the present time, the remaining amount of food that will remain when the expiration date is reached, the amount of food that decreases over time according to the changes in the food, or the past or recent order quantity using the POS system. That is, the second estimation unit 16 may, for example, obtain an amount A by subtracting the remaining amount of food from the amount of food to be cooked estimated based on the amount of food that decreases over time according to the changes in the food, or the past or recent order quantity using the POS system, and then estimate the amount of food to be cooked again by adding the remaining amount of food that will remain when the expiration date is reached to amount A to amount B.

[0040] The timing unit 17 may start timing when the first estimation unit 15 estimates the end of cooking based on image information. That is, the timing unit 17 starts timing when cooking is finished. For example, the timing unit 17 may use the timing when the first estimation unit 15 estimates whether cooking has been done according to the cooking procedure, and further estimates whether all cooking procedures have been completed, thereby estimating the end of all cooking procedures. The timing unit 17 may be, for example, a count-up timer or a count-down timer.

[0041] The output control unit 18 may control the output unit to output, for example, at least one of the following: the cooking procedure acquired by the second acquisition unit 14, the estimation result by the first estimation unit 15, the estimation result by the second estimation unit 16, and the time group (first group) measured by the timing unit 17. The output unit may also be a communication unit 22, a storage unit 23, a display unit 24, etc. In other words, the output control unit 18 may control the communication unit 22 to transmit information relating to at least one of the first groups described above to an external device (not shown). The external device may be, for example, a server and a user terminal. The output control unit 18 may, for example, control the storage unit 23 to store information relating to at least one of the first groups described above. The output control unit 18 may, for example, control the display unit 24 to display at least one of the contents of the first group described above. Here, the output control unit 18 may, for example, control the output control unit 18 to output the cooking procedure acquired by the second acquisition unit 14, outputting the procedure details for each cooking step. For example, when the identification unit 13 identifies ingredients based on image information, the output control unit 18 may, based on each cooking detail in the cooking procedure and the ingredients used in each cooking detail, identify which cooking detail in the cooking procedure the current stage of cooking is, and control the output control unit 18 to output the current cooking detail or the next cooking detail (or subsequent cooking details). In this case, the output control unit 18 may, for example, use the estimation result from the first estimation unit 15 to estimate the cooking procedure (the current cooking detail). As an example, the output control unit 18 can estimate which stage of the cooking procedure the cooking detail of the dish being cooked is at and output the next cooking detail in advance (for example, display it on the display unit).

[0042] The output control unit 18 may control the output unit to output cooking instruction information related to cooking instructions based on the cooking procedure acquired by the second acquisition unit 14. The cooking instruction information may be, for example, information recorded in the corresponding information stored in the storage unit 23. That is, the cooking instruction information may be, for example, various contents such as images, audio and text that record the cooking procedure, various information that instructs the cooking procedure, or various information that controls cooking equipment. The output control unit 18 may, for example, refer to the corresponding information stored in the storage unit 23 and control the output unit to output cooking instruction information corresponding to the cooking procedure acquired by the second acquisition unit 14. The control of the output control unit 18 described above may also include the output control unit 18 controlling the output unit to output cooking instruction information corresponding to the ingredients identified by the identification unit 13 based on the corresponding information stored in the storage unit 23.

[0043] The output control unit 18 may, for example, control the communication unit 22 to output cooking instruction information to an external device (not shown). In this case, the external device may be, for example, a server, a user terminal, or cooking equipment. In this case, the cooking appliance may, for example, acquire cooking instruction information and then perform cooking based on that cooking procedure. That is, the information processing device 1 can, for example, control the cooking appliance to perform automatic cooking.

[0044] The output control unit 18 may, for example, control the storage unit 23 to store cooking instruction information. In this case, the output control unit 18 may store in the storage unit 23 log information that associates the image information acquired by the first acquisition unit 12, the ingredients identified by the identification unit 13, the cooking procedure acquired by the second acquisition unit 14, and the cooking instruction information. By storing this log information, it becomes possible to use it later for verifying the cooking procedure and training cooking staff.

[0045] The output control unit 18 may, for example, control the display unit 24 to display content based on cooking instruction information. In this case, the output control unit 18 may control the output of audio based on cooking instruction information from, for example, a speaker (not shown) located on or independently of the display unit 24. This allows cooking staff to perform cooking while checking the cooking procedure displayed on the display unit 24. For example, even cooking staff with little experience can reliably perform cooking according to the procedure.

[0046] The output control unit 18 may control the output unit to output a warning if the first estimation unit 15 estimates that the food has not been cooked according to the cooking procedure. The warning may be various warning content such as sound, voice, image, and text that indicates that the food has not been cooked according to the cooking procedure. That is, the warning may be various warning content that indicates, for example, the cooking procedure of the dish (for example, the cooking content and cooking time). Also, if, for example, ingredients that are not used in the dish (the cooking procedure for that dish) (for example, ingredients that are not used in the current or next cooking step) are recorded in the image information, the warning may be various warning content that indicates that the food has not been cooked according to the cooking procedure. Also, if, for example, one of the cooking steps in the cooking procedure is skipped, the warning may be various warning content that indicates that the food has not been cooked according to the cooking procedure. That is, for example, the system may control the output unit to output a warning if an employee of a restaurant or the like puts in the wrong ingredients or skips a cooking step (skipping a task). In other words, the output control unit 18 may, for example, control the communication unit 22 to transmit warning information to an external device (not shown). The external device may be, for example, a server and a user terminal. The user terminal here may be, for example, a terminal used by cooking staff who are actually cooking in the kitchen of a store, a terminal used by various users such as the on-site manager of the store, and a terminal located in the management headquarters that manages the store and used by users of the management headquarters. The output control unit 18 may, for example, control the storage unit 23 to store information related to warnings. The output control unit 18 may, for example, control the display unit 24 to display the warning content. In this case, the output control unit 18 may, for example, control the display unit 24 to output at least one of a warning sound and a warning voice from a speaker (not shown) provided with or independently provided in the display unit 24. In this case, the output control unit 18 may, for example, estimate which stage of the cooking procedure the dish being cooked is at, as described above, and output the current cooking details, or output the next cooking details in advance.

[0047] The output control unit 18 may control the output unit to output a cooking instruction for the dish if the remaining amount of the dish estimated by the second estimation unit 16 is below a threshold. The cooking instruction may be an instruction to start cooking. The threshold may be, for example, a lower limit value for the amount of food stored, such that if the amount of food continues to decrease, it will become impossible to provide food to customers in the relatively near future. The threshold may be set appropriately depending on the dish, for example. The output control unit 18 may output cooking instructions by controlling at least one selected from the group consisting of the communication unit 22, the storage unit 23, and the display unit 24, as described above.

[0048] The output control unit 18 may control the output unit to output a cooking instruction before the time estimated by the second estimation unit 16 has elapsed. In this case, the output control unit 18 may, for example, control the output unit to output a cooking instruction at a timing earlier than the estimated cooking time, based on the time when the food is estimated to be finished, using the cooking time recorded in the corresponding information stored in the memory unit 23 and the time estimated by the second estimation unit 16 as a reference. That is, the output control unit 18 may, for example, consider the cooking time of the food and output a cooking instruction so that the cooking of the food is completed before the time estimated by the second estimation unit 16 has elapsed. For this reason, the corresponding information may associate ingredients, the cooking procedure for a dish using those ingredients, and the cooking time of that dish (the time from the start of cooking to the end of all procedures). The output control unit 18 may output cooking instructions by controlling at least one selected from the group consisting of the communication unit 22, the storage unit 23, and the display unit 24, as described above. Furthermore, the output control unit 18 may, for example, control the output unit to include the cooking amount estimated by the second estimation unit 16 in the cooking instructions when outputting the instructions.

[0049] The output control unit 18 may control the output unit to output a cooking instruction at a time when the time measured by the timing unit 17 has not yet elapsed to a predetermined time set as the food's disposal deadline. A food that has been cooked may have a disposal deadline (disposal deadline), such as an expiration date or best-before date. The disposal deadline may be set to various times (predetermined times) depending on the dish. The output control unit 18 outputs a cooking instruction before the time measured by the timing unit 17 has elapsed to the food's disposal deadline (predetermined time). In this case, the output control unit 18 may, for example, take into account the cooking time of the dish, as described above, and output a cooking instruction so that the cooking of the dish is completed before the food's disposal deadline (predetermined time) has elapsed.

[0050] [Information Processing Methods] Next, an information processing method according to one embodiment will be described. Figure 3 is a flowchart illustrating an information processing method according to one embodiment.

[0051] In step ST101, the first acquisition unit 12 acquires image information relating to an image of a cooking area where a dish containing meat is prepared.

[0052] In step ST102, the identification unit 13 identifies the food ingredient based on the image information acquired in step ST101. As an example, the identification unit 13 may use a first trained model or the like to identify the type of food ingredient recorded in the image information. The first trained model may be, for example, a model generated by learning food ingredients and types of food ingredients.

[0053] In step ST103, the second acquisition unit 14 acquires a cooking procedure corresponding to the ingredients identified in step ST102, based on the correspondence information stored in the storage unit 23. In this case, the second acquisition unit 14 may acquire a cooking procedure corresponding to ingredients identified over time, for example. Alternatively, the second acquisition unit 14 may acquire a cooking procedure corresponding to multiple ingredients identified over time, for example.

[0054] In step ST104, the first estimation unit 15 estimates whether the food has been cooked according to the cooking procedure, based on the ingredients recorded in the image information acquired in step ST101 that appear in time series (ingredients identified in step ST102) and the cooking procedure acquired in step ST103. As an example, the first estimation unit 15 uses a second trained model or the like to estimate whether the ingredients recorded in the image information (ingredients that appear in time series) have been cooked according to the cooking content and cooking time. The second trained model may be, for example, a model generated by learning the ingredients that appear in time series, the cooking content of those ingredients, and the cooking procedure using those ingredients. Furthermore, the first estimation unit 15 may, for example, estimate that cooking is being carried out according to the cooking procedure based on image information, and may also estimate that all of the cooking procedures have been completed.

[0055] Step ST105 may be a process performed after the series of processes are completed, that is, after the cooking is finished. In other words, in step ST105, the second estimation unit 16 may estimate the remaining amount of food (already finished food) stored in the cooking utensils based on the image information in step ST101. The second estimation unit 16 may, for example, estimate the time it will take for the food to run out based on the change in the remaining amount of food over time. Here, the timing unit 17 may start timing if it estimates the end of cooking based on the image information in step ST104.

[0056] Step ST106 is an output process that is performed both before and after the cooking is complete.

[0057] Before the cooking is complete, in step ST106, output processing such as the following may be performed. In step ST106, the output control unit 18 may control the output unit to output cooking instruction information based on the cooking procedure acquired in step ST103. If the output control unit 18 estimates in step ST104 that the food has not been cooked according to the cooking procedure, it may control the output unit to output a warning. The output unit may be, for example, a communication unit 22, a storage unit 23, and a display unit 24.

[0058] Furthermore, if the dish has already been completed, in step ST106, output processing such as the following may be performed. In step ST106, the output control unit 18 may, for example, control the output unit to output an instruction to cook the dish if the remaining amount of the dish estimated in step ST105 is below a threshold. The output control unit 18 may, for example, control the output unit to output a cooking instruction before the time estimated in step ST105 (the time estimated to be when the food will run out) has elapsed. The output control unit 18 may, for example, control the output unit to output a cooking instruction at a timing earlier than the cooking time, based on the time estimated to be when the food will run out, using the cooking time recorded in the corresponding information stored in the memory unit 23 and the time estimated in step ST105 as a reference. The output control unit 18 may, for example, control the output unit to output a cooking instruction at a timing before the time measured by the timing unit 17 in step ST105 has elapsed, which is set as the expiration date for discarding the food. The output unit may be, for example, a communication unit 22, a storage unit 23, and a display unit 24, as described above.

[0059] Each part of the information processing device 1 described above may be implemented as a function of a computer's arithmetic processing unit or the like. That is, the first acquisition unit 12, the identification unit 13, the second acquisition unit 14, the first estimation unit 15, the second estimation unit 16, the timing unit 17, and the output control unit 18 (control unit 11) of the information processing device 1 may be implemented as a first acquisition function, identification function, second acquisition function, first estimation function, second estimation function, timing function, and output control function (control function), respectively, by a computer's arithmetic processing unit or the like. Information processing programs can enable computers to implement the functions described above. Information processing programs may be recorded on computer-readable, non-temporary recording media such as memory, solid-state drives, hard disk drives, or optical discs. The recording media may also be referred to as non-temporary computer-readable media. Furthermore, as described above, each part of the information processing device 1 may be implemented by a computer's arithmetic processing unit, etc. Such arithmetic processing unit, etc. is composed of, for example, an integrated circuit. For this reason, each part of the information processing device 1 may be implemented as a circuit that constitutes an arithmetic processing unit, etc. That is, the first acquisition unit 12, the identification unit 13, the second acquisition unit 14, the first estimation unit 15, the second estimation unit 16, the timing unit 17, and the output control unit 18 (control unit 11) of the information processing device 1 may be implemented as a first acquisition circuit, identification circuit, second acquisition circuit, first estimation circuit, second estimation circuit, timing circuit, and output control circuit (control circuit) that constitute a computer's arithmetic processing unit, etc. Furthermore, the imaging unit 21, communication unit 22, storage unit 23, and display unit 24 (output unit) of the information processing device 1 may be implemented as, for example, an imaging function including the functions of a processing unit, as well as a communication function, storage function, and display function (output function). Also, the imaging unit 21, communication unit 22, storage unit 23, and display unit 24 (output unit) of the information processing device 1 may be implemented as an imaging circuit, as well as a communication circuit, storage circuit, and display circuit (output circuit) by being composed of, for example, an integrated circuit. Furthermore, the imaging unit 21, communication unit 22, storage unit 23, and display unit 24 (output unit) of the information processing device 1 may be configured as an imaging device, as well as a communication device, storage device, and display device (output device) by being composed of, for example, multiple devices.

[0060] The information processing device 1 can be configured to combine one or any multiple of the above-described parts. In this disclosure, the term "information" is used, but the term "information" can be replaced with "data," and the term "data" can be replaced with "information."

[0061] [Aspects and Effects of This Embodiment] Next, one aspect of this embodiment and the effects achieved by each aspect will be described. Note that this embodiment is not limited to the aspects described below, and may be realized by appropriately combining the parts described above. Furthermore, the effects described below are examples, and the effects achieved by each aspect are not limited to those described below. Also, each aspect may achieve, for example, at least one of the effects described below.

[0062] (Aspect 1) One embodiment of the information processing device includes: a storage unit that stores correspondence information associated with ingredients including meat and cooking procedures for dishes including meat using those ingredients; a first acquisition unit that acquires image information relating to an image of a cooking area where a dish including meat is cooked; an identification unit that identifies ingredients based on the image information acquired by the first acquisition unit; a second acquisition unit that acquires cooking procedures corresponding to the ingredients identified by the identification unit based on the correspondence information stored in the storage unit; and an output control unit that outputs cooking instruction information relating to cooking instructions based on the cooking procedures acquired by the second acquisition unit. In other words, the information processing device can notify external parties of cooking procedures, etc. This allows cooking staff and managers to verify whether they are cooking according to the procedures. Cooking staff can, for example, cook while checking the cooking procedures. Cooking staff can, for example, cook according to the cooking procedures. Furthermore, cooking equipment can, for example, control its various parts to cook according to the cooking procedures based on the cooking instruction information. Furthermore, while cooking often requires cooking experience, the information processing device can provide cooking instructions, allowing even cooking staff with relatively little experience to follow the cooking procedures. Therefore, the information processing device can provide customers with consistently high-quality dishes and improve customer satisfaction. Furthermore, the information processing device can automatically acquire cooking procedures based on the ingredients recorded in the image information. Therefore, when employees of restaurants or other establishments use ingredients according to normal procedures, the information processing device can automatically identify the dish and issue cooking instructions. In this case, employees can save time and effort by not having to select cooking procedures, and can also prevent incorrect selection of procedures. In other words, the information processing device differs from home cooking, where users acquire cooking procedures (recipes) themselves using various media such as cookbooks.

[0063] (Aspect 2) One embodiment of the information processing device may include a first estimation unit that estimates whether the food is being cooked according to the cooking procedure, based on the ingredients recorded in the image information acquired by the first acquisition unit and the cooking procedure acquired by the second acquisition unit. In other words, the information processing device can estimate whether one or more ingredients (ingredients recorded in image information) appear sequentially in a time series, appear in the order of the cooking procedure, and whether they are being cooked according to that procedure. This allows cooking staff and managers to verify, based on the estimation results, whether the cooking is being carried out according to the procedure. Furthermore, the information processing device can be used to estimate whether cooking is being carried out according to the cooking procedure, and this result can also be used for training the cooking staff who are performing the cooking.

[0064] (Aspect 3) In one embodiment of the information processing device, the output control unit may output a warning if the first estimation unit estimates that the food has not been cooked according to the cooking procedure. This allows cooking staff and managers to verify, based on the warning, that they are not following the cooking procedures.

[0065] (Aspect 4) One embodiment of the information processing device may include a second estimation unit that estimates the remaining amount of food stored in cooking utensils based on image information acquired by a first acquisition unit. This allows the information processing device to notify cooking staff and managers of the remaining amount of food.

[0066] (Appendix 5) In one embodiment of the information processing device, the output control unit may output an instruction to cook the dish if the remaining amount of the dish estimated by the second estimation unit is below a threshold. This allows the information processing device to send a notification prompting the preparation of the dish before it runs out.

[0067] (Aspect 6) In one embodiment of the information processing device, the second estimation unit estimates the time it will take for the food to run out based on the change in the remaining amount of food over time, and the output control unit outputs an instruction to cook the food before the time estimated by the second estimation unit has elapsed. This allows the information processing device to send a notification prompting the preparation of the dish before it runs out.

[0068] (Aspect 7) In one embodiment of the information processing device, the output control unit may output a cooking instruction earlier than the cooking time, based on the estimated time when the food will be gone, using the cooking time recorded in the corresponding information stored in the memory unit and the time estimated by the second estimation unit as a reference. This allows the information processing device to send a notification prompting the preparation of the dish before it runs out.

[0069] (Pattern 8) In one embodiment, the information processing device includes a timing unit that starts timing when the first estimation unit estimates the end of cooking based on image information, and the output control unit may output a cooking instruction at a timing before the time timing measured by the timing unit has elapsed, which is set as the expiration date for discarding the food. This allows the information processing device to issue a notification urging the user to prepare the food before discarding it.

[0070] (Aspect 9) In one embodiment of the information processing method, a computer equipped with a storage unit that stores correspondence information relating ingredients including meat and cooking procedures for dishes including meat using those ingredients performs the following steps: a first acquisition step of acquiring image information relating to an image of a cooking area where a dish including meat is cooked; an identification step of identifying ingredients based on the image information acquired in the first acquisition step; a second acquisition step of acquiring cooking procedures corresponding to the ingredients identified in the identification step based on the correspondence information stored in the storage unit; and an output control step of outputting cooking instruction information relating to cooking instructions based on the cooking procedures acquired in the second acquisition step. As a result, the information processing method can achieve the same effects as the information processing apparatus of the above-described embodiment.

[0071] (Aspect 10) One embodiment of an information processing program enables a computer to implement: a memory function that stores correspondence information relating ingredients including meat and cooking procedures for dishes containing meat using those ingredients; a first acquisition function that acquires image information relating to an image of a cooking area where a dish containing meat is prepared; an identification function that identifies ingredients based on the image information acquired by the first acquisition function; a second acquisition function that acquires cooking procedures corresponding to the ingredients identified by the identification function based on the correspondence information stored in the memory function; and an output control function that outputs cooking instruction information relating to cooking instructions based on the cooking procedures acquired by the second acquisition function. As a result, the information processing program can achieve the same effects as the information processing apparatus of the above-described embodiment.

[0072] (Aspect 11) One embodiment of the information processing device includes: a storage unit that stores correspondence information associated with ingredients and cooking procedures for dishes using those ingredients; a first acquisition unit that acquires image information relating to an image of a cooking area where a dish is prepared; an identification unit that identifies ingredients based on the image information acquired by the first acquisition unit; a second acquisition unit that acquires cooking procedures corresponding to the ingredients identified by the identification unit based on the correspondence information stored in the storage unit; and an output control unit that outputs cooking instruction information relating to cooking instructions based on the cooking procedures acquired by the second acquisition unit. As a result, the information processing apparatus of this embodiment can achieve the same effects as the information processing apparatus of the first embodiment described above. This embodiment can also take the same (or different) configuration as the information processing method and information processing program.

[0073] (Aspect 12) One embodiment of the information processing device includes: a storage unit that stores correspondence information associated with ingredients including meat and cooking procedures for stews containing meat using those ingredients; a first acquisition unit that acquires image information relating to an image of a cooking area for cooking a stew containing meat; an identification unit that identifies ingredients based on the image information acquired by the first acquisition unit; a second acquisition unit that acquires cooking procedures corresponding to the ingredients identified by the identification unit based on the correspondence information stored in the storage unit; and an output control unit that outputs cooking instruction information relating to cooking instructions based on the cooking procedures acquired by the second acquisition unit. As a result, the information processing apparatus of this embodiment can achieve the same effects as the information processing apparatus of the first embodiment described above. This embodiment can also take the same (or different) configuration as the information processing method and information processing program. [Explanation of Symbols]

[0074] 1. Information Processing Device 11 Control Unit 12 First acquisition part 13 Specific section 14 Second acquisition part 15 1st estimation part 16 Second estimation part 17 Timing section 18 Output control unit 21 Imaging Department 22 Communications Department 23 Memory section 24 Display 101 Cooking Area 102 Ingredients 103 Dishes

Claims

1. A storage unit that stores correspondence information that associates ingredients with cooking procedures for dishes that use the ingredients; A first acquisition unit that acquires image information relating to an image of a cooking area in which the food is cooked; An identification unit that identifies an ingredient based on the image information acquired by the first acquisition unit; a second acquisition unit that acquires a cooking procedure corresponding to the ingredients identified by the identification unit based on the correspondence information stored in the storage unit; an output control unit that outputs cooking instruction information regarding cooking instructions based on the cooking procedure acquired by the second acquisition unit; An information processing device comprising:

2. a first estimation unit that estimates whether the food has been cooked according to a cooking procedure based on ingredients that appear in chronological order and are recorded in the image information acquired by the first acquisition unit and the cooking procedure acquired by the second acquisition unit; The information processing device according to claim 1 .

3. The output control unit outputs a warning when the first estimation unit estimates that the food has not been cooked according to a cooking procedure. The information processing device according to claim 2 .

4. A second estimation unit is provided for estimating the remaining amount of food stored in the utensil used for cooking based on the image information acquired by the first acquisition unit.

4. The information processing device according to claim 2 or 3.

5. The output control unit outputs an instruction to cook the food when the remaining amount of the food estimated by the second estimation unit is equal to or less than a threshold value. The information processing device according to claim 4.

6. The second estimation unit estimates a time until the food runs out based on a change over time in the remaining amount of food, The output control unit outputs an instruction to cook a dish before the time estimated by the second estimation unit has elapsed. The information processing device according to claim 4.

7. The output control unit outputs an instruction to cook a dish at a timing before the cooking time based on a time when the dish is estimated to run out, based on a cooking time recorded in the correspondence information stored in the storage unit and the time estimated by the second estimation unit. The information processing device according to claim 6.

8. a timing unit that starts timing when the first estimation unit estimates the end of cooking based on the image information, The output control unit outputs a cooking instruction before the time measured by the timing unit reaches a predetermined time set as a disposal deadline for the food. The information processing device according to claim 7.

9. A computer having a memory unit that stores correspondence information that associates ingredients with cooking procedures for dishes using the ingredients, a first acquisition step of acquiring image information relating to an image of a cooking area in which the dish is to be cooked; An identification step of identifying an ingredient based on the image information acquired by the first acquisition step; A second acquisition step of acquiring a cooking procedure corresponding to the ingredients identified in the identification step based on the corresponding information stored in the storage unit; an output control step of outputting cooking instruction information regarding cooking instructions based on the cooking procedure acquired in the second acquisition step; An information processing method for performing the above.

10. On the computer, A storage function for storing correspondence information that associates ingredients with cooking procedures for dishes using the ingredients; a first acquisition function for acquiring image information relating to an image of a cooking area in which the food is to be cooked; An identification function that identifies ingredients based on the image information acquired by the first acquisition function; A second acquisition function that acquires a cooking procedure corresponding to the ingredients identified by the identification function based on the corresponding information stored in the storage function; an output control function that outputs cooking instruction information regarding cooking instructions based on the cooking procedure acquired by the second acquisition function; An information processing program that realizes this.

Images