Information processing apparatus, information processing method, and information processing program

Abstract

To provide an information processing apparatus, an information processing method and an information processing program configured to increase customer satisfaction.SOLUTION: An information processing apparatus includes: a first receiving unit which receives a trained model generated by learning a container, food in the container, and the quantity of the food in the container; a second receiving unit which receives image information on an image of the container containing the food; an estimation unit which estimates the quantity of food in the container recorded in the image information on the basis of the trained model received by the first receiving unit and the image information received by the second receiving unit; and an output control unit which performs control to generate a predetermined output in accordance with the quantity of the food estimated by the estimation unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there exists an analysis device that analyzes the sales status of products (see Patent Document 1). The analysis device detects when a product has disappeared from an area where the product is placed, and when the product has returned to that area, and detects a product replacement based on the detection result. Furthermore, the analysis device generates an analysis result based on the detection result of the product replacement. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-149687 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, in a store that provides food and drink in a buffet format, if the food and drink in the distribution area runs out, customers cannot take the food and drink, which reduces customer satisfaction. For this reason, there is a desire in the store to replenish the food and drink in the distribution area before it runs out. The analysis device described in Patent Document 1 is a device that analyzes the sales status of products, and does not, for example, detect the amount of products before the products disappear from the placement area and issue a notification urging the product to be replenished.

[0005] The present disclosure provides an information processing device, an information processing method, and an information processing program that can improve customer satisfaction. [Means for solving the problem]

[0006] An information processing device of one embodiment includes a first reception unit that receives a trained model that has learned about a container, the food in the container, and the amount of food in the container; a second reception unit that receives image information related to an image of the container containing food; an estimation unit that estimates the amount of food in the container recorded in the image information based on the trained model received by the first reception unit and the image information received by the second reception unit; and an output control unit that controls a predetermined output to be performed according to the amount of food estimated by the estimation unit. [Effects of the Invention]

[0007] According to one aspect, the amount of food in a container recorded in image information is estimated based on a trained model that has learned about the container, the food in the container, and the amount of food in the container, and image information, and an output is provided according to the estimated amount of food, thereby improving customer satisfaction. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an information processing device according to an embodiment. [Figure 2] FIG. 1 is a block diagram illustrating an information processing device according to an embodiment. [Figure 3] FIG. 1 is a first diagram illustrating an example of characters, images, etc. displayed on a display unit as an example of a predetermined output. [Figure 4] FIG. 2 is a second diagram illustrating an example of characters, images, etc. displayed on the display unit as an example of a predetermined output. [Figure 5] 10A and 10B are diagrams for explaining modified examples of predetermined outputs, in which (A) shows an example of an image when displaying the time count, and (B) shows an example of an image when displaying a container to be disposed of. [Figure 6] 1 is a flowchart illustrating an information processing method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] An embodiment will be described below.

[0010] [Overview of information processing device 1] First, an overview of an information processing device 1 according to an embodiment will be described. FIG. 1 is a diagram illustrating an information processing device 1 according to an embodiment.

[0011] The information processing device 1 may be configured, for example, as a recognition device that recognizes food in the container 101. The information processing device 1 may be configured, for example, as an estimation device that estimates the amount of food in the container 101. The information processing device 1 may be configured, for example, as an output device that performs output according to the amount of food in the container 101. The information processing device 1 may be configured, for example, as a replenishment instruction device that instructs the replenishing of food according to the amount of food in the container 101. The information processing device 1 may be a computer such as a server, a desktop, a laptop, a tablet, or a smartphone.

[0012] The information processing device 1 may be installed in, for example, a store such as a restaurant, a management headquarters that manages the store, etc. The restaurant may provide food to customers in a buffet format, or may provide beverages and the like to customers in a self-service drink bar format.

[0013] The imaging unit 21 captures images of the above-described exemplary food and beverage (hereinafter referred to as food) to generate image information. The imaging unit 21 may capture images of the food continuously over time, or may capture images of the food at predetermined intervals. An example of the predetermined interval may be various timings, such as every predetermined time. The imaging unit 21 may generate image information by capturing, for example, a moving image or a still image.

[0014] The information processing device 1 receives image information from the imaging unit 21 and estimates the amount of food recorded in the image information. In this case, the information processing device 1 may estimate the amount of food recorded in the image information, for example, using a trained model that has trained on the container 101, the food in the container 101, and the amount of food in the container 101. The information processing device 1 performs a predetermined output depending on the estimated amount of food. For example, when the amount of food is relatively small, the information processing device 1 may output, as the predetermined output, a message 201 instructing the user to replenish the food. For example, when the amount of food is relatively large, the information processing device 1 may output, as the predetermined output, a message 202 instructing the user that the food does not need to be replenished.

[0015] As a specific example, the trained model described above may be a model that has learned multiple stages of replenishment, such as (1) still safe to replenish, (2) needing replenishment soon, and (3) needing replenishment immediately, depending on the container 101, the food in the container 101, and the amount of food in the container 101. In this case, the information processing device 1 may estimate, based on the trained model and the image information, which of the multiple stages of replenishment the food recorded in the image information falls into as the amount of food.

[0016] Alternatively, as a specific example, the trained model described above may be a model that has learned the container 101, the food in the container 101, and the amount of food in the container 101. In this case, the information processing device 1 may estimate the amount of food recorded in the image information as the amount of food, based on the trained model and the image information.

[0017] [Details of information processing device 1] Next, the information processing device 1 according to an embodiment will be described in detail. FIG. 2 is a block diagram illustrating an information processing device 1 according to an embodiment.

[0018] The information processing device 1 includes, for example, an imaging unit 21, a communication unit 22, a storage unit 23, a display unit 24, and a control unit 11. The communication unit 22, the storage unit 23, and the display unit 24 may be an embodiment of an output unit. The control unit 11 includes, for example, a first reception unit 12, a second reception unit 13, a recognition unit 14, an estimation unit 15, and an output control unit 16. The control unit 11 may be configured, for example, by an arithmetic processing device of the information processing device 1. The control unit 11 (for example, an arithmetic processing device) may realize the functions of each unit (for example, the first reception unit 12, the second reception unit 13, the recognition unit 14, the estimation unit 15, and the output control unit 16) by, for example, appropriately reading and executing various programs stored in the storage unit 23.

[0019] The imaging unit 21 may be, for example, a camera. The imaging unit 21 generates image information by, for example, capturing an image of food. Specifically, the imaging unit 21 generates image information by, for example, capturing an image of food (food in containers) served at a facility. For example, when food is served buffet-style at a facility, the imaging unit 21 may be disposed above a table or the like on which one or more containers are placed. The imaging unit 21 may capture an image of, for example, multiple containers.

[0020] Furthermore, one or more imaging units 21 may be arranged above the food. Specifically, one or more imaging units 21 may be arranged, for example, above a table on which the food is placed. When multiple imaging units 21 are arranged, each imaging unit 21 may image the same or different containers. As an example, when two imaging units 21 are arranged, one imaging unit 21 may image one or more containers arranged on the right side of the table, and the other imaging unit 21 may image one or more containers arranged on the left side of the table. In this example, furthermore, at the boundary (near the center of the table) between the imaging ranges of one imaging unit 21 and the other imaging unit 21, one imaging unit 21 and the other imaging unit 21 may or may not image the same container.

[0021] The imaging unit 21 may generate image information by capturing a moving or still image of the container. For example, when capturing a still image of the container, the imaging unit 21 may capture an image of the container at predetermined intervals. An example of the predetermined intervals may be every predetermined time, or may be any other timing.

[0022] The communication unit 22 can transmit and receive various information to and from, for example, a device (external device) (not shown) outside the information processing device 1. The external device may be, for example, a server or a terminal. The terminal may be, for example, a desktop, a laptop, a tablet, a smartphone, or the like.

[0023] The storage unit 23 may store, for example, various information and programs. Examples of the storage unit 23 may be a memory, a solid state drive, a hard disk drive, etc. Note that the storage unit 23 may be, for example, a storage area or a server on a cloud.

[0024] The display unit 24 is capable of displaying, for example, various characters, symbols, images, and the like.

[0025] The first receiving unit 12 receives a trained model that has trained the container, the food in the container, and the amount of food in the container. The first receiving unit 12 may receive, for example, a trained model stored in the storage unit 23. The first receiving unit 12 may also receive the trained model from an external device (not shown) via the communication unit 22 and an external storage device (e.g., an external memory) (not shown). The external device may be, for example, a server that stores the trained model and a learning device (not shown) that generates the trained model. Furthermore, the first receiving unit 12 may receive, for example, a trained model generated by a learning unit (learning device) (not shown) inside the information processing device 1. In this case, the learning unit may be arranged, for example, in the control unit 11 or the like, or may be arranged outside the control unit 11 (for example, in a calculation processing device different from the control unit 11). In this case, the learning unit generates the trained model described above.

[0026] The containers used for learning may be various containers, such as bowls, plates, pots, cups, and cups, used to hold food served at facilities such as restaurants and hotels. Here, the facilities may provide food in a buffet format, or beverages in a self-service drink bar format. A self-service drink bar may also be referred to as a self-service drink area, drink station, or the like.

[0027] The food used for learning may be, for example, various dishes served at the facility, such as salad. As a specific example in this case, if the food is salad, the type of food may be ingredients used in the salad (e.g., lettuce, tomato, cucumber, etc.). That is, the food used for learning may be, for example, a type of food. The food may also be various dishes (ingredients) served in buffet style, such as sausages, fried eggs, cake, miso soup, and soup. The food may also be, for example, beverages such as juice, milk, and coffee, or portion packs of natto, seaweed, tea bags, sugar, milk, etc. The food used for learning may be capable of recognizing the type of food, for example, in order to recognize the type of food in a container.

[0028] The amount of food used for learning may be, for example, the actual amount in a container, or may be an amount obtained by dividing the amount of food in a container into multiple stages (e.g., large, medium, small, etc.). The actual amount in a container may be, for example, the amount (large or small) of food contained in a container. The amount of food may also be set in multiple stages related to replenishment, such as (1) still OK to replenish, (2) needing replenishment soon, and (3) needing replenishment immediately, depending on the amount of food in the container. In other words, the amount of food may be set in multiple stages related to replenishment, such as whether replenishment is necessary or not, depending on the food in the container. In other words, the stages used for learning are not limited to the three stages (1) to (3) described above, but may be two stages or four or more stages.

[0029] The trained model may include a model that has learned the amount of food in a container and the amount of food in the container. The trained model may also include a model that has learned the amount of food in a container. This enables the estimation unit 15, which will be described later, to estimate the amount of food. The trained model may also include a model that has learned about containers and the amount of each container. Furthermore, the trained model may also include a model that has learned about the amount of each container. This enables the estimation unit 15, which will be described later, to estimate the amount of various containers (e.g., bowls, plates, pots, cups, and cups) provided in a buffet or self-service drink bar format.

[0030] The second reception unit 13 receives image information related to an image of a container containing food. The image information received by the second reception unit 13 may record multiple containers containing food. The second reception unit 13 may receive image information from the imaging unit 21, for example. In this case, the second reception unit 13 may receive image information from the imaging unit 21 via a dedicated interface, or may receive image information from the imaging unit 21 via the communication unit 22, or may receive image information via an external storage device. Furthermore, the second reception unit 13 may receive image information stored in a server, for example, via the communication unit 22.

[0031] The recognition unit 14 recognizes the food in the container based on the image information. That is, the recognition unit 14 may recognize the type of food in the container based on the image information received by the second reception unit 13. The recognition unit 14 may recognize the type of food in the container, for example, by using a trained model received by the first reception unit 12. As a specific example, the recognition unit 14 may recognize the type of salad as the food, that is, the ingredients used in the salad (for example, lettuce, tomato, cucumber, etc.).

[0032] The estimation unit 15 estimates the amount of food in a container recorded in the image information based on the trained model received by the first reception unit 12 and the image information received by the second reception unit 13. That is, the estimation unit 15 estimates the amount of food in a container recorded in the image information, for example, using the trained model. For example, when multiple containers are recorded in the image information, the estimation unit 15 may estimate the amount of food placed in each of the multiple containers. In this case, the estimation unit 15 may estimate the amount of food in the container based on the type of food in the container recognized by the recognition unit 14. That is, the estimation unit 15 may estimate the amount of food according to the type of food in the container recognized by the recognition unit 14. In other words, the estimation unit 15 may identify the food in the container using the recognition result (type of food) of the food in the container, and estimate the amount of food in the container (identified food). The estimation unit 15 may function as a recognition estimation unit that includes the functions of the above-described recognition unit 14. That is, the recognition unit 14 and the estimation unit 15 may be integrated into one function.

[0033] Here, the estimation unit 15 may, for example, use a trained model to estimate one of multiple stages (amount) of replenishment status, such as (1) it is still okay to replenish, (2) replenishment is needed soon, or (3) replenishment is needed immediately, depending on the amount of food in the container recorded in the image information. Alternatively, the estimation unit 15 may estimate the amount of food in a container recorded in the image information (for example, the actual amount in the container) by using, for example, a trained model.

[0034] For example, when a container is imaged using multiple imaging units 21, if the same container is recorded in each of the image information generated by the multiple imaging units 21, the estimation unit 15 may estimate the amount of food using any one of the image information. Alternatively, the estimation unit 15 may, for example, use each of multiple pieces of image information to estimate the amount of food in the same container and identify the average of the estimated results as the amount of food, or may identify the smaller of the estimated results as the amount of food. In this case, the recognition unit 14 may estimate whether the containers are the same or not by performing recognition using each of the multiple pieces of image information, for example.

[0035] The output control unit 16 controls to perform a predetermined output in accordance with the amount of food estimated by the estimation unit 15. The predetermined output may be, for example, an output in accordance with the amount of food estimated by the estimation unit 15, regarding whether or not food needs to be replenished. As an example, the predetermined output may be an output regarding one of multiple replenishment stages, such as (1) it is still okay to replenish, (2) replenishment is needed soon, and (3) replenishment is needed immediately, in accordance with the amount of food estimated by the estimation unit 15. That is, the output control unit 16 may use, for example, text, images, audio, etc. as the predetermined output to output one of multiple stages, such as (1) it is still okay to replenish, (2) replenishment is needed soon, and (3) replenishment is needed immediately, in accordance with the amount of food estimated by the estimation unit 15.

[0036] FIG. 3 is a first diagram illustrating an example of characters, images, etc. displayed on the display unit 24 as an example of a predetermined output. FIG. 4 is a second diagram illustrating an example of characters, images, etc. displayed on the display unit 24 as an example of a predetermined output.

[0037] The output control unit 16 may, for example, as a predetermined output, display an image (one or more containers 101 (food)) based on the image information received by the second receiving unit 13 (see FIGS. 3 and 4), and may also control the display unit 24 to associate text, an image, etc. corresponding to the amount of food in each container 101 with that container 101, i.e., to display the text, an image, etc. relatively close to that container 101. In this case, the memory unit 23 may, for example, store correspondence information that associates the amount of food with text, an image, etc. The output control unit 16 may, for example, refer to the correspondence information stored in the memory unit 23 to display text, an image, etc. corresponding to the amount of food in the container 101 on the display unit 24.

[0038] In the example shown in Figure 3, since there is a sufficient amount of food in each of the multiple containers 101 and no replenishment is necessary, the output control unit 16 controls the display unit 24 to display a smiling image 203 as text and an image corresponding to the amount of food in the container 101, in association with that container 101.

[0039] In the example shown in FIG. 4, when the amount of food in the container 101 (101A) is relatively low and it will soon be necessary to replenish the food, the output control unit 16 controls the display unit 24 to display, in association with the container 101 (101A), an image 204 of a stern-looking face as text and an image corresponding to the amount of food in the container 101 (101A). In this case, the output control unit 16 may, for example, control the display unit 24 to extract from the image information an image of the container 101 (101A) in which the amount of food in the container 101 (101A) is relatively low and it will soon be necessary to replenish the food, and to display this image separately (see reference numeral 211). The output control unit 16 may, for example, control the display unit 24 to display the extracted image 211 together with text 212 indicating that replenishment is necessary.

[0040] 4, when the amount of food in the container 101 (101B) becomes relatively low and immediate replenishment is required, the output control unit 16 controls the display unit 24 to display, in association with the container 101 (101B), an image 205 of a very stern-looking face as text and an image corresponding to the amount of food in the container 101 (101B). In this case, the output control unit 16 may, for example, control the display unit 24 to extract from the image information an image of the container 101 (101B) in which the amount of food in the container 101 (101B) becomes relatively low and immediate replenishment of food is required, and to display the image separately (see reference numeral 213). The output control unit 16 may, for example, control the display unit 24 to display the extracted image 213 together with text 214 indicating that urgent replenishment is required. In this case, the output control unit 16 may display the image 213 (separately extracted image) of the container 101 (101B) containing the food that needs to be replenished immediately in a manner that indicates a higher priority for replenishment than the image 211 (separately extracted image) of the container 101 (101A) containing the food that will soon need to be replenished. In the example shown in FIG. 4, the output control unit 16 may display the image 213 of the container 101 (101B) containing the food that needs to be replenished immediately on the left side of the upper row of the display unit 24, and the image 211 of the container 101 (101A) containing the food that will soon need to be replenished on the right side of the upper row of the display unit 24. Furthermore, the output control unit 16 may, for example, change the appearance of the text 214 that reads "Hurry and replenish" and the text 212 that reads "Replenish." As an example, the output control unit 16 may display the text 214 that reads "Hurry and replenish" in red and the text 212 that reads "Replenish" in yellow. Furthermore, the output control unit 16 may not only change the color of the characters, but also highlight the characters and change the color of the highlight.

[0041] That is, as an example of a predetermined output, when the output control unit 16 controls the display unit 24 to display an image based on image information (an image showing the container 101), the output control unit 16 may also control the display unit 24 to display an additional image relating to the amount of food in the container 101 (in the example shown in FIGS. 3 and 4, facial images 203, 204, 205 according to the amount of food). The facial images 203, 204, 205 shown in FIGS. 3 and 4 may not only change their facial expressions according to the amount of food, but also change their facial color according to the amount of food. As the predetermined output, the output control unit 16 may perform control to output an image of the container 101 based on the image information received by the second receiving unit 13 in association with an image (face images 203, 204, 205) corresponding to the amount of food in the container 101 estimated by the estimation unit 15. The association may be, for example, content such that the face images 203, 204, 205 are displayed relatively close to the container 101. Relatively close may be within a predetermined distance from the position of the container 101 in the image.

[0042] Alternatively, the output control unit 16 may output, for example, the amount of food estimated by the estimation unit 15 (the actual amount of food) as the predetermined output. The output control unit 16 may, for example, use text, images, audio, or the like as the predetermined output to output an amount of food estimated by the estimation unit 15. As one example, the output control unit 16 may output the amount of food estimated by the estimation unit 15 (the amount of food (percentage, etc.) relative to 100%), with 100% being the amount when the container is filled with food. In this case, the output control unit 16 may control to change the predetermined output based on, for example, one or more thresholds related to the amount of food set for each container depending on the food. That is, the output control unit 16 may set, for example, one or more thresholds related to whether replenishment is necessary, identify one of multiple levels related to whether replenishment is necessary based on the estimated amount of food for each container and the threshold, and output the identified level as the predetermined output. As an example of this, the output control unit 16 may change the output content, such as text, images, and sound, depending on the identified level.

[0043] As one example, when the amount of food in the container estimated by the estimation unit 15 is less than a predetermined threshold, the output control unit 16 may control the output to output a warning or alert regarding the amount of food as a predetermined output. That is, as a specific example, the output control unit 16 may output a warning regarding the amount of food based on multi-stage amount thresholds such as (1) still OK to replenish, (2) needing replenishment soon, and (3) needing replenishment immediately, and the amount of food estimated by the estimation unit 15, if the amount of food estimated by the estimation unit 15 corresponds to (2) needing replenishment soon. Similarly, the output control unit 16 may compare the amount of food estimated by the estimation unit 15 with the multi-stage amount thresholds (1) to (3) described above, and output a warning regarding the amount of food if the amount of food estimated by the estimation unit 15 corresponds to (3). Note that the thresholds are not limited to the three stages (1) to (3) described above, and may be, for example, various values ​​used to determine whether replenishment is necessary. As an example, the predetermined threshold may be various values ​​at which the food amount is set to require replenishment, such as 20%, 30%, 40% and 50%.

[0044] Furthermore, when the amount of food in a container estimated by the estimation unit 15 is less than a predetermined threshold, the output control unit 16 may control the display unit 24 to output an image of the container as a predetermined output. That is, when the amount of food is less than the predetermined threshold, the output control unit 16 may control the display unit 24 to separately display an image of the container in which the food is placed. As a specific example, when the actual amount of food corresponds to either (2) or (3) above, the output control unit 16 may extract and separately display an image of the container, as in the example shown in FIG. 4.

[0045] For example, when multiple containers are recorded in the image information, the output control unit 16 may perform the predetermined output described above for each of the multiple containers. That is, as illustrated in Figures 3 and 4, when multiple containers 101 are recorded in the image information, the output control unit 16 may display facial images 203, 204, and 205 corresponding to the amount of food in each of the multiple containers 101. Furthermore, the output control unit 16 may perform a predetermined output according to, for example, the type of food recognized by the recognition unit 14 and the amount of the food estimated by the estimation unit 15. That is, even when the recognition unit 14 recognizes a food and the estimation unit 15 estimates the amount of the food, the output control unit 16 may perform the above-mentioned predetermined output according to the estimated amount.

[0046] The output control unit 16 may control the output unit to perform the predetermined output described above, for example. The output unit may be, for example, a communication unit 22, a storage unit 23, a display unit 24, etc. As an example, the output control unit 16 may control the communication unit 22 to transmit an image (image related to a predetermined output) (image information) based on the amount estimated by the estimation unit 15 to an external device (not shown). The external device may be, for example, a server or a user terminal. The user terminal is a terminal used by a user of the information processing device 1, and may be, for example, a desktop, a laptop, a tablet, a smartphone, or the like. As another example, the output control unit 16 may control the storage unit 23 to store an image based on the amount estimated by the estimation unit 15 (an image related to a predetermined output) (image information) or the like. As another example, the output control unit 16 may control the display unit 24 to display an image based on the amount estimated by the estimation unit 15 (an image related to a predetermined output) (image information) or the like.

[0047] Furthermore, the information processing device 1 may include, for example, a speaker (not shown) as an output unit. As an example, the output control unit 16 may control the speaker to output at least one of a sound and a voice based on the amount estimated by the estimation unit 15. That is, for example, when the amount of food corresponds to either (2) Refilling required soon or (3) Refilling immediately, the output control unit 16 may control to output a sound and a voice indicating that the amount of food corresponds to either (2) or (3) together with or instead of the image display described above. For example, when the amount of food corresponds to (2), the output control unit 16 may control to output at least one of a caution sound and a caution voice. For example, when the amount of food corresponds to (3), the output control unit 16 may control to output at least one of a warning sound and a warning voice that are different from the caution sound and the caution voice.

[0048] [Variations] 5A and 5B are diagrams for explaining modified examples of predetermined outputs, in which Fig. 5A shows an example of an image when displaying the measured time, and Fig. 5B shows an example of an image when displaying a container to be disposed of.

[0049] In addition, when the output control unit 16 controls the output unit to perform a predetermined output as described above (control 1), the estimation unit 15 may then estimate, for example, based on image information received by the second receiving unit 13, whether the amount of food in the container has increased compared to before control 1 was performed. Alternatively, the estimation unit 15 may estimate whether the amount of food in the container has increased, for example, based on image information received at any time by the second reception unit 13. In this case, the estimation unit 15 may estimate whether the amount of food in the container has increased compared to the immediately preceding timing or the timing a predetermined time ago. For example, the estimation unit 15 may estimate whether the amount of food in a container has increased by estimating the amount of food in the container recorded in the image information based on the trained model received by the first reception unit 12 and the image information received by the second reception unit 13. For example, when multiple dishes are served, the estimation unit 15 may perform estimation for each dish.

[0050] In this case, the output control unit 16 may start timing from the timing when it is estimated by the estimation unit 15 that the amount of food in the container has increased. In this case, as illustrated in Fig. 5(A), the output control unit 16 may control the display unit 24 to display an image 206 relating to the amount of food in the container 101 (an image similar to the facial images 203, 204, and 205 described with reference to Figs. 3 and 4) and a time measurement 221, in association with the container 101.

[0051] For example, the output control unit 16 may control the output unit to output a predetermined output instructing the user to discard the food when a predetermined time has elapsed since the start of timing. As illustrated in Fig. 5(B), the output control unit 16 may control the display unit 24 to extract an image of the food (container 101 (101C)) for which a predetermined time has elapsed since the start of timing from the image information received by the second receiving unit 13 and separately display it (see reference numeral 215) as in the example illustrated in Fig. 4. For example, the output control unit 16 may control the display unit 24 to display the separately extracted image 215 with text 216 indicating that the food should be discarded.

[0052] In this case, the output control unit 16 may display, for example, the image (separately extracted image) 215 of the container 101 (101C) containing the food to be discarded in a manner that indicates that it has a higher priority than the image (separately extracted image) 213 of the container 101 (101B) containing the food that needs to be immediately replenished and the image (separately extracted image) 211 of the container 101 (101A) containing the food that will soon need to be replenished. As an example, the output control unit 16 may place the image 215 of the container 101 (101C) containing the food to be discarded further to the left in the upper row of the display unit 24 to indicate that it has a higher priority. Furthermore, the output control unit 16 may, for example, change the appearance of the characters "Discard" 216, the characters "Refill quickly" 214, and the characters "Refill" 212. As an example, the output control unit 16 may color the characters "Discard" 216 purple. Furthermore, the output control unit 16 may not only change the color of the characters, but also highlight the characters and change the color of the highlighting. Furthermore, when it becomes necessary to discard food, for example, the output control unit 16 may change the expression of the facial image 206 illustrated in Fig. 5(A) and change the color of the face. As an example, when it becomes necessary to discard food, the output control unit 16 may change the facial expression to indicate that it needs to be discarded and change the color of the face to purple. Furthermore, for example, when it becomes necessary to discard food, the output control unit 16 may control the speaker to output a warning sound and a warning voice instructing the user to discard the food.

[0053] For example, if the estimation unit 15 estimates that the amount of food has been newly increased before the predetermined time has elapsed after starting the timekeeping as described above, the output control unit 16 may reset the timekeeping and start the timekeeping from the timing when the amount of food has been newly increased. For example, when multiple dishes are served, the output control unit 16 may perform timekeeping and output for each dish. Here, the predetermined time may be, for example, the time it takes to discard the food, or a different time may be set for each of the multiple dishes, or the same time may be set for each of the multiple dishes.

[0054] Note that the output control unit 16 may not only start the timer based on the estimation result of the estimation unit 15, but may also reset and start the timer when an operation to start the timer (a timer start operation) is performed by a facility employee or the like. The timer start operation may be, for example, an operation of touching an image (a face image in the example shown in FIGS. 3, 4, and 5(A)) corresponding to the amount of food displayed in association with the container to be replenished.

[0055] [Information processing method] Next, an information processing method according to an embodiment will be described. FIG. 6 is a flowchart illustrating an information processing method according to an embodiment.

[0056] In step ST101, the second receiving unit 13 receives image information relating to an image of a container containing food. The image information may include a plurality of containers containing food. The image information may be generated by the imaging unit 21.

[0057] In step ST102, the recognition unit 14 recognizes the food in the container based on the image information received in step ST101. In this case, the recognition unit 14 may recognize, for example, the type of food. As another example, the recognition unit 14 may recognize the type of food in a container using a trained model received by the first reception unit 12. The first reception unit 12 may receive a trained model that has learned the container, the food in the container, and the amount of food in the container.

[0058] In step ST103, the estimation unit 15 estimates the amount of food in the container recorded in the image information based on the trained model received by the first reception unit 12 and the image information received in step ST101. Here, for example, if multiple containers are recorded in the image information, the estimation unit 15 may estimate the amount of food contained in each of the multiple containers. Furthermore, the estimation unit 15 may estimate the amount of food according to the type of food in the container recognized in step ST102.

[0059] For example, the estimation unit 15 may estimate the amount of food at any one of multiple stages of replenishment status, such as (1) still able to replenish, (2) needing replenishment soon, (3) needing replenishment immediately, etc. The multiple stages are not limited to three stages, and may be two stages or four or more stages. Alternatively, the estimation unit 15 may estimate, for example, the amount of food in a container recorded in the image information (for example, the actual amount in the container, etc.).

[0060] In step ST104, the output control unit 16 controls to perform a predetermined output according to the amount of food estimated in step ST103. For example, if multiple containers are recorded in the image information, the output control unit 16 may perform the predetermined output described above for each of the multiple containers. For example, the output control unit 16 may perform a predetermined output according to the type of food recognized in step ST102 and the amount of the food estimated in step ST103. The output control unit 16 may, for example, control the output to correspond to an image of the container based on the image information received in step ST101 and an image as a predetermined output corresponding to the amount of food in the container estimated in step ST103. The output control unit 16 may control to change the predetermined output based on, for example, one or more thresholds related to the amount of food set for each container depending on the food. For example, when the amount of food in the container estimated in step ST103 is less than the predetermined threshold, the output control unit 16 may control to output a warning or alert related to the amount of food as the predetermined output. For example, when the amount of food in the container estimated in step ST103 is less than the predetermined threshold, the output control unit 16 may control to output an image of the container as the predetermined output.

[0061] Each unit of the information processing device 1 described above may be realized as a function of a computer's arithmetic processing device, etc. That is, the first reception unit 12, the second reception unit 13, the recognition unit 14, the estimation unit 15, and the output control unit 16 (control unit 11) of the information processing device 1 may be realized as a first reception function, a second reception function, a recognition function, an estimation function, and an output control function (control function), respectively, by a computer's arithmetic processing device, etc. The information processing program can cause a computer to realize each of the above-described functions. The information processing program may be recorded on a computer-readable non-transitory recording medium, such as a memory, a solid-state drive, a hard disk drive, or an optical disk. As described above, each unit of the information processing device 1 may be realized by an arithmetic processing device of a computer or the like. The arithmetic processing device or the like is configured by, for example, an integrated circuit or the like. Therefore, each unit of the information processing device 1 may be realized as a circuit that constitutes the arithmetic processing device or the like. That is, the first reception unit 12, the second reception unit 13, the recognition unit 14, the estimation unit 15, and the output control unit 16 (control unit 11) of the information processing device 1 may be realized as a first reception circuit, a second reception circuit, a recognition circuit, an estimation circuit, and an output control circuit (control circuit) that constitute the arithmetic processing device of a computer or the like. The imaging unit 21, the communication unit 22, the storage unit 23, and the display unit 24 (output unit) of the information processing device 1 may be realized as, for example, an imaging function including the functions of an arithmetic processing device, as well as a communication function, a storage function, and a display function (output function). The imaging unit 21, the communication unit 22, the storage unit 23, and the display unit 24 (output unit) of the information processing device 1 may be realized as, for example, an imaging circuit, a communication circuit, a storage circuit, and a display circuit (output circuit) by being configured using integrated circuits, etc. The imaging unit 21, the communication unit 22, the storage unit 23, and the display unit 24 (output unit) of the information processing device 1 may be realized as, for example, an imaging device, a communication device, a storage device, and a display device (output device) by being configured using multiple devices.

[0062] The information processing device 1 can be configured by combining one or any combination of the above-described multiple units. In this disclosure, the term "information" is used, but the term "information" can be replaced with "data" and the term "data" can be replaced with "information."

[0063] In the above-described embodiment, the amount of food in the container is estimated, and whether or not food needs to be replenished is estimated based on the estimated amount of food. The present disclosure is not limited to the example of estimating the amount of food, but may also estimate the amount of products (products in containers or other receptacles) provided at various facilities and estimate whether product replenishment is necessary based on the estimated product amount. In this case, the information processing device may output whether product replenishment is necessary based on the estimated product amount. As an example, if it is estimated that product replenishment is necessary, the information processing device may output an indication that product replenishment is necessary.

[0064] [Aspects and Effects of the Present Embodiment] Next, one aspect of this embodiment and the effects of each aspect will be described. Note that each aspect described below is an example at the time of filing, and this embodiment is not limited to those described below. In other words, this embodiment is not limited to each aspect described below, and may be realized by appropriately combining the above-mentioned parts. Also, the effects described below are an example, and the effects achieved by each aspect are not limited to those described below. Also, each aspect may achieve, for example, at least one of the effects described below.

[0065] (Aspect 1) An information processing device of one embodiment includes a first reception unit that receives a trained model that has learned about a container, the food in the container, and the amount of food in the container; a second reception unit that receives image information related to an image of the container containing food; an estimation unit that estimates the amount of food in the container recorded in the image information based on the trained model received by the first reception unit and the image information received by the second reception unit; and an output control unit that controls a predetermined output to be performed according to the amount of food estimated by the estimation unit. As a result, the information processing device can output a predetermined amount of food in the container, for example, and when the amount of food in the container becomes relatively low, can notify at least one of the amount of food and the need to replenish the food. In other words, the information processing device can, for example, prompt the user to replenish the food in the container. Furthermore, the information processing device can reduce the possibility of buffet-style food running out by, for example, having an employee of a facility such as a restaurant replenish food in response to a notification that the amount of food in the container is relatively low. Therefore, the information processing device can prevent a decrease in customer satisfaction at the facility. In other words, the information processing device can allow customers at a facility that provides food in buffet style to freely take food, thereby improving customer satisfaction.

[0066] (Aspect 2) In one embodiment of the information processing device, the image information received by the second receiving unit records multiple containers containing food, the estimation unit estimates the amount of food contained in each of the multiple containers, and the output control unit performs a predetermined output for each of the multiple containers. The information processing device outputs predetermined information regarding the amount of food that can fit in each of the multiple containers, thereby reducing the possibility that at least one of the dishes will run out even when multiple dishes are served at a facility, thereby improving customer satisfaction.

[0067] (Aspect 3) An information processing device according to one embodiment may include a recognition unit that recognizes the type of food in the container based on image information received by the second reception unit, an estimation unit that estimates the amount of food according to the type of food in the container recognized by the recognition unit, and an output control unit that performs a predetermined output according to the type of food recognized by the recognition unit and the amount of food estimated by the estimation unit. This allows the information processing device to estimate the amount of food in the container depending on the type of food.

[0068] (Aspect 4) In one embodiment of the information processing device, the output control unit may be configured to control the output of an image of the container based on the image information received by the second receiving unit in association with an image as a predetermined output corresponding to the amount of food in the container estimated by the estimation unit. As a result, when the amount of food in the container becomes low, i.e., when it becomes necessary to replenish the food, the information processing device can output, as a predetermined output, a message indicating that it is necessary to replenish the food.

[0069] (Aspect 5) In one aspect of the information processing device, the output control unit may control to change the predetermined output based on one or more threshold values ​​related to the amount of food set for each container depending on the food. This allows the information processing device to determine, for example, based on the amount of food compared to a threshold value, whether food needs to be replenished, and if food needs to be replenished, whether it needs to be replenished immediately or whether there is a relative amount of time before the food needs to be replenished.

[0070] (Aspect 6) In one embodiment of the information processing device, the output control unit may be controlled to output a warning or caution regarding the amount of food as a predetermined output when the amount of food in the container estimated by the estimation unit is less than a predetermined threshold. This allows the information processing device to determine whether there is enough time before the food needs to be replenished (for example, it is about time to replenish) as a warning output, or whether the food needs to be replenished immediately as a warning output, and output the result of that determination.

[0071] (Aspect 7) In one embodiment of the information processing device, the output control unit may be configured to control the output of an image of the container as a predetermined output when the amount of food in the container estimated by the estimation unit is less than a predetermined threshold. The information processing device separately outputs an image of a container containing the food estimated to need replenishment. As an example, the information processing device separately displays an image of the container containing the food estimated to need replenishment on a display unit. This allows the information processing device to clearly indicate to facility employees and the like that the food needs to be replenished, and also to show the status of the food in the container.

[0072] (Aspect 8) In one aspect of the information processing method, a computer executes a first reception step of receiving a trained model that has learned about a container, the food in the container, and the amount of food in the container; a second reception step of receiving image information relating to an image of the container containing food; an estimation step of estimating the amount of food in the container recorded in the image information based on the trained model received in the first reception step and the image information received in the second reception step; and an output control step of controlling to produce a predetermined output according to the amount of food estimated in the estimation step. As a result, the information processing method may achieve the same effect as the information processing device of the above-described aspect.

[0073] (Aspect 9) An information processing program of one embodiment causes a computer to realize the following: a first reception function that receives a trained model that has learned about a container, the food in the container, and the amount of food in the container; a second reception function that receives image information related to an image of the container containing food; an estimation function that estimates the amount of food in the container recorded in the image information based on the trained model received by the first reception function and the image information received by the second reception function; and an output control function that controls a predetermined output to be performed according to the amount of food estimated by the estimation function. As a result, the information processing program may achieve the same effect as the information processing device of the above-described aspect. [Explanation of symbols]

[0074] 1. Information processing equipment 11 Control section 12 First Reception Section 13 Second Reception Section 14 Recognition part 15 Estimation part 16 Output control section 21 Imaging unit 22 Communications Department 23 Memory section 24 Display

Claims

1. A first reception unit that receives a trained model that has learned the amount of food in a container; A second reception unit that receives image information relating to an image of a container in which food is placed; An estimation unit that estimates an amount of food in a container recorded in image information based on the trained model accepted by the first acceptance unit and the image information accepted by the second acceptance unit; an output control unit that controls to perform a predetermined output in accordance with the amount of food estimated by the estimation unit; An information processing device comprising:

2. The image information received by the second reception unit includes a plurality of containers containing food, The estimation unit estimates an amount of food contained in each of a plurality of containers, The output control unit performs a predetermined output for each of the plurality of containers. The information processing device according to claim 1 .

3. A recognition unit that recognizes the type of food in the container based on the image information received by the second reception unit, The estimation unit estimates an amount of food in the container according to the type of food recognized by the recognition unit, The output control unit performs a predetermined output in accordance with the type of food recognized by the recognition unit and the amount of the food estimated by the estimation unit.

3. The information processing device according to claim 1 or 2.

4. The output control unit controls to output an image of a container based on the image information received by the second reception unit in association with an image as a predetermined output according to the amount of food in the container estimated by the estimation unit.

4. The information processing device according to claim 1.

5. The output control unit controls to change the predetermined output based on one or more threshold values ​​related to the amount of food set for each container according to the food. The information processing device according to any one of claims 1 to 4.

6. The output control unit controls to output, as the predetermined output, a warning or an alert regarding the amount of food in the container when the amount of food estimated by the estimation unit is less than a predetermined threshold value. The information processing device according to claim 5 .

7. The output control unit controls the device to output an image of the container as a predetermined output when the amount of food in the container estimated by the estimation unit is less than a predetermined threshold.

7. The information processing device according to claim 5 or 6.

8. The computer A first reception step of receiving a trained model that has learned the amount of food in a container; A second reception step of receiving image information relating to an image of a container containing food; an estimation step of estimating an amount of food in a container recorded in the image information based on the trained model received in the first reception step and the image information received in the second reception step; an output control step of controlling a predetermined output in accordance with the amount of food estimated by the estimation step; An information processing method for performing the above.

9. On the computer, A first reception function that receives a trained model that has learned the amount of food in the container; A second reception function for receiving image information relating to an image of a container in which food is placed; An estimation function that estimates the amount of food in a container recorded in image information based on the trained model accepted by the first reception function and the image information accepted by the second reception function; an output control function that controls a predetermined output in accordance with the amount of food estimated by the estimation function; An information processing program that realizes this.

Images