Food product identification system, information terminal, food product identification method, display method, and program

Abstract

A food product identification system (10) comprises: an acquiring unit (54) that acquires an identification result, obtained by means of a machine learning model, of a food product shown in an image of the inside of a storage cabinet; and an output unit (55) that, when a candidate food product determined to have the highest probability of being the food product by means of the machine learning model, among a plurality of candidate food products indicated by the acquired identification result, is the same as a candidate food product presented in the past, outputs, on the basis of correction information indicating that the candidate food product presented in the past was corrected to a food product specified by the user, presentation information for presenting at least some of the plurality of candidate food products and the specified food product as options to the user.

Description

Food identification system, information terminal, food identification method, display method, and program

[0001] The present disclosure relates to a food identification system, an information terminal, a food identification method, a display method, and a program.

[0002] Conventionally, technologies relating to a storage cabinet for storing objects have been proposed. Patent Document 1 discloses a food management system that can accept registration of food items stored in a refrigerator by voice recognition.

[0003] Japanese Patent Application Laid-Open No. 2021-128556

[0004] In registering ingredients (food) as in the cited document 1, a method of identifying the food stored in a storage cabinet by image recognition processing may be considered.

[0005] The present disclosure provides a food identification system that can assist a user in correcting food identification results obtained through image recognition processing.

[0006] A food identification system according to one aspect of the present disclosure includes: a control unit that presents a user with a plurality of candidate foods indicated by a first identification result obtained by a machine learning model of a first food shown in a first image inside a storage cabinet having a cooling function for stored items as options; and, when a food not included in the options is designated as the user's identification result of the first food, stores correction information in a memory unit indicating that a top candidate food among the plurality of candidate foods that is determined by the machine learning model to have the highest probability of being the first food has been corrected to the designated food; an acquisition unit that acquires a second identification result obtained by the machine learning model of a second food shown in a second image inside the storage cabinet; and, when the candidate food that is determined by the machine learning model to have the highest probability of being the second food among the plurality of candidate foods shown in the acquired second identification result is the same as the top candidate food, outputs presentation information based on the correction information to present the user with at least some of the plurality of candidate foods indicated by the second identification result and the designated food as options.

[0007] A food identification system according to one aspect of the present disclosure can assist a user in correcting food identification results obtained through image recognition processing.

[0008] FIG. 1 is a block diagram showing the functional configuration of a food identification system according to an embodiment. FIG. 2 is an external view of a refrigerator and an image capture device provided in the food identification system according to an embodiment. FIG. 3 is a diagram showing an example of a list screen of the identification results of each of a plurality of foods. FIG. 4 is a diagram showing an example of a confirmation screen for details of the food identification results. FIG. 5 is a sequence diagram of an operation for displaying the food identification results. FIG. 6 is a sequence diagram of an operation example 1 for confirming the food identification results. FIG. 7 is a sequence diagram of an operation example 2 for confirming the food identification results. FIG. 8 is a diagram showing another example of a confirmation screen for details of the food identification results.

[0009] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection forms, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.

[0010] It should be noted that the drawings are schematic diagrams and are not necessarily strict illustrations. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and overlapping descriptions may be omitted or simplified.

[0011] (Embodiment) [Configuration] First, the configuration of a food identification system according to an embodiment will be described. Fig. 1 is a block diagram showing the functional configuration of a food identification system according to an embodiment.

[0012] The food identification system 10 shown in Figure 1 is a system that can manage the types and quantities of food stored in a refrigerator 20 by identifying the types of food stored in the refrigerator 20 from images of the inside of the refrigerator 20. Specifically, the food identification system 10 includes a refrigerator 20, an image capture device 30, an image recognition server 40, a server device 50, and an information terminal 60.

[0013] The refrigerator 20 is an example of a storage unit capable of refrigerating stored items, and is installed in a user's home or the like to refrigerate food. Fig. 2 is an external view of the refrigerator 20 (and the image capturing device 30).

[0014] The image capturing device 30 captures an image of a drawer (e.g., a vegetable drawer) of the refrigerator 20 in an open state from above the refrigerator 20. That is, the image capturing device 30 captures, for example, an image of the drawer of the refrigerator 20 viewed from above the refrigerator 20. The image capturing device 30 is realized, for example, by a camera having a wide-angle lens and a telephoto lens.

[0015] The image recognition server 40 is a computer located outside the facility where the refrigerator 20 is installed, and specifically, is a cloud server. The image recognition server 40 acquires images captured by the image capture device 30 by communicating with the image capture device 30 via the wide area communication network 70, and identifies the type of food shown in the acquired image. The type of food is, for example, a specific variety of vegetable (cucumber, tomato, etc.). In the following embodiment, an example of identifying the type of vegetable will be described, but the image recognition server 40 can also identify the type of vegetable, the type of meat, the type of beverage, etc.

[0016] Server device 50 is a computer located outside the facility where refrigerator 20 is installed, and specifically, is a cloud server. Server device 50 presents the food type identification results obtained by image recognition server 40 to the user via information terminal 60, and accepts instructions from the user to confirm or correct the presented identification results. Server device 50 includes a communication unit 51, an information processing unit 52, and a storage unit 53.

[0017] The communication unit 51 is a communication circuit that enables the server device 50 to communicate with the image recognition server 40 and the information terminal 60 via the wide area communication network 70. The communication unit 51 is, for example, a wired communication circuit that performs wired communication, but may also be a wireless communication circuit that performs wireless communication. There are no particular limitations on the communication standard used for communication by the communication unit 51.

[0018] The information processing unit 52 performs information processing and the like to present the food type identification results obtained by the image recognition server 40 to the user via the information terminal 60. The information processing unit 52 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. The information processing unit 52 has, as functional components, an acquisition unit 54, an output unit 55, and a control unit 56. The functions of the acquisition unit 54, the output unit 55, and the control unit 56 are realized, for example, by the microcomputer or the like constituting the information processing unit 52 executing a computer program stored in the storage unit 53. The functions of the acquisition unit 54, the output unit 55, and the control unit 56 will be described in detail below.

[0019] The storage unit 53 is a storage device that stores the computer programs executed by the information processing unit 52 and various information required for the information processing. The storage unit 53 is realized by, for example, a semiconductor memory.

[0020] The information terminal 60 is an information terminal owned by a user. The information terminal 60 displays the food type identification result obtained by the image recognition server 40 and accepts instructions from the user to confirm or correct the displayed identification result. The information terminal 60 is, for example, a portable information terminal such as a smartphone or a tablet terminal, but may also be a stationary information terminal such as a personal computer. The information terminal 60 includes an operation receiving unit 61, a display unit 62, an information processing unit 63, a storage unit 64, and a communication unit 65.

[0021] The operation reception unit 61 receives operations from the user. The operation reception unit 61 is realized by, for example, a touch panel, but may also be realized by a mouse and keyboard.

[0022] The display unit 62 displays display screens such as those shown in Fig. 3 and Fig. 4. Fig. 3 is a diagram showing an example of a list screen of the classification results of a plurality of foods by the image recognition server 40, and Fig. 4 is a diagram showing an example of a confirmation screen of the details of the classification result of a certain food by the image recognition server 40. In other words, the confirmation screen in Fig. 4 is a display screen for receiving instructions from the user to confirm or correct the classification result. The display unit 62 is realized by a display panel such as a liquid crystal panel or an organic EL (Electro-Luminescence) panel, for example.

[0023] The information processing unit 63 performs information processing and the like to display the display screens such as those shown in Figures 3 and 4. The information processing unit 63 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. The information processing unit 63 has, as functional components, an acquisition unit 66 and a display control unit 67. The functions of the acquisition unit 66 and the display control unit 67 are realized, for example, by the microcomputer or the like constituting the information processing unit 63 executing a computer program stored in the storage unit 64. The functions of the acquisition unit 66 and the display control unit 67 will be described in detail below.

[0024] The storage unit 64 is a storage device that stores the computer program executed by the information processing unit 63 and various information required for the information processing (such as the food management information described below). The storage unit 64 is realized, for example, by a semiconductor memory. In order to display the display screens of Figures 3 and 4, a predetermined application program is pre-installed in the storage unit 64, and the computer program executed by the information processing unit 63 includes such a predetermined application program.

[0025] The communication unit 65 is a communication circuit that enables the information terminal 60 to communicate with the server device 50 via the wide area communication network 70. The communication unit 65 is, for example, a wireless communication circuit that performs wireless communication, but may also be a wired communication circuit that performs wired communication. There are no particular limitations on the communication standard of the communication performed by the communication unit 65.

[0026] [Display Operation of Food Identification Results] Next, a description will be given of the operation of displaying food identification results on the information terminal 60, which is performed by the food identification system 10. Fig. 5 is a sequence diagram of the operation of displaying food identification results.

[0027] The image recognition server 40 communicates with the photographing device 30 to receive an image (image information) of the inside of the refrigerator 20 from the photographing device 30 (S11).

[0028] The image recognition server 40 performs object detection processing on the received image to detect food appearing in the image and identify the type of the detected food (S12). Specifically, a detection unit (not shown) in the image recognition server 40 uses a machine learning model (trained model) to detect objects (food) in the image and identify the detected food.

[0029] The food identification result is expressed, for example, by the type of food and an identification score. The identification score is a score that indicates the likelihood of the identification result and is expressed between 0 and 1, with larger values ​​indicating higher likelihood. Likelihood can also be referred to as validity, accuracy, or reliability.

[0030] The food classification results indicate, for example, n food types in descending order of classification score and the classification scores for each of the n foods. n is a natural number greater than or equal to 2, and in the following embodiments, n = 4. Specifically, the food classification results are information indicating the following: tomato: 0.4, broccoli: 0.2, lettuce: 0.15, pumpkin: 0.1. Below, the four foods included in the classification results are referred to as candidate foods, and the candidate food with the highest classification score among the candidate foods is referred to as the top candidate food.

[0031] The image recognition server 40 transmits to the server device 50, identification result list information indicating the top candidate food for each of the multiple foods shown in the image received in step S11 (S13). The identification result list information associates the ID of each of the multiple foods with the top candidate food.

[0032] The communication unit 51 of the server device 50 receives the identification result list information and transmits the received identification result list information to the information terminal 60 (S14).

[0033] The communication unit 65 of the information terminal 60 receives the identification result list information. The acquisition unit 66 acquires the identification result list information (S15), and the display control unit 67 displays an identification result list screen as shown in Fig. 3 on the display unit 62 based on the acquired identification result list information (S16).

[0034] The list screen in FIG. 3 displays a list of the classification results of five foods shown in the image, which are tomato, lettuce, potato, green onion, and broccoli.

[0035] In this way, the food identification system 10 can display the identification results for each of the multiple foods shown in the image on the display unit 62 of the information terminal 60.

[0036] [Operation Example 1 for Confirming Identification Results] The user confirms whether the identification results by the image recognition server 40 are correct for each of the five foods displayed on the list screen in Figure 3. The food identification system 10 confirms the identification results that the user determines to be correct, and corrects the identification results that the user determines to be incorrect in accordance with the user's instructions before confirming them. As a result, the inventory of food items in the refrigerator 20 is updated in the inventory management server (not shown). Operation Example 1 for confirming the identification results of food items will be described below. Figure 6 is a sequence diagram of Operation Example 1 for confirming the identification results of food items.

[0037] For example, the user performs a confirmation operation to check the details of the identification result for a certain food (hereinafter also referred to as the target food). For example, a confirmation operation to check the identification result for a food identified as a tomato is a tap operation on the word "Confirm" in the column for the food identified as a tomato. The operation accepting unit 61 accepts this confirmation operation (S17). In response to the accepted operation, the display control unit 67 transmits a request for details of the confirmation screen to the image recognition server 40 (S18). The request for details includes the ID of the target food.

[0038] The image recognition server 40 receives the detailed request and transmits detailed identification result information, which is a detailed identification result of the food having the ID included in the received detailed request (i.e., the target food), to the server device 50 (S19). Note that the detailed identification result information is information indicating the four candidate foods, and also includes the image etc. received in step S11.

[0039] The communication unit 51 of the server device 50 receives the identification result detailed information, and the acquisition unit 54 acquires the received identification result detailed information (S20). The output unit 55 communicates with the information terminal 60 using the communication unit 51, and outputs (transmits) to the information terminal 60 presentation information for presenting the four candidate foods indicated in the received identification result detailed information to the user as options (S21).

[0040] The communication unit 65 of the information terminal 60 receives the presentation information. The acquisition unit 66 acquires the presentation information (S22), and the display control unit 67 causes the display unit 62 to display a confirmation screen for the details of the identification result as shown in FIG. 4 based on the acquired presentation information (S23).

[0041] The image received in step S11 and included in the detailed identification result information is displayed at the top of the confirmation screen shown in Fig. 4, with a detection frame superimposed on the target food in the image. The detection frame is sometimes called a bounding box.

[0042] Furthermore, the four candidate foods indicated by the detailed identification result information are displayed as options at the bottom of the confirmation screen. In the example of FIG. 4 , the top candidate food of the four candidate foods is tomato, and the other three candidate foods are broccoli, lettuce, and pumpkin. As shown in FIG. 4 , the option (icon) of the top candidate food with the highest identification score is displayed, for example, in a larger display size than the other three candidate food options. The other three candidate food options are displayed in a horizontal row below the top candidate food option. The candidate food option with the second highest identification score is displayed on the left, the candidate food option with the third highest identification score is displayed in the center, and the candidate food option with the fourth highest identification score is displayed on the right.

[0043] If the user determines that the food in the detection frame at the top of the confirmation screen (i.e., the target food) does not correspond to any of the four candidate foods, the user selects "Enter text" on the confirmation screen in Figure 4 and performs a correction operation to manually input the user's identification result. In other words, the user specifies a food that is not included in the options as the user's identification result of the target food.

[0044] The operation receiving unit 61 receives the correction operation (S24), and the information processing unit 63 communicates with the server device 50 using the communication unit 65 to transmit correction information to the server device 50 (S25). The correction information is information indicating that the top candidate food (e.g., tomato) has been corrected to a food designated by the user (e.g., onion).

[0045] The communication unit 51 of the server device 50 receives the correction information, and the control unit 56 updates the correction history information stored in the storage unit 53 by adding the received correction information to the correction history information (S26). The server device 50 also confirms the identification result and notifies an inventory management server (not shown) that the food (onion) has been stored in the refrigerator 20 (not shown).

[0046] After the confirmation screen is displayed in step S23, if the user determines that the food in the detection box at the top of the confirmation screen corresponds to one of the four candidate foods, the user selects the corresponding candidate food option. In this case, information indicating that the food corresponding to the option selected by the user has been stored in refrigerator 20 is transmitted from information terminal 60 to server device 50. Server device 50 notifies the inventory management server (not shown) that the food corresponding to the option selected by the user has been stored in refrigerator 20.

[0047] In this way, in operation example 1, the information terminal 60 can reduce the effort required for the user to input the identification results by displaying the identification results of the target food based on the machine learning model as options.

[0048] Furthermore, if the identification result of the target food by the machine learning model does not match the identification result by the user, the user can manually input the identification result by the user. In other words, the user can correct the identification result by the machine learning model. How the identification result by the machine learning model was corrected is saved as correction history information. Using the correction history information, the following Operation Example 2 can be realized.

[0049] [Operation Example 2 for Confirming Identification Results] The machine learning model used by the image recognition server 40 identifies the food in the image as one of approximately 50 types of foods that are subject to identification. Therefore, the server device 50 cannot present foods that are not subject to identification, that is, foods that are not included in the approximately 50 types of foods, to the user as candidate foods. For foods that are not subject to identification, the user must manually input (specify) the food name from the confirmation screen in Figure 4 each time, which is time-consuming for the user. For example, if lotus root is subject to identification but nagaimo is not subject to identification, when nagaimo is stored in the refrigerator 20, there is a possibility that the nagaimo will be identified as lotus root every time.

[0050] Furthermore, if the machine learning model does not perform re-learning, the individual user's tendencies are not reflected in the machine learning model, and the same corrections may occur frequently. According to the inventors' studies, there are cases where spinach is identified as chives when it is shrunk and stored in the vegetable compartment, and users who tend to store spinach shrunk and stored in the vegetable compartment may be more likely to correct chives to spinach. Furthermore, according to the inventors' studies, there are cases where red onions are identified as tomatoes, and users who tend to prefer purchasing red onions may be more likely to correct tomatoes to onions.

[0051] Therefore, the food identification system 10 reduces the effort required for the user to input the identification results (correct the identification results obtained by the machine learning model) by displaying on the confirmation screen options based on the correction history information (correction information) described in step S26 of operation example 1, in addition to options for candidate foods based on the identification results of the machine learning model. Operation example 2 will now be described. Figure 7 is a sequence diagram of operation example 2 for confirming the food identification results.

[0052] The process up to the display of the list screen in Fig. 3 is the same as that in Fig. 5 , and therefore a detailed description thereof will be omitted. The user performs a confirmation operation to check the details of the identification result for the target food when the list screen in Fig. 3 is displayed, and the operation accepting unit 61 accepts this confirmation operation (S27). In response to the accepted confirmation operation, the display control unit 67 transmits a request for details of the confirmation screen to the image recognition server 40 (S28). The request for details includes the ID of the target food.

[0053] The image recognition server 40 receives the detailed request and transmits detailed identification result information, which is a detailed identification result of the food having the ID included in the received detailed request (i.e., the target food), to the server device 50 (S29). Note that the detailed identification result information is information indicating the four candidate foods, and also includes the image etc. received in step S11.

[0054] The communication unit 51 of the server device 50 receives the identification result detailed information, and the acquisition unit 54 acquires the received identification result detailed information (S30).

[0055] The output unit 55 determines whether or not there is correction information for the food that the machine learning model has determined to have the highest probability of being the target food among the four candidate foods indicated in the acquired detailed identification result information (S31) by referring to the correction history information stored in the storage unit 53. If it is determined that there is no correction information, the same processing as that from step S21 onward in Operation Example 1 is performed.

[0056] If the output unit 55 determines that correction information exists, it generates presentation information by replacing the candidate food with the lowest classification level among the four candidate foods indicated in the classification result detailed information with the correction target food indicated in the correction information (S32). The presentation information is information for presenting to the user as options some of the multiple candidate foods indicated in the classification result detailed information (the top three) and foods that the user previously specified as correction targets.

[0057] For example, it is conceivable that the detailed identification result information acquired in step S30 indicates tomato: 0.4, broccoli: 0.2, lettuce: 0.15, and pumpkin: 0.1, and the correction history information includes correction information indicating that tomato has been corrected to onion. In such a case, the output unit 55 determines in step S31 that correction information exists, and in step S32 generates presentation information for presenting a total of four foods as options: the three candidate foods of tomato, broccoli, and lettuce, and onion, which is the food previously designated by the user as a correction target.

[0058] If it is determined in step S31 that there is more than one piece of revision information, the most recent revision information (the revision information with the most recent revision date and time) is referenced. That is, the candidate food with the lowest classification level among the four candidate foods indicated by the classification result detailed information is replaced with the revised food indicated by the most recent revision information. Furthermore, in step S32, if a food that was previously designated as a revision target is included among the four candidate foods, presentation information is generated to present the four candidate foods indicated by the classification result detailed information as options, as in operation example 1.

[0059] After generating the presentation information as described above, the output unit 55 outputs (transmits) the generated presentation information to the information terminal 60 (S33).

[0060] The communication unit 65 of the information terminal 60 receives the presentation information. The acquisition unit 66 acquires the presentation information (S34), and the display control unit 67 displays a confirmation screen for the details of the identification result on the display unit 62 based on the acquired presentation information (S35). Figure 8 is a diagram showing another example of the confirmation screen for the details of the identification result.

[0061] In the example of Figure 8, in the position where the option for the lowest-ranked candidate food among the four candidate foods is displayed (the right position at the bottom), an option for the food (onion) that was previously specified by the user as the target for correction is displayed in place of the option for the lowest-ranked candidate food.

[0062] As shown in Figure 8, the top candidate food option (icon) with the highest identification score is displayed in a larger display size than, for example, the other two candidate food options and the option of a food previously designated as a revision destination. In other words, the display size of the top candidate food option is larger than the display size of the other candidate food options and the option of a food previously designated as a revision destination. Meanwhile, the display size of the other two candidate food options is the same as the display size of the option of a food previously designated as a revision destination. Furthermore, the number of other candidate food options (two) is greater than the number of food options previously designated as a revision destination (one).

[0063] Note that foods previously designated as targets for correction may be foods that are subject to identification by a machine learning model, or foods that are not subject to identification by a machine learning model. If a food previously designated as a target for correction is a food that is not subject to identification by a machine learning model, a photo (picture) of the food to be displayed in the option (icon) may not be displayed in the option (icon) of the food previously designated as a target for correction, because no photo (picture) of the food has been prepared.

[0064] In this way, in Operation Example 2, if the identification result of the target food by the machine learning model has been corrected by the user in the past, the information terminal 60 displays the food to be corrected as an option for the identification result. As a result, if the same correction as in the past is necessary, the user can easily make the correction by selecting an option without having to enter characters, etc.

[0065] [Modification of Display of Options] In the above-described example 2 for determining the identification result, the option (icon) of the top candidate food with the highest identification score was displayed in a larger size than the other two candidate food options and the option of the food previously designated as a correction target, for example. However, the display size of each option is not particularly limited. For example, the top candidate food option may be displayed in the same display size as the other two candidate food options and the option of the food previously designated as a correction target. In this case, the top candidate food option may be displayed more prominently than the other options by, for example, changing the display color of the option.

[0066] In the second example of operation for confirming the identification result, the option for the food previously designated by the user as a target for correction is displayed in place of the option for the lowest-ranked candidate food among the four candidate foods. However, in addition to the four candidate foods, the option for the food previously designated by the user as a target for correction may also be displayed. In other words, a total of five options may be displayed. In this way, in the second example of operation for confirming the identification result, it is sufficient that at least some of the four candidate foods are displayed; for example, the top three of the four candidate foods with the highest identification scores may be displayed, or all four candidate foods may be displayed.

[0067] In the second example of operation for confirming the identification result, the food option previously designated by the user as a candidate for correction appears at first glance to have the fourth-highest identification score and is indistinguishable from the fourth candidate food. However, the candidate food options and the food options previously designated by the user as candidates for correction may be displayed in a manner that makes them easy to distinguish. For example, the food option previously designated by the user as a candidate for correction may be annotated with a comment such as "option based on correction information."

[0068] In the above-described operation example 2 for confirming the identification result, only one option for the food item previously designated by the user as the target for correction is displayed on the confirmation screen, but multiple options may be displayed. For example, if it is determined in step S31 that multiple pieces of correction information exist and the multiple pieces of correction information each designate different foods as the target for correction, multiple foods previously designated as the target for correction can be displayed.

[0069] In the second example of operation for confirming the above-mentioned identification result, the number of food options previously designated by the user as options to be corrected on the confirmation screen is less than the number of candidate food options ranked second and below, but the number of options is not particularly limited. The number of food options previously designated by the user as options to be corrected on the confirmation screen may be the same as or greater than the number of candidate food options ranked second and below.

[0070] [Other Modifications] In the above-described embodiment, refrigerator 20 may be a refrigerator for general household use, a refrigerator used for product display in a retail store such as a convenience store, or a refrigerator for other commercial use.

[0071] Furthermore, the refrigerator 20 is an example of a storage cabinet, and the present disclosure can also be realized as other storage cabinets that have a function of cooling stored items, such as a freezer. Furthermore, the items (i.e., objects) stored in the storage cabinet are not limited to food, and may be other items. Cases in which items other than food are stored in the refrigerator 20 are also conceivable.

[0072] [Effects, etc.] Hereinafter, examples of techniques that can be obtained from the disclosure of this specification will be given, and effects, etc. that can be obtained from these techniques will be described.

[0073] Technology 1 relates to a food identification system (10) that includes: a control unit (56) that stores, in a memory unit (53), correction information indicating that a first identification result obtained by a machine learning model of a first food shown in a first image of a storage cabinet having a cooling function for stored items has been presented to a user as options; and, if the user selects a food that is not included in the options as the user's identification result of the first food, that the top candidate food among the plurality of candidate foods determined by the machine learning model to be the first food has been corrected to the selected food; an acquisition unit (54) that acquires a second identification result obtained by the machine learning model of a second food shown in a second image of the storage cabinet; and, if the candidate food determined by the machine learning model to be the second food most likely to be the second food among the plurality of candidate foods shown in the acquired second identification result is the same as the top candidate food, an output unit (55) that outputs, based on the correction information, presentation information for presenting the user with at least some of the plurality of candidate foods shown in the second identification result and the selected food as options. The detailed identification result information in the above embodiment is an example of the first identification result and the second identification result. The first image and the second image may be the same image or may be different images.

[0074] Such a food identification system 10 can assist the user in correcting the food identification results obtained by the machine learning model (image recognition processing) by presenting options based on the correction information.

[0075] Technology 2 is the food identification system 10 of Technology 1, in which a display unit 62 of an information terminal 60 that acquires the presentation information output by the output unit 55 displays a display screen based on the presentation information, and the display screen displays at least some of the multiple candidate foods indicated by the second identification result and the specified food as options.

[0076] The food identification system 10 can assist the user in correcting the food identification result by displaying options based on the correction information on the display unit 62 of the information terminal 60.

[0077] Technique 3 is the food identification system 10 of Technique 2, in which food options that are not the subject of identification by the machine learning model are displayed on the display screen as options for the specified food.

[0078] Such a food identification system 10 can assist in correcting the food identification results of a machine learning model to foods that are not the target of the machine learning model's identification.

[0079] Technique 4 is the food identification system 10 of Technique 2 or 3, wherein on the display screen, the display size of the candidate food option determined by the machine learning model in the second identification result to have the highest probability of being the second food is larger than the display size of the other candidate food options indicated in the second identification result and the display size of the specified food option.

[0080] Such a food identification system 10 can highlight the candidate food option that is determined by the machine learning model to have the highest probability of being the second food item over the other options.

[0081] Technique 5 is the food identification system 10 of Technique 4, in which the display size of the other candidate food options indicated by the second identification result and the display size of the specified food option on the display screen are the same.

[0082] Such a food identification system 10 can display other candidate food options and the specified food option in the same size.

[0083] Technique 6 is the food identification system 10 of Technique 4 or 5, in which the number of other candidate food options shown by the second identification result on the display screen is greater than the number of specified food options.

[0084] Such a food identification system 10 may display fewer of the specified food options than other candidate food options.

[0085] Technology 7 is the food identification system 10 of any of Technologies 2 to 6, in which the food identification system 10 includes a server device 50 including a control unit 56, an acquisition unit 54, and an output unit 55, and an information terminal 60, and the information terminal 60 acquires the output presentation information by communicating with the server device 50.

[0086] The food identification system 10 can assist the user in correcting the food identification result by transmitting presentation information from the server device 50 to the information terminal 60.

[0087] Technology 8 is an information terminal 60 including: a display control unit 67 that causes a display unit 62 to display a first display screen for presenting to a user, as options, a plurality of candidate foods indicated by a first identification result by a machine learning model of a first food shown in a first image inside a storage cabinet having a cooling function for stored items; an operation receiving unit 61 that receives from a user an operation to specify a food not included in the options as the user's identification result of the first food after the first display screen is displayed on the display unit 62; and an acquisition unit 66 that acquires presentation information for presenting to the user, as options, a plurality of candidate foods indicated by a second identification result by the machine learning model of a second food shown in a second image inside the storage cabinet, wherein the display control unit 67 causes the display unit 62 to display a second display screen based on the acquired presentation information; and, if the candidate food determined by the machine learning model in the second identification result to have the highest probability of being the second food is the same as the candidate food determined by the machine learning model in the first identification result to have the highest probability of being the first food, the second display screen displays, as options, at least some of the plurality of candidate foods indicated by the second identification result and the food specified by the operation. The detailed identification result information in the above embodiment is an example of a first identification result and a second identification result. The confirmation screen in Fig. 4 is an example of a first display screen, and the confirmation screen in Fig. 8 is an example of a second display screen. Note that the first image and the second image may be the same image or may be different images.

[0088] Such an information terminal 60 can support the user in correcting the food identification results obtained by the machine learning model by displaying options based on the correction information.

[0089] Technique 9 is a food identification method executed by a computer, the food identification method including: a step S26 of storing in a memory unit 53 correction information indicating that a top candidate food determined by the machine learning model to be the first food among the plurality of candidate foods that is most likely to be the first food among the plurality of candidate foods has been corrected to the specified food, when a food not included in the choices is specified as the user's identification result of the first food; a step S30 of acquiring a second identification result determined by the machine learning model of a second food shown in a second image inside the storage; and a step S33 of outputting presentation information for presenting to the user, based on the correction information, at least some of the plurality of candidate foods indicated by the second identification result and the specified food, when the candidate food determined by the machine learning model to be the second food among the plurality of candidate foods that is most likely to be the second food among the plurality of candidate foods that is specified as the top candidate ...

[0090] Such a food identification method can assist a user in correcting the food identification results obtained by a machine learning model by presenting options based on the correction information.

[0091] Technique 10 is a program for causing a computer to execute the food identification method of technique 8.

[0092] According to such a program, the computer can assist the user in correcting the food identification results obtained by the machine learning model by presenting options based on the correction information.

[0093] and a step S35 of displaying a second display screen based on the acquired presentation information on the display unit 62. In the display unit 62, when the candidate food determined by the machine learning model in the second classification result to be the second food item in the second classification result is the same as the candidate food determined by the machine learning model in the first classification result to be the first food item, at least some of the candidate foods indicated by the second classification result and the food specified by the machine learning model in the first classification result are displayed as options on the second display screen.

[0094] This display method can assist the user in correcting the food identification results obtained by the machine learning model by displaying options based on the correction information.

[0095] Technique 12 is a program for causing a computer to execute the display method described in Technique 11.

[0096] According to such a program, the computer can assist the user in correcting the food identification results obtained by the machine learning model by displaying options based on the correction information.

[0097] (Other Embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0098] For example, in the above embodiments, the food identification system is realized by multiple devices, but it may also be realized by a single device. For example, the food identification system may be realized as a single device corresponding to a server device. When the food identification system is realized by multiple devices, the components (particularly functional components) of the food identification system may be allocated in any way among the multiple devices. For example, some or all of the processing described in the above embodiments as being executed by the server device may be executed by an information terminal. Furthermore, some or all of the processing described in the above embodiments as being executed by the image recognition server may be executed by the server device.

[0099] Furthermore, for example, the communication method between the devices in the above-described embodiment is not particularly limited. Furthermore, a relay device (not shown) may be involved in the communication between the devices. Furthermore, the information transmission path described in the above-described embodiment is not limited to the transmission path shown in the sequence diagram.

[0100] For example, in the above embodiment, a process executed by a specific processing unit may be executed by another processing unit, the order of multiple processes may be changed, or multiple processes may be executed in parallel.

[0101] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0102] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0103] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, etc. Furthermore, the general or specific aspects of the present disclosure may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0104] For example, the present disclosure may be realized as a food identification method or display method executed by a computer, or as a program for causing a computer to execute the food identification method or display method. The present disclosure may also be realized as a computer-readable non-transitory recording medium having such a program recorded thereon.

[0105] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope of this disclosure.

[0106] The food identification system of the present disclosure is useful as a system that allows a user to easily correct the identification results obtained by image recognition of items in a storage facility.

[0107] REFERENCE SIGNS LIST 10 Food identification system 20 Refrigerator (storage) 30 Photography device 40 Image recognition server 50 Server device 51, 65 Communication unit 52, 63 Information processing unit 53, 64 Storage unit 54, 66 Acquisition unit 55 Output unit 56 Control unit 60 Information terminal 61 Operation acceptance unit 62 Display unit 67 Display control unit 70 Wide area communication network

Claims

1. After presenting a plurality of candidate foods indicated by a first identification result of a first food by a machine learning model in a first image reflected in a storage having a cooling function for stored items to a user as options, when a food not included in the options is designated as the identification result of the first food by the user, a control unit stores in a storage unit correction information indicating that the top candidate food determined by the machine learning model to have the highest probability of being the first food among the plurality of candidate foods has been corrected to the designated food; an acquisition unit that acquires a second identification result of a second food by the machine learning model in a second image reflected in the storage; and an output unit that outputs presentation information for presenting to the user at least a part of the plurality of candidate foods indicated by the second identification result and the designated food as options based on the correction information when a candidate food determined by the machine learning model to have the highest probability of being the second food among the plurality of candidate foods indicated by the acquired second identification result is the same as the top candidate food. A food identification system.

2. On a display unit of an information terminal that has acquired the presentation information output by the output unit, a display screen based on the presentation information is displayed, and at least a part of the plurality of candidate foods indicated by the second identification result and the designated food are displayed as options on the display screen. The food identification system according to claim 1.

3. On the display screen, as an option for the designated food, an option for a food that is not an object of identification by the machine learning model is displayed. The food identification system according to claim 2.

4. On the display screen, the display size of the option for the candidate food determined by the machine learning model to have the highest probability of being the second food in the second identification result is larger than the display sizes of the options for other candidate foods indicated by the second identification result and the option for the designated food. The food identification system according to claim 2.

5. On the display screen, the display sizes of the options for other candidate foods indicated by the second identification result and the option for the designated food are the same. The food identification system according to claim 4.

6. On the display screen, the number of options for other candidate foods indicated by the second identification result is larger than the number of options for the designated food. The food identification system according to claim 4.

7. The food identification system includes a server device including the control unit, the acquisition unit, and the output unit, and the information terminal. The information terminal acquires the presented information output by communicating with the server device. The food identification system according to any one of claims 2 to 6.

8. A display control unit that causes a display unit to display a first display screen for presenting, as options to a user, a plurality of candidate foods indicated by a first identification result obtained by a machine learning model of a first food shown in a first image in a storage having a cooling function for stored items; an operation reception unit that, after the first display screen is displayed on the display unit, receives from the user an operation of designating a food not included in the options as an identification result of the first food by the user; and an acquisition unit that acquires presentation information for presenting, as options to the user, a plurality of candidate foods indicated by a second identification result obtained by the machine learning model of a second food shown in a second image in the storage. The display control unit causes the display unit to display a second display screen based on the acquired presentation information. When the candidate food determined to have the highest probability of being the second food by the machine learning model in the second identification result and the candidate food determined to have the highest probability of being the first food by the machine learning model in the first identification result are the same, at least a part of the plurality of candidate foods indicated by the second identification result and the food designated by the operation are displayed as options on the second display screen. Information terminal.

9. A food identification method executed by a computer, comprising: presenting, as options to a user, a plurality of candidate foods indicated by a first identification result of a first food shown in a first image in a storage having a cooling function for stored items by a machine learning model; and when a food not included in the options is designated as the identification result of the first food by the user, storing, in a storage unit, correction information indicating that the top candidate food determined to have the highest probability of being the first food by the machine learning model among the plurality of candidate foods is corrected to the designated food; obtaining a second identification result of a second food shown in a second image in the storage by the machine learning model; and when the candidate food determined to have the highest probability of being the second food by the machine learning model among the plurality of candidate foods indicated by the obtained second identification result is the same as the top candidate food, outputting, based on the correction information, presentation information for presenting, as options to the user, at least a part of the plurality of candidate foods indicated by the second identification result and the designated food. A food identification method.

10. A program for causing the computer to execute the food identification method according to claim 9.

11. A display method executed by a computer, comprising: displaying, on a display unit, a first display screen for presenting, to a user, as options, a plurality of candidate foods indicated by a first identification result of a first food in a first image in a storage having a cooling function for stored items; after the first display screen is displayed on the display unit, receiving, from the user, an operation of designating a food not included in the options as an identification result of the first food by the user; acquiring presentation information for presenting, to the user, as options, a plurality of candidate foods indicated by a second identification result of a second food in a second image in the storage by the machine learning model; and displaying, on the display unit, a second display screen based on the acquired presentation information, wherein when a candidate food determined to have the highest probability of being the second food by the machine learning model in the second identification result and a candidate food determined to have the highest probability of being the first food by the machine learning model in the first identification result are the same, at least a part of the plurality of candidate foods indicated by the second identification result and the food designated by the operation are displayed as options on the second display screen.

12. A program for causing the computer to execute the display method according to claim 11.