Display method, program, information terminal, and food identification system

Abstract

This display method is executed by an information terminal (60). The display method includes: an acquisition step of acquiring information on a detection frame applied to each of a plurality of foods, the detection frame indicating identification results, obtained by a machine learning model, on the plurality of foods appearing in an internal image of a refrigerator; a determination step (S18a) of determining a display order of the identification results on the plurality of foods on the basis of the acquired information on the detection frame; and a display step (S19a) of displaying a confirmation screen for the user to confirm whether or not the identification results on the plurality of foods are correct. On the confirmation screen, the identification results on the plurality of foods are displayed in order on the basis of the determined display order.

Description

Display method, program, information terminal, and food identification system

[0001] The present disclosure relates to a display method, a program, an information terminal, and a food identification system.

[0002] Conventionally, technologies relating to storage cabinets for storing objects have been proposed. Patent Document 1 discloses a terminal device that acquires the types of food stored in a refrigerator from an image captured from above the front of the refrigerator and displays recipe information corresponding to the food stored in the refrigerator on a display unit.

[0003] JP 2023-79899 A

[0004] The present disclosure provides a display method that allows a user to sequentially check the identification results of a machine learning model for multiple foods shown in an image of the interior of a storage cabinet.

[0005] A display method according to one aspect of the present disclosure is a display method executed by a computer, and includes an acquisition step of acquiring information regarding detection frames assigned to each of a plurality of foods that appear in an image of the interior of a storage cabinet having a cooling function for stored items, the detection frames indicating the identification results of the plurality of foods by a machine learning model; a determination step of determining a display order for the identification results of the plurality of foods based on the acquired information regarding the detection frames; and a display step of displaying a confirmation screen for a user to confirm whether the identification results of the plurality of foods are correct, and the identification results of the plurality of foods are displayed in order on the confirmation screen based on the determined display order.

[0006] According to a display method according to one aspect of the present disclosure, a user can sequentially check the identification results of a machine learning model for multiple foods shown in an image of the interior of a storage cabinet.

[0007] FIG. 1 is a block diagram showing the functional configuration of a food identification system according to an embodiment. FIG. 2 is an external view of a refrigerator and an imaging device included in the food identification system according to an embodiment. FIG. 3 is a diagram showing an example of a list screen of the identification results of a plurality of foods. FIG. 4 is a diagram showing an example of a confirmation screen for details of the food identification results. FIG. 5 is a sequence diagram of an operation for displaying the food identification results. FIG. 6 is a sequence diagram of an operation example 1 for confirming the food identification results. FIG. 7 is a sequence diagram of an operation example 2 for confirming the food identification results. FIG. 8 is a diagram showing an example of a confirmation screen in which a relatively large detection frame is superimposed on an image of the refrigerator interior. FIG. 9 is a sequence diagram of an operation example 3 for confirming the food identification results. FIG. 10 is a diagram for explaining the degree of overlap of detection frames. FIG. 11 is a diagram showing an example of multiple detection frames having an inclusion relationship. FIG. 12 is a diagram showing a confirmation screen according to Modification 1. FIG. 13 is a diagram showing an example of a confirmation screen in which large detection frames are assigned to two or more foods. FIG. 14 is a diagram showing a confirmation screen according to Modification 2. FIG. 15 is a diagram showing a confirmation screen according to Modification 3.

[0008] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection forms, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.

[0009] It should be noted that the drawings are schematic diagrams and are not necessarily strict illustrations. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and overlapping descriptions may be omitted or simplified.

[0010] (Embodiment) [Configuration] First, the configuration of a food identification system according to an embodiment will be described. Fig. 1 is a block diagram showing the functional configuration of a food identification system according to an embodiment.

[0011] The food identification system 10 shown in Figure 1 is a system that can manage the types and quantities of food stored in a refrigerator 20 by identifying the types of food stored in the refrigerator 20 from images of the inside of the refrigerator 20. Specifically, the food identification system 10 includes a refrigerator 20, an image capture device 30, an image recognition server 40, a server device 50, and an information terminal 60.

[0012] The refrigerator 20 is an example of a storage unit capable of refrigerating stored items, and is installed in a user's home or the like to refrigerate food. Fig. 2 is an external view of the refrigerator 20 (and the image capturing device 30).

[0013] The image capturing device 30 captures an image of a drawer (e.g., a vegetable drawer) of the refrigerator 20 in an open state from above the refrigerator 20. That is, the image capturing device 30 captures, for example, an image of the drawer of the refrigerator 20 viewed from above the refrigerator 20. The image capturing device 30 is realized, for example, by a camera having a wide-angle lens and a telephoto lens.

[0014] The image recognition server 40 is a computer located outside the facility where the refrigerator 20 is installed, and specifically, is a cloud server. The image recognition server 40 acquires images captured by the image capture device 30 by communicating with the image capture device 30 via the wide area communication network 70, and identifies the type of food shown in the acquired image. The type of food is, for example, a specific variety of vegetable (cucumber, tomato, etc.). In the following embodiment, an example of identifying the type of vegetable will be described, but the image recognition server 40 can also identify the type of vegetable, the type of meat, the type of beverage, etc.

[0015] Server device 50 is a computer located outside the facility where refrigerator 20 is installed, and specifically, is a cloud server. Server device 50 presents the food type identification results obtained by image recognition server 40 to the user via information terminal 60, and accepts instructions from the user to confirm or correct the presented identification results. Server device 50 includes a communication unit 51, an information processing unit 52, and a storage unit 53.

[0016] The communication unit 51 is a communication circuit that enables the server device 50 to communicate with the image recognition server 40 and the information terminal 60 via the wide area communication network 70. The communication unit 51 is, for example, a wired communication circuit that performs wired communication, but may also be a wireless communication circuit that performs wireless communication. There are no particular limitations on the communication standard used for communication by the communication unit 51.

[0017] The information processing unit 52 performs information processing and the like to present the food type identification results obtained by the image recognition server 40 to the user via the information terminal 60. The information processing unit 52 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. The functions of the information processing unit 52 are realized, for example, by the microcomputer or the like constituting the information processing unit 52 executing a computer program stored in the storage unit 53.

[0018] The storage unit 53 is a storage device that stores the computer programs executed by the information processing unit 52 and various information required for the information processing. The storage unit 53 is realized by, for example, a semiconductor memory.

[0019] The information terminal 60 is an information terminal owned by a user. The information terminal 60 displays the food type identification result obtained by the image recognition server 40 and accepts instructions from the user to confirm or correct the displayed identification result. The information terminal 60 is, for example, a portable information terminal such as a smartphone or a tablet terminal, but may also be a stationary information terminal such as a personal computer. The information terminal 60 includes an operation receiving unit 61, a display unit 62, an information processing unit 63, a storage unit 64, and a communication unit 65.

[0020] The operation reception unit 61 receives operations from the user. The operation reception unit 61 is realized by, for example, a touch panel, but may also be realized by a mouse and keyboard.

[0021] The display unit 62 displays display screens such as those shown in Fig. 3 and Fig. 4. Fig. 3 is a diagram showing an example of a list screen of the classification results of a plurality of foods by the image recognition server 40, and Fig. 4 is a diagram showing an example of a confirmation screen of the details of the classification result of a certain food by the image recognition server 40. In other words, the confirmation screen of Fig. 4 is a display screen on which the user can confirm whether the classification result by the machine learning model is correct or not, and is also a display screen on which the user can confirm or correct the classification result. The display unit 62 is realized by a display panel such as a liquid crystal panel or an organic EL (Electro-Luminescence) panel, for example.

[0022] The information processing unit 63 performs information processing and the like to display the display screens such as those shown in Figures 3 and 4. The information processing unit 63 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. The information processing unit 63 has, as functional components, an acquisition unit 66, a determination unit 67, and a display control unit 68. The functions of the acquisition unit 66, the determination unit 67, and the display control unit 68 are realized, for example, by the microcomputer or the like constituting the information processing unit 63 executing a computer program stored in the storage unit 64. The functions of the acquisition unit 66, the determination unit 67, and the display control unit 68 will be described in detail below.

[0023] The storage unit 64 is a storage device that stores the computer program executed by the information processing unit 63 and various information required for the information processing (such as the food management information described below). The storage unit 64 is realized, for example, by a semiconductor memory. In order to display the display screens of Figures 3 and 4, a predetermined application program (hereinafter also simply referred to as a predetermined app) is pre-installed in the storage unit 64, and the computer program executed by the information processing unit 63 includes such a predetermined app.

[0024] The communication unit 65 is a communication circuit that enables the information terminal 60 to communicate with the server device 50 via the wide area communication network 70. The communication unit 65 is, for example, a wireless communication circuit that performs wireless communication, but may also be a wired communication circuit that performs wired communication. There are no particular limitations on the communication standard of the communication performed by the communication unit 65.

[0025] [Display Operation of Food Identification Results] Next, a description will be given of the operation of displaying food identification results on the information terminal 60, which is performed by the food identification system 10. Fig. 5 is a sequence diagram of the operation of displaying food identification results.

[0026] The image recognition server 40 communicates with the image capturing device 30 to receive an image of the interior of the refrigerator 20 (hereinafter also referred to as an interior image) from the image capturing device 30 (S11).

[0027] The image recognition server 40 performs object detection processing on the received fridge interior image to detect food appearing in the fridge interior image and identify the type of food detected (S12). Specifically, a detection unit (not shown) within the image recognition server 40 uses a machine learning model (trained model) to detect objects (food) in the image and identify the type of food detected.

[0028] The food type identification result is expressed, for example, by the food type and an identification score. The identification score is a score that indicates the likelihood of the identification result and is expressed between 0 and 1, with larger values ​​indicating higher likelihood. Likelihood can also be referred to as validity, accuracy, or reliability.

[0029] For example, for one food (detection frame), the food classification result indicates n food types in descending order of classification score, and the classification score for each of the n foods. n is a natural number greater than or equal to 2, and in the following embodiments, the description will be given assuming n = 4. Specifically, the food classification result is information indicating the following: tomato: 0.4, broccoli: 0.2, lettuce: 0.15, pumpkin: 0.1. Below, the four foods included in the classification result will be referred to as candidate foods, and the candidate food with the highest classification score among the candidate foods will be referred to as the top candidate food.

[0030] The image recognition server 40 transmits to the server device 50 identification result information indicating the identification results for each of the multiple foods shown in the fridge interior image received in step S11 (S13). For each of the multiple foods, the identification result information includes, for example, the management ID of the food, name information of the four candidate foods for that food (i.e., label information for the food type corresponding to the top four identification results), identification scores for each of the four candidate foods, and the coordinates of the four corners of the detection frame. The detection frame refers to the rectangular frame assigned to the vegetables in FIG. 4 and is sometimes referred to as a bounding box. The identification result information can be said to indirectly indicate the size and shape of the detection frame based on the coordinates of the four corners of the detection frame. The identification result information also includes image information of the fridge interior image. The identification result information is an example of information related to the detection frame.

[0031] The communication unit 51 of the server device 50 receives the identification result information and transmits the received identification result information to the information terminal 60 (S14).

[0032] The communication unit 65 of the information terminal 60 receives the identification result information. The acquisition unit 66 acquires the identification result information (S15), and the display control unit 68 causes the display unit 62 to display a list screen of the identification results as shown in FIG. 3 based on the acquired detection frame information (S16).

[0033] The list screen in FIG. 3 displays a list of the classification results (top candidate foods) of five foods shown in the image, and the classification results of the five foods are tomato, lettuce, potato, green onion, and broccoli.

[0034] In this way, the food identification system 10 can display the identification results for each of the multiple foods shown in the image on the display unit 62 of the information terminal 60.

[0035] [Operation Example 1 for Confirming Identification Results] The user confirms whether the identification results by the image recognition server 40 are correct for each of the five foods displayed on the list screen in Figure 3. The food identification system 10 confirms the identification results that the user determines to be correct, and corrects the identification results that the user determines to be incorrect in accordance with the user's instructions before confirming them. As a result, the inventory of food items in the refrigerator 20 is updated in the inventory management server (not shown). Operation Example 1 for confirming the identification results of food items will be described below. Figure 6 is a sequence diagram of Operation Example 1 for confirming the identification results of food items.

[0036] For example, the user performs a confirmation operation to check the details of the identification result for a certain food (hereinafter also referred to as the food to be confirmed). For example, the confirmation operation to check the identification result for a food identified as a tomato is a tap operation on the word "Confirm" in the column for the food identified as a tomato. The operation accepting unit 61 accepts such a confirmation operation (S17).

[0037] The display control unit 68 causes the display unit 62 to display a confirmation screen for the details of the identification result as shown in FIG. 4, based on the identification result information acquired in step S15 (S18).

[0038] The upper part of the confirmation screen shown in FIG. 4 displays the fridge interior image received in step S11 and included in the identification result information, and a detection frame is superimposed on the food to be confirmed that appears in the fridge interior image.

[0039] Furthermore, the four candidate foods indicated by the identification result information are displayed as options at the bottom of the confirmation screen. In the example of FIG. 4 , the top candidate food of the four candidate foods is onion, and the other three candidate foods are tomato, lettuce, and pumpkin. As shown in FIG. 4 , the option (icon) of the top candidate food with the highest identification score is displayed, for example, in a larger size than the other three candidate food options. The other three candidate food options are displayed in a horizontal row below the top candidate food option. The candidate food option with the second highest identification score is displayed on the left, the candidate food option with the third highest identification score is displayed in the center, and the candidate food option with the fourth highest identification score is displayed on the right.

[0040] The user performs an operation to input which of the four candidate foods the food in the detection frame at the top of the confirmation screen (i.e., the food to be confirmed) corresponds to (i.e., the user's identification result of the food to be confirmed). If the user determines that the food to be confirmed does not correspond to any of the four candidate foods, the user can select "Enter by text" on the confirmation screen of Figure 4 and manually input the user's identification result.

[0041] The operation receiving unit 61 receives the predetermined operation as described above (S19), and the information processing unit 63 communicates with the server device 50 using the communication unit 65 to transmit the confirmation result information to the server device 50 (S20). The confirmation result information is information indicating the type of food item to be confirmed, as specified by the user.

[0042] The communication unit 51 of the server device 50 receives the confirmation result information, and the information processing unit 52 determines the identification result based on the received confirmation result information (S21). The information processing unit 52 notifies the inventory management server (not shown) using the communication unit 51 that the food item entered by the user has been stored in the refrigerator 20 (not shown).

[0043] In this way, in operation example 1, the information terminal 60 can reduce the effort required for the user to input the identification results by displaying the identification results based on the machine learning model of the food to be confirmed as options.

[0044] Furthermore, if the identification result of the machine learning model for the food being checked does not match the identification result by the user, the user can manually input the identification result by the user, i.e., the user can correct the identification result by the machine learning model.

[0045] [Operation Example 2 for Confirming Identification Results] In Operation Example 1, an example was described in which the identification result of one food item selected by the user is displayed on the list screen of FIG. 3. Here, by tapping the text "Confirm corrections all at once" on the list screen of FIG. 3, the user can collectively confirm the identification results of multiple foods displayed on the list screen. Specifically, confirmation screens (display screens equivalent to FIG. 4) for the multiple foods displayed on the list screen are displayed in sequence. Below, Operation Example 2 for confirming the identification results of foods will be described. FIG. 7 is a sequence diagram of Operation Example 2 for confirming the identification results of foods.

[0046] After step S16, the user performs a collective confirmation operation to collectively check the details of the classification results of the multiple foods displayed on the list screen. As described above, the collective confirmation operation is, for example, a tap operation on the text "Confirm corrections collectively." The operation accepting unit 61 accepts such a collective confirmation operation (S17a).

[0047] The determination unit 67 of the information terminal 60 determines the display order of the plurality of foods based on the identification result information received in step S15 (S18a). The determination unit 67 determines the display order based on, for example, the size of the detection frame indicated by the identification result information. More specifically, the determination unit 67 determines the display order so that the identification results are displayed in order of the size of the detection frame of the food item, starting with the smallest.

[0048] The determination unit 67 may determine the size of a detection frame based on the length of the diagonal of the detection frame, the area within the detection frame, or the perimeter of the detection frame. Furthermore, the determination unit 67 may determine the size of a detection frame based on two or more of the three parameters of the length of the diagonal of the detection frame, the area within the detection frame, and the perimeter of the detection frame. For example, the determination unit 67 may determine the size of a detection frame based on the length of the diagonal of the detection frame, and, if the diagonal lengths are the same, determine the size based on the area within the detection frame, thereby using two or more parameters in combination. That is, in step S18a, the determination unit 67 may determine the display order based on at least one of the length of the diagonal of the detection frame, the area within the detection frame, and the perimeter of the detection frame.

[0049] The process of step S18a may be performed after step S16 and before step S17a, or may be performed after step S15 and before step S16.

[0050] Based on the identification result information acquired in step S15 and the display order determined in step S18a, the display control unit 68 causes the display unit 62 to display a confirmation screen (corresponding to Figure 4) showing details of the identification result of the food item that is first in the display order (the food item with the smallest detection frame size) (S19a).

[0051] The user performs an operation to input which of the four candidate foods the food in the detection frame at the top of the confirmation screen (i.e., the food to be confirmed) corresponds to (i.e., the user's identification result of the food to be confirmed). If the user determines that the food to be confirmed does not correspond to any of the four candidate foods, the user can select "Enter by text" on the confirmation screen of Figure 4 and manually input the user's identification result.

[0052] The operation receiving unit 61 receives the predetermined operation as described above (S20a), and the information processing unit 63 communicates with the server device 50 using the communication unit 65 to transmit the confirmation result information to the server device 50 (S21a). The confirmation result information is information indicating the type of food to be confirmed that was input by the user.

[0053] The processing of steps S19a to S21a is repeated the same number of times as the number of foods displayed on the list screen. If five foods are displayed on the list screen, the processing of steps S19a to S21a is repeated five times. At this time, the food identification results are displayed in the display order determined in step S18a.

[0054] Then, communication unit 51 of server device 50 receives the confirmation result information of the plurality of foods, and information processing unit 52 determines the identification result based on the received confirmation result information (S22a). Information processing unit 52 notifies an inventory management server (not shown) using communication unit 51 that the foods specified by the user have been stored in refrigerator 20 (not shown).

[0055] In this way, according to Operation Example 2, the user can check the classification results of multiple foods by machine learning models all at once.

[0056] As described above, the confirmation screen displays the classification results in order of the size of the detection frame, starting with the smallest. The following describes the effect of displaying the classification results in order of the size of the detection frame, starting with the smallest.

[0057] When the size of the detection frame is large, there is a high possibility that other foods are displayed within the detection frame in addition to the food to be checked. FIG. 8 is a diagram showing an example of a confirmation screen in which a relatively large detection frame is superimposed on an image of the refrigerator interior. In the example of FIG. 8, a detection frame is attached to a green onion, and the food to be checked is the green onion, but other foods are also displayed within the detection frame. In this case, the user may misidentify the food to be checked or be confused about which type of food within the detection frame to input.

[0058] Here, if the detection frame is small, it is unlikely that a food other than the food being checked is visible within the detection frame. Furthermore, if the confirmation result for a first food, which has a small detection frame, is entered first, when the confirmation result for another second food is subsequently entered, even if the first food is visible within the detection frame for the second food, the user is likely to realize that they should enter the confirmation result for the second food because they have already confirmed the first food. Therefore, if the identification results are displayed in order of the size of the detection frame, starting with the food with the smallest size, it is possible to prevent misidentification of the food to be checked and the need to retry due to incorrect input.

[0059] [Operation Example 3 for Confirming Identification Results] The user can also check the identification results of only some of the multiple food items displayed on the list screen at once. Operation Example 3 for confirming the food item identification results will be described below. Figure 9 is a sequence diagram of Operation Example 3 for confirming such food item identification results.

[0060] After step S16, the user performs an operation to collectively check the details of only some of the classification results of the multiple foods displayed on the list screen. Such an operation is, for example, an operation to specify a dish. In this case, for example, an interface for accepting the operation to specify a dish is provided on the list screen of FIG. 3.

[0061] The operation receiving unit 61 receives such a dish designation operation (S17b) In other words, the determining unit 67 receives designation information for allowing the user to confirm the classification results of some of the multiple food items.

[0062] Here, for a plurality of dishes, recipe information indicating one or more foods required for the dishes is stored in the storage unit 64. The recipe information is stored in the storage unit 64 when the above-mentioned predetermined app is installed, for example.

[0063] The determination unit 67 selects some of the foods necessary for the dish specified by the specifying operation from the multiple foods displayed on the list screen by referring to the recipe information, and determines the display order of the selected some of the foods (S18b). As in Operation Example 2, the determination unit 67 determines the display order so that the classification results are displayed in order of the size of the detection frame, for example, from smallest to largest. The subsequent processing is the same as in Operation Example 2.

[0064] In this way, the number of foods for which the identification results should be checked is narrowed down, allowing the user to check only the minimum number of identification results. Note that the method of specifying a portion of a plurality of foods is not limited to specifying a dish, and for example, the user may directly specify the food that the user wants to check from the list screen in Figure 3.

[0065] [Variation 1 of the method for determining the display order] In the above-described operation example 2 or operation example 3, the display order of the classification results of multiple foods is determined based on the size of the detection frame indicated by the information related to the detection frame. The display order of the classification results of multiple foods may also be determined based on the size and shape of the detection frame. The shape of the detection frame is expressed by, for example, an aspect ratio. The aspect ratio can be determined from the coordinates of the four corners of the detection frame included in the classification result information.

[0066] For example, when the size of the detection frame for a certain food is large, as in the example of green onion in Fig. 8, it is considered that the closer the aspect ratio of the detection frame is to 1:1, the more likely it is that other foods are included within the detection frame. Also, when the size of the detection frame for a certain food is small, it is considered that the further the aspect ratio of the detection frame is from 1:1, the more likely it is that other foods are included within the detection frame.

[0067] Therefore, the determination unit 67 of the information terminal 60 sets the display priority of foods assigned with detection frames whose size is less than a threshold higher than the display priority of foods assigned with detection frames whose size is equal to or greater than the threshold. Among foods assigned with detection frames whose size is less than the threshold, the determination unit 67 sets a higher display priority for foods assigned with detection frames whose aspect ratio is closer to 1:1, and among foods assigned with detection frames whose size is equal to or greater than the threshold, the determination unit 67 sets a higher display priority for foods whose aspect ratio is further from 1:1. Then, on the confirmation screen, the determination unit 67 displays the identification results of foods in descending order of the set display priority. This can prevent misidentification of foods that should be confirmed and the occurrence of rework due to incorrect input.

[0068] [Variation 2 of Display Order Determining Method] The determination unit 67 of the information terminal 60 may determine the display order of the identification results of multiple foods based on the overlap of the detection frames, instead of the size of the detection frames. FIG. 10 is a diagram illustrating the overlap of the detection frames. The overlap of the detection frame A shown in FIG. 10 can be calculated using the formula: overlap = (area of ​​the region overlapping with other detection frames B and C) / (area of ​​the entire region surrounded by the detection frame A). The area of ​​the region overlapping with other detection frames B and C is the area of ​​the region hatched with diagonal lines in FIG. 10. The area of ​​the region overlapping with other detection frames B and C and the area of ​​the entire region surrounded by the detection frame A can be identified from the coordinates of the four corners of each of the detection frames A, B, and C included in the identification result information.

[0069] The higher the overlapping degree of a detection frame, the higher the likelihood that the detection frame contains a food that is not the food to be checked. Therefore, the determination unit 67 determines the display order so that foods with detection frames with low overlapping degrees are displayed in order. This can prevent misidentification of the food to be checked and the need to go back and forth due to incorrect input.

[0070] [Third Modification of Display Order Determining Method] When multiple detection frames corresponding to multiple foods have an inclusion relationship, the determination unit 67 of the information terminal 60 may determine the display order of the identification results of the multiple foods based on the inclusion relationship. Fig. 11 is a diagram showing an example of multiple detection frames having an inclusion relationship. In the example of Fig. 11, detection frame D includes detection frames E and F. Whether or not multiple detection frames D, E, and F have an inclusion relationship can be determined from the coordinates of the four corners of each of detection frames D, E, and F included in the identification result information.

[0071] In this way, for multiple detection frames D, E, and F that have an inclusion relationship, the determination unit 67 first displays the identification results of the foods assigned to the inner (encompassing) detection frames E and F, and lastly displays the identification result of the outer (encompassing) detection frame D. This makes it possible to prevent misidentification of foods that should be confirmed and the need to go back and forth due to incorrect input.

[0072] Whether the classification results of the foods with detection frames E and F to be displayed first may be determined based on another method described in this specification.

[0073] [Fourth Variation of Display Order Determining Method] The determination unit 67 of the information terminal 60 may determine the display order of the identification results of multiple foods based on the identification scores of the foods to which detection frames are assigned. The identification score here refers to the identification score of the top candidate food. The identification score is included in the identification result information. For example, the determination unit 67 displays the food identification results in descending order of identification score. This allows for early confirmation of identification results that are deemed not to require correction, thereby reducing misidentification of foods that should be confirmed later and the occurrence of incorrect input.

[0074] [Fifth Variation of Display Order Determination Method] The determination unit 67 of the information terminal 60 may determine the display order of the food identification results based on the food identification results (i.e., the top candidate food type). For example, the designer of the food identification system 10 may set priorities in advance for the food identification results (top candidate food type). For example, the designer may generally set a higher priority for smaller foods. The designer may also set priorities so that round foods have a higher priority than elongated foods.

[0075] The priority information indicating the preset priorities is stored in the storage unit 64 when the above-mentioned predetermined app is installed, etc. The determination unit 67 can determine the display order of the food identification results by referring to the priority information stored in the storage unit 64. If the priorities are set appropriately in the priority information, it is possible to prevent misidentification of foods that should be confirmed and the need to go back and forth due to incorrect input.

[0076] It is not essential that the priorities be set in advance by a designer or the like; for example, the information terminal 60 may determine the priorities so that the more frequently a food item is used by the user, the higher the priority.

[0077] For example, when the identification result information is acquired in step S15, the information processing unit 63 counts the number of top candidate foods indicated by the identification result information for each food type, and stores the count results as usage frequency information in the storage unit 64. By referring to the usage frequency information stored in the storage unit 64, the determination unit 67 can determine the display order of the food identification results so that foods with higher usage frequencies have higher priority.

[0078] By determining the display order in this way, the identification results of foods that are frequently used and are likely to appear in the refrigerator interior image are confirmed first, thereby reducing the chance of misidentification of foods that should be checked later and the occurrence of incorrect input.

[0079] [Confirmation Screen Variation 1] When two or more foods with the same classification result (the same top candidate food) are included among the multiple foods displayed on the list screen of Fig. 3, the display control unit 68 of the information terminal 60 may display a confirmation screen on the display unit 62 that allows the classification results of the two or more foods to be confirmed together. Fig. 12 is a diagram showing such a confirmation screen according to Variation 1.

[0080] The confirmation screen shown in Figure 12 displays the identification results of two or more foods (lower row) and an image of the inside of the refrigerator (upper row) with individual detection frames superimposed on each of the two or more foods to be identified. Displaying such a confirmation screen reduces the number of operations the user must perform to confirm the identification results.

[0081] Note that when two or more foods with the same identification result are included among multiple foods, the identification results of the two or more foods may be grouped and treated as a single identification result. Specifically, the information processing unit 52 of the server device 50 may merge the detection frames of two or more foods with the same identification result (same type) based on the identification result information received from the image recognition server 40, and provide the information terminal 60 with identification result information in which a single merged detection frame is defined for the two or more foods. In this case, the display control unit 68 of the information terminal 60 generally superimposes the large merged detection frame on the fridge interior image. Figure 13 is a diagram showing an example of a confirmation screen in which two or more foods are given large detection frames surrounding the two or more foods.

[0082] However, the display control unit 68 of the information terminal 60 does not simply superimpose the merged detection frame on the fridge interior image, but rather deletes unnecessary areas from the merged detection frame to generate two or more detection frames corresponding to two or more foods to be confirmed, and superimposes the generated two or more detection frames on the fridge interior image, thereby ultimately assigning individual detection frames to each of the two or more foods as in Fig. 12. This allows the information terminal 60 to reduce the amount of information processing while preventing misidentification of foods to be confirmed and the need for rework due to incorrect input.

[0083] [Second modification of confirmation screen] The display control unit 67 of the information terminal 60 may display a frame that conforms to the shape of the food to be confirmed (identified) in the fridge interior image, thereby displaying the food in a segmented manner. Figure 14 is a diagram showing such a confirmation screen according to the second modification. The segmented display of food can be achieved by performing a segmentation process that labels each pixel in the fridge interior image with what each pixel represents.

[0084] When food items are displayed in segmented form in this way, it becomes less likely that other foods will be included in the frame surrounding the food item, which reduces the chance of misidentifying the food item that needs to be checked and the need to go back and forth due to incorrect input.

[0085] In this example, a rectangular detection frame is defined in the identification result information provided by the image recognition server 40, and it is assumed that the food is displayed in a segmented form on the confirmation screen. However, the detection frame defined in the identification result information may itself have a shape that follows the outline of the food. In this case, the image recognition server 40 generates the detection frame by performing a segmentation process on the refrigerator interior image.

[0086] [Modification 3 of confirmation screen] When a dish designation operation is received by the operation receiving unit 61 as in the above-described operation example 3, a detection frame may be added to each of one or more foods required for the designated dish in the refrigerator interior image. Fig. 15 is a diagram showing such a confirmation screen according to modification 3.

[0087] In this way, if a detection frame is assigned to each of one or more foods required for a specified dish, the user can see at a glance where the one or more foods required for the dish are stored.

[0088] [Variations of Display Mode of Detection Frame] In the above embodiment, the detection frame superimposed on the fridge interior image is realized by a frame line, but it may also be realized by changing the display mode of the fridge interior image inside and outside the detection frame.

[0089] For example, the detection frame may be realized by displaying the outside of the detection frame at a lower brightness than the inside of the detection frame, or by displaying the outside of the detection frame in a lighter color than the inside of the detection frame, or by displaying the outside of the detection frame with a higher transparency than the inside of the detection frame, or by superimposing a pattern only on the outside of the inside and outside of the detection frame.

[0090] The detection frame is not limited to a rectangular shape, and may be a circle or an ellipse. The image inside the detection frame may be enlarged, and the enlarged image may be superimposed on the fridge interior image at the position of the detection frame.

[0091] [Other Modifications] In the above embodiment, the process of acquiring the identification result information and determining the display order of the identification results of a plurality of foods based on the acquired identification result information was described as being performed by the information terminal 60. However, this process may also be performed by the server device 50. For example, the information processing unit 52 of the server device 50 may acquire the identification result information received in step S14 and determine the display order of the food identification results based on the acquired identification result information. In this case, the information processing unit 52 adds display order information indicating the determined display order to the identification result information received in step S14, and transmits the identification result information with the added display order information to the information terminal 60 using the communication unit 51 in step S15. This reduces the amount of information processing by the information terminal 60.

[0092] In the above embodiment, refrigerator 20 may be a refrigerator for general household use, a refrigerator used for product display in a retail store such as a convenience store, or a refrigerator for other commercial use.

[0093] Furthermore, the refrigerator 20 is an example of a storage cabinet. The present disclosure may be applied to other storage cabinets, such as freezers, that have a function of cooling stored items. Furthermore, the items (in other words, objects) stored in the storage cabinet are not limited to food, and may be other items. Cases in which items other than food are stored in the refrigerator 20 are also conceivable.

[0094] [Effects, etc.] Hereinafter, examples of techniques that can be obtained from the disclosure of this specification will be given, and effects, etc. that can be obtained from these techniques will be described.

[0095] Technique 1 is a display method executed by a computer, the display method including: an acquisition step S15 of acquiring information about detection frames assigned to each of a plurality of food items shown in an image of the interior of a storage cabinet having a cooling function for stored items, the information indicating the identification results, based on a machine learning model, of the plurality of food items; a determination step S18a of determining a display order for the identification results of the plurality of food items based on the acquired information about the detection frames; and a display step S19a of displaying a confirmation screen for a user to confirm the accuracy of the identification results of the plurality of food items, where the identification results of the plurality of food items are displayed in order based on the determined display order on the confirmation screen. The refrigerator 20 of the above embodiment is an example of a storage cabinet, and the identification result information is an example of information about the detection frames.

[0096] This display method allows the user to sequentially check the classification results of the machine learning model for multiple foods shown in the image of the storage cabinet interior.

[0097] Technique 2 is a display method of Technique 1 in which, in determination step S18a, the display order is determined based on the size of the detection frame indicated by the information related to the detection frame.

[0098] According to this display method, the user can check the classification results in order according to the size of the detection frame.

[0099] Technique 3 is a display method of Technique 2 in which the display order is determined in the determination step S18a so that the classification results are displayed in descending order of the size of the detection frame of the food item.

[0100] According to this display method, when the identification result of a food item with a large detection frame is displayed, the small food item that appears within the large detection frame will already have been confirmed. Therefore, the display method can prevent the user from misidentifying the food item that should be confirmed and the need to go back and forth due to incorrect user input.

[0101] Technique 4 is a display method of Technique 2 or 3 in which, in decision step S18a, the size of the detection frame is based on at least one of the length of the diagonal line of the detection frame, the area inside the detection frame, and the perimeter of the detection frame.

[0102] This display method can determine the display order of the classification results of multiple foods based on at least one of the diagonal length of the detection frame, the area within the detection frame, and the perimeter of the detection frame.

[0103] Technique 5 is a display method of any of Techniques 2 to 4 in which the detection frame is rectangular, and in decision step S18a, the display order is determined based on the size and aspect ratio of the detection frame indicated by the information related to the detection frame.

[0104] This display method can determine the display order of the classification results for multiple foods by using the size and aspect ratio of the detection frame in combination.

[0105] Technique 6 is a display method of Technique 1 in which, in the determination step S18a, the display order is determined based on at least one of the overlap of the detection frames, the identification score of the food to which the detection frame is assigned, and the food identification result, as indicated by the information regarding the detection frames.

[0106] Such a display method can determine the display order of the identification results of multiple foods based on at least one of the degree of overlap of the detection frames, the identification score of the food to which the detection frame is assigned, and the food identification result.

[0107] Technique 7 is a display method according to any of techniques 1 to 6, in which an image of the interior of the refrigerator is displayed on the confirmation screen, and the food to be confirmed that appears in the image of the interior of the refrigerator is displayed in a segmented manner.

[0108] This display method displays an image of the inside of the refrigerator with only the food to be checked in focus, thereby preventing misidentification of the food to be checked and the need to go back and forth due to incorrect input.

[0109] Technique 8 is a display method of any of techniques 1 to 7, in which an image of the interior of the refrigerator is displayed on the confirmation screen with a detection frame superimposed on the food to be confirmed, and when the user confirms two or more foods to be confirmed at once, a detection frame is superimposed individually on each of the two or more foods to be confirmed on the confirmation screen.

[0110] This display method can prevent misidentification of the food to be checked and the need to go back and forth due to incorrect input by superimposing a detection frame individually on each of two or more foods to be checked.

[0111] Technology 9 is a display method described in Technology 8, which includes a step of generating detection frames from one detection frame that are individually superimposed on each of the two or more foods to be confirmed when one detection frame is defined for the two or more foods to be confirmed.

[0112] With this display method, even if one detection frame is defined for two or more foods to be checked, it is possible to superimpose a detection frame individually on each of the two or more foods to be checked.

[0113] Technique 10 is a display method according to any one of Techniques 1 to 9, further including step S17b of receiving designation information for allowing the user to confirm the identification results of some of the plurality of foods, and on the confirmation screen, the identification results of some of the plurality of foods are displayed in order based on the determined display order and the received designation information.

[0114] According to this display method, the user can sequentially check the identification results of only a specified portion of the food items displayed in the image of the interior of the storage cabinet.

[0115] Technique 11 is a program for causing a computer to execute any one of the display methods of techniques 1 to 10.

[0116] Such a program allows the user to sequentially check the identification results of a machine learning model for multiple foods shown in an image of the interior of a storage cabinet.

[0117] Technology 12 is an information terminal 60 that includes an acquisition unit 66 that acquires information about detection frames assigned to each of a plurality of foods that show the identification results, based on a machine learning model, of a plurality of foods that appear in an image inside a storage cabinet that has a cooling function for stored items; a determination unit 67 that determines the display order of the identification results of the plurality of foods based on the acquired information about the detection frames; and a display control unit 68 that displays a confirmation screen for a user to confirm whether the identification results of the plurality of foods are correct, and on the confirmation screen, the identification results of the plurality of foods are displayed in order based on the determined display order.

[0118] With such an information terminal 60, the user can sequentially check the classification results of the machine learning model for multiple foods shown in the image of the interior of the storage cabinet.

[0119] Technique 13 is a food identification system 10 including the information terminal 60 of technique 12 and a server device 50 that transmits information about the detection frame to the information terminal 60.

[0120] According to the food identification system 10, the user can sequentially check the identification results of the machine learning model for multiple foods shown in the image of the interior of the storage cabinet.

[0121] (Other Embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0122] For example, in the above embodiments, the food identification system is realized by multiple devices, but it may also be realized by a single device. For example, the food identification system may be realized as a single device corresponding to a server device. When the food identification system is realized by multiple devices, the components (particularly functional components) of the food identification system may be allocated in any way among the multiple devices. For example, some or all of the processing described in the above embodiments as being executed by the server device may be executed by an information terminal. Furthermore, some or all of the processing described in the above embodiments as being executed by the image recognition server may be executed by the server device.

[0123] Furthermore, for example, the communication method between the devices in the above-described embodiment is not particularly limited. Furthermore, a relay device (not shown) may be involved in the communication between the devices. Furthermore, the information transmission path described in the above-described embodiment is not limited to the transmission path shown in the sequence diagram.

[0124] For example, in the above embodiment, a process executed by a specific processing unit may be executed by another processing unit. Also, the order of multiple processes may be changed, or multiple processes may be executed in parallel.

[0125] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0126] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0127] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, etc. Furthermore, the general or specific aspects of the present disclosure may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0128] For example, the present disclosure may be realized as a food identification method or display method executed by a computer, or as a program for causing a computer to execute the food identification method or display method. The present disclosure may also be realized as a computer-readable non-transitory recording medium having such a program recorded thereon.

[0129] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure.

[0130] According to the display method of the present disclosure, a user can sequentially check the classification results of a machine learning model for multiple foods shown in an image of the interior of a storage cabinet.

[0131] REFERENCE SIGNS LIST 10 Food identification system 20 Refrigerator (storage) 30 Photography device 40 Image recognition server 50 Server device 51, 65 Communication unit 52, 63 Information processing unit 53, 64 Storage unit 60 Information terminal 61 Operation acceptance unit 62 Display unit 66 Acquisition unit 67 Decision unit 68 Display control unit 70 Wide area communication network

Claims

1. A display method executed by a computer, comprising: an acquisition step of acquiring information relating to detection frames assigned to each of a plurality of foods shown in an image of the interior of a storage cabinet having a cooling function for stored items, the detection frames indicating the identification results of the plurality of foods by a machine learning model; a determination step of determining the display order of the identification results of the plurality of foods based on the acquired information about the detection frames; and a display step of displaying a confirmation screen for a user to confirm whether the identification results of the plurality of foods are correct or not, wherein the identification results of the plurality of foods are displayed in order on the confirmation screen based on the determined display order.

2. The display method according to claim 1, wherein in the determining step, the display order is determined based on the size of the detection frame indicated by the information relating to the detection frame.

3. The display method according to claim 2, wherein in the determining step, the display order is determined so that the classification results are displayed in order from the food with the smallest detection frame size.

4. The display method according to claim 2, wherein the size of the detection frame is based on at least one of the length of a diagonal line of the detection frame, the area within the detection frame, and the perimeter of the detection frame.

5. The display method according to claim 2, wherein the detection frames are rectangular, and in the determining step, the display order is determined based on the size and aspect ratio of the detection frames indicated by the information relating to the detection frames.

6. The display method according to claim 1, wherein in the determination step, the display order is determined based on at least one of the degree of overlap of the detection frames, the identification score of the food to which the detection frame is assigned, and the food identification result, which are indicated by the information regarding the detection frames.

7. The display method according to claim 1, wherein the confirmation screen displays the image of the interior of the refrigerator, and the food to be confirmed that appears in the image of the interior of the refrigerator is displayed in a segmented manner.

8. The display method of claim 1, wherein the confirmation screen displays the image of the interior of the refrigerator with the detection frame superimposed on the food to be confirmed, and when the user confirms two or more foods to be confirmed at once, the detection frame is superimposed individually on each of the two or more foods to be confirmed on the confirmation screen.

9. The display method according to claim 8, wherein, when one detection frame is defined for two or more of the foods to be confirmed, the display method includes a step of generating detection frames from the one detection frame to be individually superimposed on each of the two or more foods to be confirmed.

10. The display method according to claim 1, further comprising a step of accepting designation information for allowing the user to confirm the identification results of some of the plurality of foods, and the identification results of some of the plurality of foods are displayed in order on the confirmation screen based on the determined display order and the accepted designation information.

11. A program for causing the computer to execute the display method according to any one of claims 1 to 10.

12. An information terminal comprising: an acquisition unit that acquires information relating to detection frames assigned to each of a plurality of foods that appear in an image of the interior of a storage cabinet that has a cooling function for stored items, the detection frames indicating the identification results of the plurality of foods by a machine learning model; a determination unit that determines the display order of the identification results of the plurality of foods based on the acquired information about the detection frames; and a display control unit that displays a confirmation screen for a user to confirm whether the identification results of the plurality of foods are correct or not, wherein the identification results of the plurality of foods are displayed in order on the confirmation screen based on the determined display order.

13. A food identification system comprising: an information terminal according to claim 12; and a server device that transmits information relating to the detection frame to the information terminal.

Images