Updating method, information processing system, display method, information terminal and program

Abstract

To provide an updating method capable of improving identification accuracy of a food product by using a machine learning model.SOLUTION: An updating method is an updating method for a machine learning model, which is executed by an image processing system 40. The updating method includes: a step S13 of outputting identification result information for displaying an interior image in which inside of a storage having a function for cooling a stored object is displayed and a detection frame superposed on a region determined to display a food product through a machine learning model of the interior image, on an information terminal 60; a step S17b of acquiring a result of a determination as to whether or not a food product is displayed within the detection frame made by a user; and a step S21b of causing the machine learning model to learn first correct answer data that associates predetermined identification information indicating that a food product is not displayed within the detection frame with a learning image including at least an image within the detection frame when the determination result indicates that a food product is not displayed within the detection frame.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to a method for updating a machine learning model that can identify food items in an image. [Background technology]

[0002] Conventionally, there have been proposed techniques relating to a storage cabinet for storing objects. Patent Document 1 discloses a refrigerator system equipped with a camera that captures images of the inside of a refrigerator compartment. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-168134 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides an information processing system that can improve the accuracy of food identification using a machine learning model. [Means for solving the problem]

[0005] An information processing system according to one aspect of the present disclosure is a method for updating a machine learning model executed by a computer, the method including the steps of: outputting first information for displaying on an information terminal an interior image showing the interior of a storage cabinet having a cooling function for stored items and a detection frame that is superimposed on an area of ​​the interior image that has been determined by the machine learning model to contain food; acquiring a user's determination result as to whether food is contained within the detection frame; and, if the determination result indicates that food is not contained within the detection frame, having the machine learning model learn first ground truth data that links predetermined identification information indicating that food is not contained within the detection frame with a learning image that includes at least an image within the detection frame. [Effects of the Invention]

[0006] An information processing system according to one aspect of the present disclosure can improve the accuracy of food identification using a machine learning model. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing the functional configuration of a food identification system according to an embodiment. [Figure 2] FIG. 2 is an external view of a refrigerator and a photographing device provided in the food identification system according to the embodiment. [Figure 3] FIG. 3 is a diagram showing an example of a confirmation screen for the food identification result. [Figure 4] FIG. 4 is a sequence diagram of an example 1 of the operation of updating a machine learning model. [Figure 5] FIG. 5 is a diagram showing an example of a confirmation screen on which a detection frame is superimposed due to an erroneous determination. [Figure 6] FIG. 6 is a sequence diagram of a second example of the operation of updating a machine learning model. [Figure 7] FIG. 7 is a diagram showing an example of an area excluding the detection frame in which food is captured. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection forms, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.

[0009] It should be noted that the drawings are schematic diagrams and are not necessarily strict illustrations. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and overlapping descriptions may be omitted or simplified.

[0010] (Embodiment) [composition] First, the configuration of the food identification system according to the embodiment will be described. Fig. 1 is a block diagram showing the functional configuration of the food identification system according to the embodiment.

[0011] The food identification system 10 shown in Fig. 1 is a system that can manage the types and quantities of food stored in a refrigerator 20 by identifying the types of food stored in the refrigerator 20 using images of the inside of the refrigerator 20. Specifically, the food identification system 10 includes a refrigerator 20, a photography device 30, an information processing system 40, an inventory management server 50, and an information terminal 60.

[0012] The refrigerator 20 is an example of a storage unit that can cool stored items, and is installed in a user's home or the like to refrigerate food. Figure 2 is an external view of the refrigerator 20 (and the image capturing device 30).

[0013] The photographing device 30 photographs an image of a drawer (for example, a vegetable drawer) of the refrigerator 20 that is pulled out from above the refrigerator 20. That is, the photographing device 30 photographs, for example, an image of the drawer of the refrigerator 20 viewed from above the refrigerator 20. The photographing device 30 is realized, for example, by a camera having a wide-angle lens and a telephoto lens.

[0014] The information processing system 40 is a computer located outside the facility where the refrigerator 20 is installed, and is specifically realized by one or more cloud servers. The information processing system 40 communicates with the image capturing device 30 via a wide area communication network 70 to acquire images of the interior of the refrigerator 20 captured by the image capturing device 30 and identify the type of food shown in the acquired images. The type of food may be, for example, a specific variety of vegetable (cucumber, tomato, etc.). In the following embodiment, an example of identifying the type of vegetable will be described, but the information processing system 40 can also identify the type of vegetable, type of meat, type of beverage, etc.

[0015] The information processing system 40 also presents the food type identification results to the user via the information terminal 60 and accepts instructions from the user to confirm or correct the presented identification results. The information processing system 40 includes a communication unit 41, an information processing unit 42, and a storage unit 43.

[0016] The communication unit 41 is a communication circuit that enables the information processing system 40 to communicate with the image capturing device 30 and the information terminal 60 via the wide area communication network 70. The communication unit 41 is, for example, a wired communication circuit that performs wired communication, but may also be a wireless communication circuit that performs wireless communication. There are no particular limitations on the communication standard of the communication performed by the communication unit 41.

[0017] The information processing unit 42 performs information processing and the like to present the food type identification results obtained by the information processing system 40 to the user via the information terminal 60. The information processing unit 42 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. The information processing unit 42 has, as functional components, an acquisition unit 44, an output unit 45, a learning unit 46, and a classification unit 47. The functions of the acquisition unit 44, the output unit 45, the learning unit 46, and the classification unit 47 are realized, for example, by the microcomputer or the like constituting the information processing unit 42 executing a computer program stored in the memory unit 43. The functions of the acquisition unit 44, the output unit 45, the learning unit 46, and the classification unit 47 will be described in detail below.

[0018] The storage unit 43 is a storage device that stores the computer program executed by the information processing unit 42, the machine learning model that identifies foods, and various information required for the above information processing. The storage unit 43 is realized by, for example, a semiconductor memory.

[0019] Inventory management server 50 is a computer located outside the facility where refrigerator 20 is installed, and is specifically realized by a cloud server. Inventory management server 50 manages the inventory of food items in refrigerator 20. Inventory management server 50 manages, for example, the quantity of food items by type.

[0020] The information terminal 60 is an information terminal owned by a user. The information terminal 60 displays the food type identification results obtained by the information processing system 40 and accepts instructions from the user to confirm or correct the displayed identification results. The information terminal 60 is, for example, a portable information terminal such as a smartphone or tablet terminal, but may also be a stationary information terminal such as a personal computer. The information terminal 60 includes an operation receiving unit 61, a display unit 62, an information processing unit 63, a storage unit 64, and a communication unit 65.

[0021] The operation reception unit 61 receives operations from the user. The operation reception unit 61 is realized by, for example, a touch panel, but may also be realized by a mouse and a keyboard.

[0022] A display screen such as that shown in Fig. 3 is displayed on the display unit 62. Fig. 3 is a diagram showing an example of a confirmation screen of the identification result of a certain food product by the information processing system 40. In other words, the confirmation screen of Fig. 3 is a display screen for receiving instructions from the user to confirm or correct the identification result. The display unit 62 is realized by a display panel such as a liquid crystal panel or an organic EL (Electro-Luminescence) panel, for example.

[0023] The information processing unit 63 performs information processing and the like to display a display screen such as that shown in Fig. 3. The information processing unit 63 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. The functions of the information processing unit 63 are realized, for example, by the microcomputer or the like constituting the information processing unit 63 executing a computer program stored in the storage unit 64.

[0024] The storage unit 64 is a storage device that stores the computer program executed by the information processing unit 63 and various information necessary for the information processing. The storage unit 64 is realized by, for example, a semiconductor memory. In order to display the display screen of Fig. 3, a predetermined application program is pre-installed in the storage unit 64, and the computer program executed by the information processing unit 63 includes such a predetermined application program.

[0025] The communication unit 65 is a communication circuit that enables the information terminal 60 to communicate with the information processing system 40 via the wide area communication network 70. The communication unit 65 is, for example, a wireless communication circuit that performs wireless communication, but may also be a wired communication circuit that performs wired communication. There are no particular limitations on the communication standard of the communication performed by the communication unit 65.

[0026] [Example 1 of machine learning model update behavior] Next, a description will be given of an update operation of a machine learning model that identifies food products, which is performed by the food identification system 10. Fig. 4 is a sequence diagram of an example 1 of an update operation of a machine learning model.

[0027] The communication unit 41 of the information processing system 40 communicates with the photographing device 30 to receive an image of the interior of the refrigerator 20 (hereinafter also referred to as an interior image) from the photographing device 30 (S11). The received interior image is stored in the memory unit 43.

[0028] The identification unit 47 performs object detection processing on the received fridge interior image to detect food appearing in the image and identify the type of food detected (S12). Specifically, the identification unit 47 uses a machine learning model (trained model) to detect objects (food) in the image and identify the detected food. As described below, a detection frame is assigned to the detected food.

[0029] The food type identification result is expressed, for example, by the food type and an identification score. The identification score is a score that indicates the likelihood of the identification result and is expressed between 0 and 1, with larger values ​​indicating higher likelihood. Likelihood can also be referred to as validity, accuracy, or reliability.

[0030] The food type classification result indicates, for example, n food types for one food (detection frame) in descending order of classification score, and the classification score for each of the n foods. n is a natural number greater than or equal to 2, and in the following embodiments, n=4. Specifically, the food classification result is information indicating the following: tomato: 0.4, broccoli: 0.2, lettuce: 0.15, pumpkin: 0.1. Below, the four foods included in the classification result are referred to as candidate foods, and the candidate food with the highest classification score among the candidate foods is referred to as the top candidate food.

[0031] The output unit 45 transmits (outputs) the identification result information for displaying the identification result to the information terminal 60 using the communication unit 41 (S13). The identification result information includes image information of the fridge interior image received in step S11, detection frame information indicating the position and size of the detection frame assigned to each of the multiple foods shown in the fridge interior image, and candidate food information for displaying the top four candidate foods for each of the multiple foods as options. The multiple foods shown in the fridge interior image are managed by food management IDs (e.g., numerical values), and the position and size of the detection frame are associated with the food management ID.

[0032] The communication unit 65 of the information terminal 60 receives the identification result information. Based on the received identification result information, the information processing unit 63 displays a confirmation screen (FIG. 3) for confirming whether the identification result of the food shown in the refrigerator interior image is correct (S14a).

[0033] The confirmation screen in Fig. 3 shows the identification result for one food item, and if it is determined that multiple foods are shown in one refrigerator interior image, a confirmation screen similar to that in Fig. 3 is displayed multiple times corresponding to the multiple foods. Specifically, the processes of step S14a and step S15a (described below) are repeated multiple times, and selection result information indicating the selection results for the multiple times is transmitted in step S16a (described below). For simplicity, the following description will be given assuming that the processes of step S14a and step S15a (described below) are performed only once.

[0034] An interior fridge image included in the identification result information is displayed in the upper part of the confirmation screen shown in Fig. 3, and a detection frame is superimposed on the interior fridge image based on the detection frame information included in the identification result information. In this way, the image information of the interior fridge image and the detection frame information can be said to be an example of first information for displaying the upper part of the confirmation screen in Fig. 3. The detection frame is also sometimes called a bounding box.

[0035] Furthermore, the lower part of the confirmation screen displays four candidate foods indicated by the candidate food information as options based on the candidate food information included in the identification result information. In other words, the candidate food information can be said to be an example of second information for displaying the lower part of the confirmation screen in Figure 3. The lower part of the confirmation screen is a display screen that displays the identification result (determination result) of the machine learning model regarding what food is displayed in the detection frame in the upper part of the confirmation screen, and is also an input screen for inputting the user's determination result regarding what food is displayed in the detection frame.

[0036] In the example of FIG. 3, the top candidate food of the four candidate foods is tomato, and the other three candidate foods are broccoli, lettuce, and pumpkin. As shown in FIG. 3, the top candidate food option (icon) with the highest identification score is displayed, for example, in a larger display size than the other three candidate food options. The other three candidate food options are displayed in a horizontal row below the top candidate food option. The candidate food option with the second highest identification score is displayed on the left, the candidate food option with the third highest identification score is displayed in the center, and the candidate food option with the fourth highest identification score is displayed on the right.

[0037] If the user determines that the food in the detection frame at the top of the confirmation screen corresponds to one of the four candidate foods, the user performs a selection operation at the bottom of the confirmation screen to select an option for the corresponding candidate food, and the operation receiving unit 61 receives the selection operation (S15a).

[0038] The information processing unit 63 communicates with the information processing system 40 using the communication unit 65, thereby transmitting selection results information to the information processing system 40 (S16a). The selection results information includes the food management ID of the food shown in the detection frame and the identification information of the food corresponding to the option selected by the user (e.g., onion). In other words, the selection results information is information indicating the user's identification result of the food shown in the detection frame of FIG. 3.

[0039] The communication unit 41 of the information processing system 40 receives the selection results information, and the acquisition unit 44 acquires the selection results information (S17a). The classification unit 47 determines the classification result based on the acquired selection results information (S18a), and the output unit 45 transmits (outputs) notification information to the inventory management server 50 using the communication unit 41 to notify that the food (onion) has been stored in the refrigerator 20 (S19a).

[0040] In addition, the learning unit 46 generates correct answer data that links the image within the detection frame corresponding to the food management ID included in the selection result information with the food identification information (correct answer label) (S20a), and updates the machine learning model by having the machine learning model learn the generated correct answer data (S21a).

[0041] In this way, in Example 1 of the machine learning model update operation, the information terminal 60 can reduce the effort required for the user to input the classification results by displaying the classification results of the target food by the machine learning model as options. Furthermore, the information processing system 40 can improve the classification accuracy of the machine learning model by having the machine learning model learn the classification results by the user as correct answer data.

[0042] If the user determines that the food in the detection frame at the top of the confirmation screen in Figure 3 does not correspond to any of the four candidate foods, the user can select the text input button 62a at the bottom of the confirmation screen in Figure 3 (the area displaying the words "Enter text") and manually input the user's identification results.

[0043] [Example 2 of machine learning model update behavior] The detection frame is superimposed on an area of ​​the refrigerator interior image that the machine learning model determines to contain food. However, due to an erroneous determination by the machine learning model, the detection frame may be superimposed on an area that does not actually contain food. Figure 5 shows an example of a confirmation screen on which a detection frame is superimposed due to an erroneous determination.

[0044] The following describes an operation for updating a machine learning model when detection frames overlap due to an erroneous determination in this way. Fig. 6 is a sequence diagram of an example 2 of the operation for updating a machine learning model.

[0045] The processing of steps S11 to S13 is the same as in example 1 of the machine learning model update operation, and therefore a detailed description thereof will be omitted.

[0046] Based on the received identification result information, the information processing unit 63 displays a confirmation screen (FIG. 5) for confirming whether the identification result of the food shown in the refrigerator interior image is correct (S14b).

[0047] When the user determines that no food is shown within the detection frame at the top of the confirmation screen, the user performs a skip operation by selecting the skip button 62b at the bottom of the confirmation screen (the area where the word "skip" is displayed), and the operation accepting unit 61 accepts the skip operation (S15b).

[0048] In this way, the skip button 62b is displayed at the bottom of the confirmation screen as an object for inputting that no food is shown within the detection frame. In other words, the bottom of the confirmation screen can be said to be an input screen for the user to input the result of their determination as to whether or not food is shown within the detection frame.

[0049] The information processing unit 63 communicates with the information processing system 40 using the communication unit 65 to transmit skip information to the information processing system 40 (S16b). The skip information includes a food management ID corresponding to the detection frame and predetermined identification information indicating that no food appears within the detection frame (in other words, predetermined identification information indicating that a background appears within the detection frame; in other words, a correct answer label). In other words, the skip information is information indicating the user's determination that no food appears within the detection frame in FIG. 5. Note that the predetermined identification information is, for example, identification information that is handled in parallel with the identification information of the food (such as onion).

[0050] The communication unit 41 of the information processing system 40 receives the skip information, and the acquisition unit 44 acquires the skip information (S17b). The recognition unit 47 determines the recognition result based on the acquired skip information (S18b). In this case, since no food was captured in the interior image, the inventory management server 50 is not notified that food has been stored in the refrigerator 20.

[0051] In addition, the learning unit 46 generates correct answer data that links the image within the detection frame corresponding to the food management ID included in the skip information with predetermined identification information that indicates that no food is shown (S20b), and updates the machine learning model by having the machine learning model learn the generated correct answer data (S21b).

[0052] In this way, in Example 2 of the machine learning model update operation, when the information processing system 40 acquires skip information indicating the user's determination that no food appears in the detection frame, the information processing system 40 causes the machine learning model to learn correct answer data that links predetermined identification information indicating that no food appears in the detection frame with the image in the detection frame. This can improve the identification accuracy of the machine learning model.

[0053] Note that the learning images used as the correct answer data are not limited to images within a detection frame that does not show food. For example, the entire area of ​​the fridge interior image excluding the detection frame that shows food may be used as a learning image. FIG. 7 is a diagram showing an example of the area excluding the detection frame that shows food, and the hatched area corresponds to the area excluding the detection frame that shows food. In FIG. 7, the hatched area, i.e., the learning image, includes an image within detection frame 62c that does not show food. In this way, it is sufficient for the learning image to include at least an image within detection frame 62c that does not show food.

[0054] [Modification of the detection frame display mode] In the above embodiment, the detection frame superimposed on the fridge interior image is realized by a frame line, but it may also be realized by changing the display mode of the fridge interior image inside and outside the detection frame.

[0055] For example, the detection frame may be realized by displaying the outside of the detection frame at a lower brightness than the inside of the detection frame, or by displaying the outside of the detection frame in a lighter color than the inside of the detection frame, or by displaying the outside of the detection frame with a higher transparency than the inside of the detection frame, or by superimposing a pattern only on the outside of the inside and outside of the detection frame.

[0056] The detection frame is not limited to a rectangular shape, and may be a circle or an ellipse. The image inside the detection frame may be enlarged and superimposed on the fridge interior image at the position of the detection frame.

[0057] [Object modification to indicate that no food is visible within the detection frame] In the above embodiment, a skip button is exemplified as an object for inputting that no food is shown in the detection frame. The object for inputting that no food is shown in the detection frame may be realized by text such as "next" or "advance," which allows the user to understand that the next screen will be displayed. Furthermore, the object for inputting that no food is shown in the detection frame may be realized by an image or icon such as an arrow or triangle, which allows the user to understand that the next screen will be displayed.

[0058] The object for inputting that no food is shown within the detection frame may be an image or icon of a food such as a vegetable with a diagonal line or a prohibition mark (a diagonal line within a circle) superimposed thereon.

[0059] [Other variations] In the above embodiment, the machine learning model is used to identify food items in an interior image of one refrigerator 20. In this case, the machine learning model has the advantage that it can be retrained using the interior image of one refrigerator 20, thereby being customized specifically for the user of that refrigerator 20.

[0060] The machine learning model may be shared by multiple refrigerators 20 and used to identify foods in images of the interiors of each of the multiple refrigerators 20. In other words, the food identification system 10 may include multiple sets of refrigerators 20 and image capture devices 30. In this case, the machine learning model can be retrained using a large amount of correct answer data, which has the advantage of enabling both generalization of food identification and improvement of identification accuracy.

[0061] In the above embodiment, refrigerator 20 may be a refrigerator for general household use, a refrigerator used for product display in a retail store such as a convenience store, or a refrigerator for other commercial use.

[0062] Furthermore, the refrigerator 20 is an example of a storage cabinet, and the present disclosure can also be realized as other storage cabinets that have a function of cooling stored items, such as a freezer. Furthermore, the items (in other words, objects) stored in the storage cabinet are not limited to food, and may be other items. Cases in which items other than food are stored in the refrigerator 20 are also conceivable.

[0063] [Effects, etc.] Below, examples of techniques that can be obtained from the disclosure of this specification will be given, and the effects and the like that can be obtained from these techniques will be described.

[0064] Technique 1 is a computer-implemented method for updating a machine learning model, including: step S13 of outputting first information for displaying on an information terminal 60 an interior image of a storage cabinet having a cooling function for stored items and a detection frame superimposed on an area of ​​the interior image determined by the machine learning model to contain food; step S17a or S17b of acquiring a user's determination result as to whether food is contained within the detection frame; and step S21b of training the machine learning model with first ground truth data that links predetermined identification information indicating that food is not contained within the detection frame with a training image that includes at least an image of the detection frame, if the determination result indicates that food is not contained within the detection frame. The image information and detection frame information of the interior image in the above embodiment are examples of first information.

[0065] This type of update method can improve the accuracy of food identification by having the machine learning model learn the first correct answer data.

[0066] Technique 2 is an updating method of Technique 1, which further includes step S21a of having the machine learning model learn second correct answer data that links the identification information of the food specified by the user with the image within the detection frame when the judgment result indicates that food is reflected within the detection frame.

[0067] This type of update method can improve the accuracy of food identification by having the machine learning model learn the second correct answer data.

[0068] Technique 3 is an updating method according to Invention 1 or 2, which includes a step of outputting second information for displaying an input screen for the user's determination result on the information terminal 60 when the refrigerator interior image with the detection frame superimposed is displayed, and on the input screen, multiple candidate foods that the machine learning model has determined may be in the detection frame are displayed as options, and an object for inputting that no food is in the detection frame is displayed. The candidate food information in the above embodiment is an example of the second information.

[0069] According to this update method, the user can easily confirm or correct the determination result (classification result) obtained by the machine learning model.

[0070] Technique 4 is a program for causing a computer to execute any one of the update methods of Techniques 1 to 3.

[0071] According to such a program, the computer can improve the accuracy of identifying food items by having the machine learning model learn the first correct answer data.

[0072] Technique 5 is an information processing system 40 including: an output unit 45 that outputs first information for displaying on an information terminal 60 an interior image showing the inside of a storage cabinet that has a cooling function for stored items and a detection frame that is superimposed on an area of ​​the interior image that has been determined by a machine learning model to show food; an acquisition unit 44 that acquires a user's determination result as to whether food is shown within the detection frame; and a learning unit 46 that, when the determination result shows that food is not shown within the detection frame, causes a machine learning model to learn first correct answer data that links predetermined identification information indicating that food is not shown within the detection frame with a learning image that includes at least an image within the detection frame.

[0073] Such information processing system 40 can improve the accuracy of food identification by having the machine learning model learn the first supervised data.

[0074] Technique 6 is a display method executed by a computer, and includes step S13 of receiving first information for displaying on an information terminal 60, by communicating with an information processing system 40, an interior image showing the interior of a storage cabinet having a cooling function for stored items and a detection frame superimposed on an area of ​​the interior image that has been determined by a machine learning model to contain food; step S14a or S14b of displaying the interior image with the detection frame superimposed based on the received first information; step S14a or S14b of displaying an input screen for the user's determination result as to whether or not food is contained in the detection frame; and step S16a or S16b of transmitting the input determination result to the information processing system 40, wherein the input screen displays an object for inputting that food is not contained in the detection frame.

[0075] According to this display method, the user can easily perform an operation to make the machine learning model learn the first supervised answer data.

[0076] Technique 7 is a program for causing a computer to execute the display method of technique 6.

[0077] According to such a program, the user can easily perform an operation to make the machine learning model learn the first supervised data.

[0078] Technology 8 relates to information terminal 60, which includes: a communication unit 65 that receives first information for displaying on an information terminal an interior image showing the inside of a storage cabinet that has a cooling function for stored items and a detection frame that is superimposed on an area of ​​the interior image that has been determined by a machine learning model to show food, by communicating with information processing system 40; a display unit 62 that displays the interior image with the detection frame superimposed based on the received first information, and displays an input screen for a user's determination result as to whether food is shown in the detection frame; and an information processing unit 63 that transmits the input determination result to information processing system 40 using communication unit 65, and the input screen displays an object for inputting that food is not shown in the detection frame.

[0079] According to such an information terminal 60, the user can easily perform an operation to make the machine learning model learn the first supervised answer data.

[0080] (Other embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0081] For example, in the above embodiments, the food identification system is realized by multiple devices, but it may also be realized by a single device. When the food identification system is realized by multiple devices, the components (particularly functional components) of the food identification system may be distributed in any way among the multiple devices. For example, some or all of the processes described in the above embodiments as being executed by the information processing system may be executed by an information terminal. Furthermore, some or all of the processes described in the above embodiments as being executed by the information terminal may also be executed by the information terminal.

[0082] Furthermore, for example, the communication method between the devices in the above-described embodiment is not particularly limited. Furthermore, a relay device (not shown) may be involved in the communication between the devices. Furthermore, the information transmission path described in the above-described embodiment is not limited to the transmission path shown in the sequence diagram.

[0083] For example, in the above embodiment, a process executed by a specific processing unit may be executed by another processing unit. Also, the order of multiple processes may be changed, or multiple processes may be executed in parallel.

[0084] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0085] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0086] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, etc. Furthermore, the general or specific aspects of the present disclosure may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0087] For example, the present disclosure may be realized as an updating method or a displaying method executed by a computer, or as a program for causing a computer to execute the updating method or the displaying method. The present disclosure may also be realized as a computer-readable non-transitory recording medium having such a program recorded thereon.

[0088] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure. [Industrial Applicability]

[0089] The method for updating a machine learning model disclosed herein can improve the accuracy of food identification by the machine learning model. [Explanation of symbols]

[0090] 10 Food Identification System 20 Refrigerator (storage) 30 Imaging equipment 40 Information Processing Systems 41, 65 Communications Department 42, 63 Information Processing Department 43, 64 Storage section 44 Acquisition Department 45 Output section 46 Learning Department 47 Identification section 50 Inventory Management Server 60 Information terminal 61 Operation reception section 62 Display section 62a Text input button 62b Skip button 62c Detection Frame 70 Wide Area Communication Network

Claims

1. 1. A computer-implemented method for updating a machine learning model, comprising: A step of outputting first information for displaying on an information terminal an interior image showing the interior of a storage cabinet having a cooling function for stored items and a detection frame superimposed on an area of ​​the interior image determined by the machine learning model to contain food; acquiring a user's determination result as to whether or not food is captured within the detection frame; and when the determination result indicates that no food is captured within the detection frame, causing the machine learning model to learn first correct answer data that links predetermined identification information indicating that no food is captured within the detection frame with a learning image that includes at least an image within the detection frame. How to update.

2. Further, when the determination result indicates that a food is captured within the detection frame, the method includes a step of causing the machine learning model to learn second correct answer data linking identification information of the food designated by the user with the image within the detection frame. The updating method according to claim 1 .

3. Further, when the fridge interior image on which the detection frame is superimposed is displayed, a step of outputting second information for displaying an input screen for the determination result by the user on the information terminal, On the input screen, a plurality of candidate foods that have been determined by the machine learning model to have a possibility of being shown in the detection frame are displayed as options, and an object for inputting that no food is shown in the detection frame is displayed. The updating method according to claim 1 .

4. A program for causing the computer to execute the update method according to any one of claims 1 to 3.

5. an output unit that outputs first information for displaying on an information terminal an interior image showing the interior of a storage cabinet having a cooling function for stored items and a detection frame that is superimposed on an area of ​​the interior image that is determined by a machine learning model to contain food; an acquisition unit that acquires a user's determination result as to whether or not food is captured within the detection frame; a learning unit that, when the determination result indicates that no food is shown within the detection frame, causes the machine learning model to learn first correct answer data that links predetermined identification information indicating that no food is shown within the detection frame with a learning image that includes at least an image within the detection frame. Information processing system.

6. 1. A computer-implemented display method comprising: A step of receiving first information for displaying on an information terminal an interior image showing the interior of a storage cabinet having a cooling function for stored items and a detection frame superimposed on an area of ​​the interior image determined by a machine learning model to contain food by communicating with an information processing system; Displaying the interior image with the detection frame superimposed based on the received first information; displaying a screen for inputting a result of a user's determination as to whether or not food is captured in the detection frame; transmitting the input determination result to the information processing system; an object for inputting that no food is shown within the detection frame is displayed on the input screen; Display method.

7. A program for causing the computer to execute the display method according to claim 6.

8. A communication unit that receives first information for displaying on an information terminal an interior image showing the interior of a storage cabinet having a cooling function for stored items and a detection frame that is superimposed on an area of ​​the interior image that is determined by a machine learning model to contain food by communicating with an information processing system; A display unit that displays the fridge interior image with the detection frame superimposed based on the received first information and displays an input screen for a user's determination result as to whether or not food is reflected in the detection frame. an information processing unit that transmits the input determination result to the information processing system using the communication unit; an object for inputting that no food is shown within the detection frame is displayed on the input screen; Information terminal.

Images