Image selection device, image selection method, and storage system

The image selection device enhances food type recognition in refrigerators by identifying and utilizing low-accuracy images for additional learning, thereby improving the model's accuracy and efficiency.

JP2024157387APending Publication Date: 2024-11-07HITACHI GLOBAL LIFE SOLUTIONS INC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2023071723
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-25
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

Existing systems for inferring food types in refrigerators rely heavily on machine learning with large quantities of images, but do not effectively select learning data to improve recognition accuracy, leading to suboptimal performance.

Method used

An image selection device that identifies and selects low-accuracy images and items for additional learning, using a recognition inference processing unit to enhance the model's accuracy by incorporating images with low recognition rates and adjusting the learning dataset accordingly.

Benefits of technology

Improves the recognition accuracy of food types in refrigerators by selectively focusing on low-accuracy images and items, enhancing the model's performance through targeted data augmentation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024157387000001_ABST
    Figure 2024157387000001_ABST
Patent Text Reader

Abstract

To enable efficient selection of learning data to improve the accuracy of recognition results.SOLUTION: An image selection device of the present invention comprises a recognition inference processing unit for outputting the recognition accuracy of captured images of items, and an image selection unit configured to select a low-accuracy item with recognition accuracy that is low enough to satisfy a given standard from the multiple images, so as to select a low-accuracy image whose recognition accuracy for the low-accuracy item is low enough to satisfy a given standard from among images containing the low-accuracy item.SELECTED DRAWING: Figure 7
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an image selection device, an image selection method, and a repository system. [Background technology]

[0002] Recently, refrigerators that automatically infer what ingredients are stored inside are becoming more common. In Patent Document 1, multiple users who share the same refrigerator each operate a terminal device that stores feature information for each ingredient, and if one terminal device cannot recognize the type of ingredient, it requests another terminal device to recognize the type of ingredient. In this way, even if one terminal device cannot recognize the type of ingredient, the other terminal device can recognize the type of ingredient. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-196636 Summary of the Invention [Problem to be solved by the invention]

[0004] In general, to infer the type of an ingredient, a large number of images of the ingredient must be used as training data, and a model must be machine-learned to input the image and output the ingredient type. The accuracy of recognition is heavily dependent on the quantity and quality of the training data. It is important to select training data that will truly contribute to improving recognition results. However, Patent Document 1 does not mention the selection of learning data in order to improve the accuracy of the recognition result. [Means for solving the problem]

[0005] The image selection device of the present invention is characterized by comprising a recognition inference processing unit that outputs the recognition accuracy of an image of an item, and an image selection unit that selects low-accuracy items from a plurality of images whose recognition accuracy is low enough to satisfy a predetermined standard, and selects low-accuracy images from among images containing the low-accuracy items whose recognition accuracy is low enough to satisfy the predetermined standard. [Brief description of the drawings]

[0006] [Figure 1] FIG. 2 is a diagram illustrating the configuration of a refrigerator and an image selection device. [Diagram 2] 13 is an example of an average recognition accuracy list. [Diagram 3] FIG. 13 is a diagram showing an overall view of the processing of an image selection unit. [Figure 4] 11A and 11B are diagrams illustrating a low-accuracy image automatic selection process. [Diagram 5] FIG. 13 is a diagram showing the relationship between the number of low-accuracy items and the importance of additional learning. [Figure 6] 4 is a diagram illustrating details of a detection target information processing unit. FIG. [Figure 7] 13 is a flowchart of an image selection process. [Figure 8] This is a flowchart that adds the acquisition of low-precision item information and the output of additional learning importance to Figure 7. [Figure 9] 9 is a flowchart in which acquisition of detection target information is added to the flowchart in FIG. 8. [Figure 10] FIG. 2 is a diagram illustrating the configuration of a mobile terminal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, modes for carrying out the present invention (referred to as embodiments) will be described. (Model accuracy and quality of training data) One model takes an image of an ingredient as input and outputs the type of ingredient (egg, tofu, milk, natto, ...). This model is, for example, a neural network with an input layer, multiple intermediate layers, and an output layer. A propagation parameter is set for each node in the layer, which determines how much information to pass on to which node in the next layer. If this propagation parameter is a random initial value, it is unlikely that the model will output the true ingredient type.

[0008] Therefore, it is common to use training data to machine-learn (optimize) the model's propagation parameters. The training data here is a huge collection of images containing one or more ingredients, and is "supervised training data" in which each image is given a "label" (correct answer) verified by human eyes.

[0009] Hereinafter, the present embodiment will be described with reference to the drawings. The present embodiment is an example of recognizing the type of food stored in a refrigerator. However, the present invention is more generally applicable to storage cabinets including room temperature storage cabinets, warming cabinets, refrigerators, etc. The food ingredients in the present embodiment are a representative example of goods in a broader sense. The food ingredients may be ingredients, cooked food, or seasonings.

[0010] In this embodiment, an example will be described in which a camera captures an image of ingredients. However, a weight sensor, a mycotoxin (mold) detection sensor, an IC tag reader, a character reader, etc. may be used instead of or in addition to the camera.

[0011] In this embodiment, the recognition inference processing unit recognizes the presence of ingredients in the storage area, the usage status of ingredients consumed by the user, etc. In this embodiment, the recognition inference processing unit performs recognition using a camera image acquired via the image data processing unit, but the recognition inference processing unit may also perform recognition in response to input from a user to a mobile terminal such as a smartphone and the storage unit itself. This embodiment includes Example 1 and Example 2.

[0012] Example 1 1 is a diagram illustrating the configuration of a refrigerator and an image selection device. A refrigerator 100 is connected to a mobile terminal 115 and a computer 116 via a network 117.

[0013] The refrigerator 100 includes an image selection device 101 and a refrigerator main body 118. The refrigerator 100 is an example of a storage unit. The mobile terminal 115 as an external storage device is an information processing device such as a tablet, a smartphone, or a personal computer used by a user of the refrigerator 100. Furthermore, main processing of this embodiment may be shared and executed by the image selection device 101 and the mobile terminal 115. The computer 116 delivers various types of information, machine learning models, etc. to the image selection device 101.

[0014] 1 shows a configuration in which refrigerator 100 includes image selection device 101. However, image selection device 101 may be configured independent of refrigerator 100. In this case, refrigerator (storage) 100 and image selection device 101 configure a storage system.

[0015] A camera (not shown) for photographing the interior of refrigerator 100 is attached. The camera may be located inside or outside the refrigerator. The number of cameras is an arbitrary number equal to or greater than one. More generally, the camera does not necessarily have to be attached to refrigerator 100, and may be incorporated in mobile terminal 115, for example.

[0016] The image selection device 101 that controls the refrigerator 100 includes, for example, a processor 113, a storage device 102, a communication unit 114 connected to a network 117, and an input / output interface 112. The storage device 102 is a main storage device composed of a volatile or non-volatile memory, and an auxiliary storage device composed of a flash memory, a hard disk drive, or the like.

[0017] The image selection device 101 can also transmit some or all of the computer programs and data stored in the storage device 102 to the outside via the network 117. Conversely, the image selection device 101 can receive computer programs and data from an external computer 116 or the like via the network 117 and store them in the storage device 102.

[0018] The image selection device 101 can also transmit and receive part or all of the computer programs and data to and from a storage medium 111, such as a flash memory or a hard disk drive, connected to the image selection device 101.

[0019] The storage device 102 stores an image data processing unit 103, a recognition inference processing unit 104, an image selection unit 105, an importance index output unit 106, a detection target information processing unit 107, a data set update unit 108, a model learning unit 109, and an interior control unit 110. These are computer programs that realize predetermined functions.

[0020] The processor 113 executes each of these computer programs to realize each function. Therefore, each of these units can be read as each computer program, and the processor 113 realizes the processing and functions of each unit described below in cooperation with each computer program. However, each of these units may be realized by dedicated hardware, an FPGA (Field Programmable Gate Array), or the like. Furthermore, the computer programs may be configured in fewer numbers than those shown in the figure. In this case, each unit may be configured as a computer module.

[0021] As described above, the processor 113 operates as a functional unit that provides a predetermined function by executing processing according to a computer program. For example, the processor 113 functions as the image data processing unit 103 by executing processing according to an image data processing program. The same applies to other computer programs. Furthermore, the processor 113 also operates as a functional unit that provides each function of a plurality of processes executed by each computer program. Note that, although one processor 113 executes the computer program in this embodiment, a plurality of processors may execute the computer program.

[0022] The image data processing unit 103 acquires images from the refrigerator main body 118 via the input / output interface 112, and stores the acquired images in the storage device 102. When the image data processing unit 103 acquires images taken with a fisheye or wide-angle lens, it may convert them into planar images (standard images without distortion). Technology for converting images taken with a fisheye or wide-angle lens into planar images is well known. The supervised learning data may be created by manual annotation work (processing of attaching information called tags or metadata to data) or by automatically outputting annotations by machine learning.

[0023] The recognition and inference processing unit 104 recognizes ingredients in the storage area from an image captured of the storage area. At this time, the recognition and inference processing unit 104 inputs the image to a model that has undergone machine learning such as deep learning in advance, and causes the model to output the type of ingredient. Each image input to the model simultaneously captures multiple ingredients, such as egg, natto, milk, tofu, .... Therefore, the model outputs multiple character strings (items), such as egg, natto, milk, tofu, ..., as a recognition result. Although the explanation of FIG. 1 is in the middle of being given, the explanation will now move to FIG. 2.

[0024] FIG. 2 is an example of the average recognition accuracy list 201. The recognition inference processing unit 104 lists the average recognition accuracy (average recognition accuracy) for each item that appears in multiple images included in the supervised learning data processed by the model, as shown in FIG. 2. That is, the recognition inference processing unit 104 inputs multiple images that constitute the supervised learning data, which is a collection of combinations of images and items (labels), to the model. The model then outputs the items. The output items are not necessarily the same as the items (labels) attached to the images. Therefore, the average recognition accuracy for each item is defined. The average recognition accuracy list 201 lists the average recognition accuracy for each item. An item is a common noun (egg, tofu, milk, natto, ...) that indicates a type of item including ingredients.

[0025] An index of average recognition accuracy is, for example, the F-score (= harmonic mean or weighted mean of precision and recall). Precision is the proportion of images that are actually ingredient A among those that are recognized as ingredient A. Recall is the proportion of images that are actually ingredient A that are correctly recognized as ingredient A among those that are actually ingredient A. If you want to reduce false positives, you should focus on precision, and if you want to reduce non-detections, you should focus on recall. Note that instead of the F-score, precision, recall, accuracy, specificity, uncertainty, AUC (Area Under the Curve), etc. may be used as recognition accuracy. Refer back to Figure 1 for explanation.

[0026] The recognition inference processing unit 104 of this embodiment recognizes ingredients in the refrigerator 100 using images, but is not limited to this. When information other than images is used, the image data processing unit 103 may be omitted. Instead, a configuration for use in recognition by the recognition inference processing unit 104 is provided separately. For example, when a weight sensor is used, a functional unit is provided that performs processing to associate weight with ingredients.

[0027] The recognition inference processing unit 104 can update the model to the latest version in response to changes or additions to packaging of ingredients. For example, the recognition inference processing unit 104 may update the model to be used by itself to a model received from a server (not shown).

[0028] In general, "recognition" refers to outputting the type of an object. "Learning" refers to optimizing a model when the model performs recognition. "Inference" refers to a trained model automatically outputting the type of an object. Since this embodiment is an example in which a model is used, there is no essential difference between "recognition" and "inference." The recognition and inference processing unit 104 is named as such, in the sense that a trained model performs recognition (=inference).

[0029] The image selection unit 105 automatically selects low-accuracy images suitable for improving the average recognition accuracy of items with low average recognition accuracy. Although the explanation of FIG. 1 is in the middle of being given, the explanation will now be shifted to FIG.

[0030] FIG. 3 is a diagram showing an overall picture of the processing of the image selection unit 105. The processing of the image selection unit 105 consists of two steps (corresponding to two arrows): low-accuracy item automatic selection processing 306 and low-accuracy image automatic selection processing 307. Circle 301 indicates multiple images, and circle 302 is the result of the image selection unit 105 selecting items with low average recognition accuracy (low-accuracy items) from the multiple images. Circle 303 is the result of the image selection unit 105 selecting an image with lower accuracy (low-accuracy image) from among images including low-accuracy items. This allows the image selection unit 105 to select low-accuracy images that include many low-accuracy items.

[0031] That is, the image selection unit 105 can improve the recognition accuracy of low-accuracy items by selecting an image suitable for improving the recognition accuracy of low-accuracy items. The size of the circles 301, 302, and 303 indicates the scale of the number of images, and the larger the circle, the greater the number of images. Also, the dashed rectangle 304 indicates a high-accuracy item, and the solid rectangle 305 indicates a low-accuracy item.

[0032] In the low-accuracy item automatic selection process 306, the image selection unit 105 selects low-accuracy items with low average recognition accuracy (average recognition accuracy<β) from multiple images. "β" is a threshold. In the example of the average recognition accuracy list 201 in FIG. 2, for example, β=80%. In this case, the image selection unit 105 automatically selects milk and natto, which have average recognition accuracy<80%, as low-accuracy items (reference numeral 202). Next, in the low-accuracy image automatic selection process 307, the image selection unit 105 selects images in which the image of the part of the item that appears therein has low accuracy (recognition accuracy<β) from among the images including the low-accuracy item. The description will now move on to FIG.

[0033] FIG. 4 is a diagram for explaining the low-accuracy image automatic selection process. In FIG. 4, an "egg-containing image" is an image in which eggs are included among the multiple items (correct answer labels) that appear in the image. Similarly, an "image with tofu included" is an image in which tofu is included among the multiple items that appear in the image. The same applies to other food ingredients. Therefore, a certain egg-containing image may also be a tofu-containing image at the same time.

[0034] Egg-containing image 401 includes multiple (four in this example) squares 402. Squares 402 represent individual egg-containing images. The percentages attached to squares 402 indicate the recognition accuracy of the eggs captured in the images. The same applies to other items. Even if the same individual egg is captured in multiple egg-containing images, the position, brightness of the image, interference with other food ingredients, and other conditions will vary. Thus, the recognition accuracy of the egg differs for each egg-containing image. Also, for example, if the correct answer is an egg but it is mistakenly detected as a plastic bottle, this will result in a decrease in the recognition accuracy of the egg (F value, etc.), and so the recognition accuracy of the egg will also vary depending on the condition of the other items. The same applies to other items.

[0035] Dotted line 403 includes images containing milk and images containing natto. Milk and natto are items with low average recognition accuracy (low-accuracy items) (see FIG. 2). "□65%" within dashed line 404 indicates that the recognition accuracy of milk among those images containing milk is 65%. "□75%", "□62%", and "□58%" within dashed line 405 indicate that the recognition accuracy of natto among the three images containing natto is 75%, 62%, and 58%, respectively. It is highly likely that the images containing milk within dashed line 404 are lowering the average recognition accuracy of milk. It is highly likely that the images containing natto within dashed line 405 are lowering the average recognition accuracy of natto.

[0036] The image selection unit 105 selects milk and natto as low-accuracy items from the average recognition accuracy list 201 (Fig. 2). In Fig. 4, the image selection unit 105 selects, from among the images containing milk, images within a dashed line 404 for which the recognition accuracy of milk is <80%, as low-accuracy images. Similarly, from among the images containing natto, the image selection unit 105 selects, from among the images containing natto, images within a dashed line 405 for which the recognition accuracy of natto is <80%, as low-accuracy images.

[0037] The average recognition accuracy is defined for an item. In contrast, the percentages in FIG. 4 marked "recognition accuracy for circle" and associated with squares are "recognition accuracy" defined for a specific item that appears in the image. The threshold value β applied to the average recognition accuracy may be the same as or different from the threshold value β applied to the recognition accuracy. The recognition inference processing unit 104 is the entity that calculates (outputs) the average recognition accuracy and the recognition accuracy. The image selection unit 105 is the entity that selects low-accuracy items and low-accuracy images. The image selection unit 105 may also cut out an image of only the portion of the image that includes a low-accuracy item, in which the item appears, and select the cut-out image as a low-accuracy image. Return to FIG. 1 for explanation.

[0038] The image selection unit 105 outputs low-accuracy item information for each of the low-accuracy images selected by itself. The importance index output unit 106 outputs the importance of additional learning for each image based on the low-accuracy item information. The low-accuracy item information is, for example, the number of low-accuracy items (number of low-accuracy items) that appear in an image. The greater the number of low-accuracy items, the higher the importance of additional learning for that image. Although we are still in the middle of explaining Figure 1, the explanation will now move to Figure 5.

[0039] FIG. 5 is a diagram for explaining the relationship between the number of low-accuracy items and the importance of additional learning. In FIG. 5, the items included (shown) in the selected low-accuracy image 501 are four: eggs, tofu, milk, and natto. Of these, the only item with a recognition accuracy <80% is natto. In this case, the number of low-accuracy items in the selected low-accuracy image 501 is "1". The items included in the selected low-accuracy image 502 are also four: eggs, tofu, milk, and natto. Of these, the only two items with a recognition accuracy <80% are milk and natto. In this case, the number of low-accuracy items in the selected low-accuracy image 502 is "2".

[0040] As a result, the "selected low-accuracy image" 502 with a high additional learning importance is preferentially used as supervised learning data for additional learning. In addition, the additional learning importance is not limited to being calculated according to the number of low-accuracy items, but may be calculated according to the high or low recognition accuracy. For example, two low-accuracy images are selected, and the recognition accuracy of natto in one of them is 58%, and the recognition accuracy of natto in the other is 78%. In this case, the additional learning importance of the former may be higher. In addition, the recognition accuracy of "tofu" that happens to appear in the selected low-accuracy image 502 is less than 80%. However, tofu is not a low-accuracy item, so it is not counted in the number of low-accuracy items here. Of course, tofu may also be counted in the number of low-accuracy items. The explanation returns to FIG. 1.

[0041] Ultimately, the image selection unit 105 narrows down the images to be re-learned to those that include items with low recognition accuracy, and then narrows down the images to those that actually have low recognition accuracy in the parts that include the items. The importance index output unit 106 increases the importance of additional learning for images that can kill two birds with one stone, so to speak, from among the narrowed down images.

[0042] The detection target information processing unit 107 outputs detection target information (described in detail later) based on the detection frame output by the recognition inference processing unit 104. In this case, the importance index output unit 106 outputs the additional learning importance based on the detection target information and the low-accuracy item information. Although the explanation of FIG. 1 is in progress, the explanation will now move to FIG. 6.

[0043] 6 is a diagram illustrating details of the detection target information processing unit 107. The detection target information processing unit 107 has an over-detection module 601, a color detection module 602, a low image quality detection module 603, a human body detection module 604, a local detection module 605, a main body detection module 606, and an unlearned detection module 607.

[0044] When the same ingredient is detected excessively (duplicated), the excessive detection module 601 detects the detection frame as overlap ratio information. The color detection module 602 detects, as color information, a detection frame in which the surroundings and the food are merged or a detection frame in which the food is reflected in a dark area because the surroundings have a similar color to the food. The low image quality detection module 603 outputs the resolution within the detection frame, and detects the detection frame captured through the transparent shelf and the detection frame of the blurred image as definition information.

[0045] The human body detection module 604 detects a human body and detects a detection frame such as a human hand as environmental information. The local detection module 605 detects, as area information, a detection frame in which the area of ​​the ingredient is small and the ingredient overlaps with its surroundings, so that only a part of the ingredient is captured. The main body detection module 606 detects the partition boards and the like of the refrigerator main body 118, and detects a detection frame in which the storage cabinet main body including the partition boards is reflected as main body information. The unlearned detection module 607 detects a detection frame of a feature that has not been learned in advance as feature distance information, based on the feature distance information.

[0046] The overlap ratio information, color information, resolution information, environment information, area information, main body information, and feature distance information are collectively referred to as detection target information. Detection target information is information indicating the quality of an image, and is information indicating that the item in the image (detection frame) may not be correctly recognized due to a qualitative defect. The detection target information processing unit 107 sets the additional learning importance of the image corresponding to the detection target information to be high, thereby making it easier to apply low-precision images to additional learning preferentially. It is not necessary to use all of the seven modules as the module that outputs the detection target information, and at least one of them may be used. In addition, the module that outputs the detection target information is not limited to only the seven modules, and other modules may be added, deleted, or replaced as appropriate. The explanation returns to FIG. 1.

[0047] The dataset update unit 108 adds the low-precision images selected by the image selection unit 105 or the importance index output unit 106 to the supervised learning dataset. The supervised learning dataset is the largest population of images that can become supervised learning data for machine learning a model that recognizes ingredients. Note that the dataset update unit 108 may increase the amount of data of the low-precision images selected by the image selection unit 105 or the like by performing data extension such as rotation, shift, and horizontal flip before adding the low-precision images to the supervised learning dataset.

[0048] The model learning unit 109 uses a supervised learning dataset by deep learning or the like to machine-learn a model that recognizes ingredients in the refrigerator. The model learning unit 109 may increase the number of learning times by cross-validation in order to improve the generalization performance of the learning model and suppress overlearning. The model learning unit 109 may further fine-tune or transfer learning the learned model and re-learn the model using unused parts of the supervised learning dataset, etc., to improve the generalization performance. The model learning unit 109 may update an incorrect label to a correct label as necessary.

[0049] For example, if the label "tomato" is attached to an image of an apple that is as shiny as a tomato, the model learning unit 109 updates the label to "apple." If the label "cherry tomato" is attached to an image of a cherry tomato that is shaped like a strawberry, the model learning unit 109 does not update the label and increases the number of learning iterations, although this is a severe example.

[0050] The interior control unit 110 controls a motor and a compressor (not shown) to control the temperature and humidity inside the refrigerator 100.

[0051] FIG. 7 is a flowchart of the image selection process. In step S201, the image data processing unit 103 acquires an image captured by a camera from existing supervised learning data, and converts a fisheye image into a planar image as necessary.

[0052] In step S202, the recognition inference processing unit 104 calculates an average recognition accuracy for all items based on the multiple images. The recognition inference processing unit 104 calculates, for example, an F value such that "weighted precision = weighted recall" as the average recognition accuracy. In other words, the recognition inference processing unit 104 does not necessarily need to calculate an F value such that "precision = recall", and may calculate an F value such that "precision ≧ recall" or an F value such that "precision ≦ recall". The precision, recall, and F value may be replaced with accuracy, specificity, uncertainty, AUC, etc. Note that the "prescribed recognition accuracy" is a concept that includes precision, recall, F value, accuracy, specificity, uncertainty, and AUC.

[0053] In step S203, the image selection unit 105 determines whether the average recognition accuracy for all items is β or more, and determines whether or not to perform machine learning on the model iteratively. If the average recognition accuracy for all items is not β or more (step S203 "No"), the image selection unit 105 proceeds to S204. If the average recognition accuracy for all items is β or more (step S203 "Yes"), the image selection unit 105 displays (outputs) the low-accuracy images selected in the previous iterative process on an arbitrary device (e.g., the mobile terminal 115, the input / output interface 112, etc.), and then ends the image selection process. In addition, the image selection unit 105 may use, as an index for determining whether or not to perform machine learning on an iterative model, whether the number of low-accuracy items = 0 or the number of low-accuracy image selections = 0, in addition to whether the average recognition accuracy for all items is β or higher, and proceed to step S204 if none of the above indexes is met.

[0054] In step S204, the image selection unit 105 selects low-accuracy items (average recognition accuracy<β) from the multiple images. In step S205, the image selection unit 105 selects a low-accuracy image (recognition accuracy<β) from among images including low-accuracy items. At this time, the recognition inference processing unit 104 calculates the recognition accuracy. In step S206, the dataset update unit 108 adds the low-precision images selected in step S205 to the training dataset. The dataset update unit 108 may perform the data augmentation described above at this stage.

[0055] In step S207, the model learning unit 109 uses the learning data set added in step S206 to machine-learn a model that recognizes ingredients. The learning data set used at this stage is one to which "teacher data" has been added by the manufacturer, the user and / or machine learning. At this stage, the model learning unit 109 may perform the above-mentioned fine tuning and transfer learning. After the process of step S207, the process returns to step S202.

[0056] In the repeated process of steps S202 to S207, the threshold value β applied to the average recognition accuracy for all items corresponds to a "predetermined recognition accuracy." Therefore, the image selection unit 105 repeats the process of selecting supervised learning data from the images until the results of the machine learning model recognizing the item satisfy the predetermined recognition accuracy. Accordingly, the model learning unit 109 repeats machine learning the model until the average recognition accuracy for all items reaches a certain level or higher, thereby constructing a model that can recognize low-precision items with high accuracy.

[0057] As described above, in this embodiment, low-accuracy items with low average recognition accuracy are selected from multiple images for each additional learning, and an image with a lower accuracy is selected from among images that include low-accuracy items, thereby making it possible to select images that are suitable for improving the recognition accuracy of low-accuracy items.

[0058] Figure 8 is a flowchart in which the acquisition of low-precision item information and the output of additional learning importance are added to Figure 7. The differences from Figure 7 will be mainly explained in Figure 8. The characteristic process of Figure 7 is the output of additional learning importance based on the low-precision item information in the selected low-precision image.

[0059] Steps S201 to S205, S206, and S207 in Fig. 8 are similar to steps S201 to S205, S206, and S207 described in Fig. 7, and therefore will not be described here. Steps S301 to S303 are newly added in Fig. 8.

[0060] In step S301, the image selection unit 105 acquires low-accuracy item information, such as the number of low-accuracy items included in the low-accuracy image selected in step S205. In step S302, the importance index output unit 106 outputs the additional learning importance to an arbitrary device based on the low-accuracy item information.

[0061] In step S303, the image selection unit 105 selects low-precision images to be added to the learning dataset according to the importance of additional learning. That is, the image selection unit 105 prioritizes images with higher importance of additional learning to be added to the learning dataset. As described above, in this embodiment, the lower-accuracy images with a higher additional learning importance output based on low-accuracy item information are more suitable for improving the recognition accuracy of low-accuracy items, and are therefore more likely to be added to the learning dataset with priority.

[0062] Fig. 9 is a flowchart in which the acquisition of detection target information is added to Fig. 8. In Fig. 9, the differences from Fig. 8 will be mainly explained. The characteristic process of Fig. 9 is to output the importance of additional learning based on the low-accuracy item information and detection target information in the selected low-accuracy image.

[0063] Steps S201 to S205, S206, and S207 in FIG. 9 are similar to steps S201 to S205, S206, and S207 described in FIG. 7, and steps S301 and S303 in FIG. 9 are similar to steps S301 and S303 described in FIG. 8, so their description will be omitted.

[0064] In step S401, the detection target information processing unit 107 acquires detection target information such as overlap ratio information, color information, and definition information. In step S402, the importance index output unit 106 outputs the additional learning importance to an arbitrary device based on the low-accuracy item information and the detection target information. The importance index output unit 106 calculates the additional learning importance using a formula such as "additional learning importance = number of low-accuracy items x number of corresponding detection target information".

[0065] As described above, in this embodiment, the more an image corresponds to detection target information such as overlap ratio information, color information, and resolution information, the lower the accuracy of the image, and by using the additional learning importance output based on the low-accuracy item information and the detection target information, it is possible to select images that are suitable for improving the recognition accuracy of low-accuracy items and / or low-accuracy images.

[0066] Example 2 In the first embodiment, the image selection process is performed by an image selection device 101 that is a part of the refrigerator 100. However, the image selection process may be performed by a component in a separate housing independent of the refrigerator 100. A modified example in which the image selection process is performed by a mobile terminal 115 will be described below.

[0067] 10 is a diagram illustrating the configuration of the mobile terminal 115. The mobile terminal 115 includes a processor 902, a storage device 904, a touch panel 901, and a communication unit 903. The mobile terminal 115 is a computer such as a smartphone.

[0068] The processor 902 and the storage device 904 have the same functions as the processor 113 and the storage device 102 shown in Fig. 1. The touch panel 901 functions as an input / output unit. The communication unit 903 is connected to the network 117. This connection may be wireless or wired.

[0069] The storage device 904 stores an image selection program 912 for executing the processing of the embodiment 2. The image selection program 912 is composed of an image data processing module 905, a recognition and inference processing module 906, an image selection module 907, an importance index output module 908, a detection target information processing module 909, a data set update module 910, a model learning module 911, and an in-fridge control module 913. Some of these modules may be combined to form a larger module.

[0070] For example, the image data processing module 905 in Fig. 10 has the same functions as the image data processing unit 103 in Fig. 1. The same applies to the other modules. However, it is desirable that the in-chamber control module 913 also manages the usage status of ingredients in the chamber. For example, the in-chamber control module 913 obtains information input by the user and information read from the code of the ingredient, and the recognition and inference processing module 906 recognizes the ingredient. It is preferable that the image selection program 912 is distributed to the mobile terminal 115 via the network 117. For this reason, the network 117 is realized by the Internet.

[0071] As described above, according to the first and second embodiments, a low-accuracy item having a low average recognition accuracy is selected from a plurality of images for each additional learning, and a low-accuracy image having a lower accuracy is selected from among images including low-accuracy items, thereby making it possible to select an image suitable for improving the recognition accuracy of the low-accuracy item. In addition, an image having a higher additional learning importance output based on low-accuracy item information is deemed to be suitable for improving the recognition accuracy of the low-accuracy item, and is more likely to be preferentially added to the learning data set. This makes it possible to select an image suitable for improving the recognition accuracy of the low-accuracy item. By using the additional learning importance output based on the low-accuracy item information and the detection target information, an image suitable for improving the recognition accuracy of the low-accuracy item and / or the low-accuracy image can be selected. Furthermore, according to the first and second embodiments, the processing load (load) of the image selection device can be reduced by limiting the images to be processed.

[0072] It should be noted that the present invention (image selection device or repository) is not limited to the above-described embodiment, but includes various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to an embodiment having all of the configurations described. It is also possible to add the configuration of one embodiment to the configuration of another embodiment. It is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0073] In addition, the above-mentioned configurations, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a recording device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD. In addition, the control lines and information lines are those considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it may be considered that almost all components are connected to each other. Various information of the image selection device may be stored on the cloud. [Explanation of symbols]

[0074] 100 Refrigerator 101 Image Selection Device 102 Storage device 103 Image data processing section 104 Recognition and inference processing unit 105 Image Selection Department 106 Importance index output part 107 Detection target information processing section 108 Dataset Update Section 109 Model Learning Department 110 Storage control unit 111 Storage medium 112 Input / Output Interface 113 Processors 114 Communications Department 115 Mobile Devices 116 Calculator 117 Network 901 Touch Panel 902 Processor 903 Communications Department 904 Storage device 905 Image Data Processing Module 906 Recognition and Inference Processing Module 907 Image Selection Module 908 Importance Index Output Module 909 Detection target information processing module 910 Dataset Update Module 911 Model Learning Module 912 Image Selection Program 913 In-house control module

Claims

1. a recognition inference processing unit that outputs a recognition accuracy of an image of an item; an image selection unit that selects low-accuracy items from a plurality of images, the recognition accuracy of which is low enough to satisfy a predetermined standard, and selects low-accuracy images from among the images including the low-accuracy items, the recognition accuracy of which is low enough to satisfy the predetermined standard; To have An image selection device comprising:

2. a dataset update unit that adds the selected low-precision images to a training dataset for training a model; 2. The image selection device according to claim 1, wherein:

3. providing a model learning unit that learns the model using the learning data set; 3. The image selection device according to claim 2, wherein:

4. The image selection unit includes: repeating the process of selecting the low-accuracy images until the result of the model recognizing the item satisfies a predetermined recognition accuracy; 3. The image selection device according to claim 2, wherein:

5. The image selection unit includes: When the trained model recognizes the item and the result satisfies a predetermined recognition accuracy, the selected low-accuracy image is displayed on an arbitrary device.

5. The image selection device according to claim 4, wherein:

6. a detection target information processing unit for acquiring detection target information indicative of a quality of the image; an importance index output unit that outputs an importance of additional learning based on the detection target information; To have 2. The image selection device according to claim 1, wherein:

7. The detection target information processing unit includes: Detecting excess detection frames in the image as overlap ratio information, which is one of the detection target information; 7. The image selection device according to claim 6,

8. The detection target information processing unit includes: Detecting the merged detection frame or the detection frame in the dark area as color information, which is one of the detection target information; 7. The image selection device according to claim 6,

9. The detection target information processing unit includes: Detecting a detection frame in which only a part of an item is captured as area information, which is one of the detection target information; 7. The image selection device according to claim 6,

10. The detection target information processing unit includes: Detecting an item detection frame, which is a feature that has not been learned in advance, as feature distance information, which is one of the detection target information; 7. The image selection device according to claim 6,

11. The detection target information processing unit includes: Detecting a detection frame of a blurred image as definition information, which is one of the detection target information; 7. The image selection device according to claim 6,

12. The detection target information processing unit includes: Detecting a detection frame in which a storage unit body is shown as body information, which is one of the detection target information; 7. The image selection device according to claim 6,

13. The image selection unit includes: Obtain low-resolution item information contained in the selected low-resolution image; The image selection device includes: An importance index output unit that outputs an additional learning importance based on the low-accuracy item information; 2. The image selection device according to claim 1, wherein:

14. The recognition and inference processing unit of the image selection device includes: The recognition accuracy of the image of the item is output. The image selection unit of the image selection device includes: Selecting low-accuracy items from a plurality of images, the recognition accuracy of which is low enough to satisfy a predetermined standard, and selecting low-accuracy images from among images including the low-accuracy items, the recognition accuracy of which is low enough to satisfy a predetermined standard; The image selection method according to the present invention is characterized in that

15. a recognition inference processing unit that outputs a recognition accuracy of an image of an item; an image selection unit that selects low-accuracy items from a plurality of images, the recognition accuracy of which is low enough to satisfy a predetermined standard, and selects low-quality images from among the images including the low-accuracy items, the recognition accuracy of which is low enough to satisfy the predetermined standard; An image selection device comprising: A storage facility for storing items related to the item; To have A storage system characterized by:

Citation Information

Patent Citations

  • Recognition dictionary evaluation device and recognition dictionary evaluation program

    JP2015022411A

  • Multi-label data learning assisting apparatus, multi-label data learning assisting method and multi-label data learning assisting program

    JP2020101968A

  • Storage house, storage system and image recognition method

    JP2022094416A

  • Information processing method, information processing device, and information processing program

    WO2023286795A1

  • Food management system

    JP2021196636A