Learning data sorting device, storage facility, and learning data sorting method

Abstract

To select an image serving as effective learning data, based on an index, with low load.SOLUTION: A learning data selection apparatus includes: an information addition unit which adds, as meta information of an image, to the image, environment information of a storage region that affects a difficulty level of recognizing an article included in the image obtained by imaging the storage region storing the article, definition information relating to the resolution of the image, and / or inference information based on an output of a model which recognizes the article; and an image selection unit which selects, according to the meta information, an image in a storage unit which stores a plurality of images with the mata information added thereto.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a learning data selection device, a storage, and a learning data selection method.

Background Art

[0002] Recently, refrigerators that automatically infer what kinds of food ingredients are stored inside have become widespread. In Patent Document 1, each of a plurality of users who use the same refrigerator operates a terminal device that stores feature information for each food ingredient. When a certain terminal device cannot recognize the type of a food ingredient, it requests another terminal device to recognize the type of the food ingredient. By doing so, even when a certain terminal device cannot recognize the type of a certain food ingredient, another terminal device can recognize the type of the food ingredient.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Generally, in order to infer the type of a food ingredient, a large number of images obtained by imaging the food ingredient must be used as learning data, and a model that takes an image as an input and outputs the type of the food ingredient must be machine-learned. The accuracy of the inference result depends greatly on the quantity and quality of the learning data. It is important to select learning data that truly contributes to improving the inference result with low load.

[0005] However, Patent Document 1 does not mention selecting learning data from a huge number of candidates in order to improve the accuracy of the inference result.

Means for Solving the Problems

[0006] The learning data sorting device of the present invention includes environmental information of the storage area in which an article is stored, which affects the difficulty of recognizing the article, and detail information relating to the resolution of the image. And, The system includes an information assignment unit that assigns inference information based on the output of a model that recognizes the aforementioned item to the image as metadata for the image, and an image selection unit that selects images from a storage unit containing multiple images to which the metadata has been assigned, according to the metadata. The inference information includes a precision rate, which indicates the proportion of data that the model predicted to be a certain item that are actually that item; a recall rate, which indicates the proportion of data that are actually that item that the model correctly predicted as that item; and an F-score, which is the harmonic mean or weighted mean of the precision rate and the recall rate. The image selection unit repeats the process of selecting images until the F-score, which is such that the precision rate equals the recall rate, exceeds a predetermined threshold. It is characterized by the following. Other means will be described in the section on embodiments for carrying out the invention. [Brief explanation of the drawing]

[0007] [Figure 1] This diagram illustrates the configuration of the refrigerator and the learning data sorting device. [Figure 2] This is a flowchart of the training data selection process. [Figure 3] This diagram illustrates the details of S205 in Figure 2. [Figure 4] This diagram explains the details of S303 in Figure 3. [Figure 5] This is a diagram illustrating the configuration of a mobile device. [Modes for carrying out the invention]

[0008] The following describes embodiments of the learning data sorting device, storage facility, and learning data sorting method (learning data sorting program) of the present invention.

[0009] (Model accuracy and training data quality) One model takes an image of food as input and outputs the type of food (egg, milk, onion, ...). This model is, for example, a neural network with an input layer, multiple hidden layers, and an output layer. Each node in each layer is set with propagation parameters that determine how much information is passed to which node in the next layer. If these propagation parameters are set to random initial values, the model is unlikely to output the true type of food.

[0010] Therefore, it is common practice to use training data to machine-learn (optimize) the propagation parameters of the model. The training data here is a collection of a vast number of images containing one or more food items, and each of these images is assigned a "correct label" verified by human eyes, making it "supervised" training data.

[0011] The quality of actual training data varies. Some images, like the sample images taken in the studio, are well-lit and show ingredients from the front without overlapping with other ingredients. However, there are also many "bad images" that overlap with other ingredients, are poorly lit, include people, are distorted, or are incorrectly combined images from multiple different points in time.

[0012] Removing defective images from training data is one way to improve the accuracy of a model. However, if defective images that are likely to occur are correctly labeled as "correct" and then added to the training data, the model's accuracy can sometimes be improved. This invention aims to select "defective images" that produce such an effect.

[0013] Hereafter, these embodiments (Examples 1 and 2) will be described based on the drawings. These embodiments are examples of recognizing (inferring) the type of food stored in a refrigerator. However, the present invention is applicable not only to refrigerators but also to warming cabinets, refrigerators, and other storage facilities, and more generally, to storage areas for storing articles. In these embodiments, "food" is a representative example of articles in a broader sense. Food may be raw materials, cooked food, or seasonings.

[0014] This embodiment describes an example in which a camera captures an image of food ingredients. However, a weight sensor, a mycotoxin (mold) detection sensor, an IC tag reader, a character reader, etc., may be used in place of or in addition to the camera.

[0015] In this embodiment, the recognition inference processing unit recognizes the presence of food ingredients in the storage area, the usage status of the food ingredients consumed by the user, and the like. In this embodiment, the recognition inference processing unit performs recognition using the camera image acquired via the image data processing unit. However, the recognition inference processing unit may be configured to perform recognition in response to an input from the user to the mobile terminal such as a smartphone or the refrigerator itself.

[0016] <Example 1> (Refrigerator and Learning Data Selection Device) FIG. 1 is a diagram for explaining the configuration and the like of a refrigerator and a learning data selection device. The refrigerator 100 is connected to the mobile terminal 113 and the computer 115 via the network 116.

[0017] The refrigerator 100 includes a learning data selection device 101 and a refrigerator main body 118. The mobile terminal 113 as an external device is an information processing device such as a tablet, a smartphone, or a personal computer used by the user of the refrigerator 100. Further, the main processing of this embodiment may be executed by the learning data selection device 101, may be executed by the mobile terminal 113, or may be executed by the learning data selection device 101 and the mobile terminal 113 in cooperation. The computer 115 distributes various information, machine learning models, and the like to the learning data selection device

[0018] A camera (not shown) for imaging the interior of the refrigerator 100 is attached to the refrigerator 100. The position of the camera may be inside the refrigerator or outside the refrigerator. The number of cameras is an arbitrary number of 1 or more. More generally, the camera does not necessarily have to be attached to the refrigerator 100. For example, the camera may be incorporated in the mobile terminal 113.

[0019] The learning data selection device 101 that controls the refrigerator 100 includes, for example, a processor

[0020] The learning data selection device 101 can also transmit part or all of the computer programs and data stored in the storage device 102 to the outside via the network 116. Conversely, the learning data selection device 101 can receive computer programs and data from an external computer 115 or the like via the network 116 and store them in the storage device 102.

[0021] The learning data selection device 101 can also transmit and receive part or all of the computer programs and data to and from a storage medium 109 such as a flash memory or a hard disk drive connected to itself.

[0022] The storage device 102 stores a refrigerator information processing unit 103, an image data processing unit 104, a recognition inference processing unit 105, an image selection unit 106, a dataset update unit 107, a model learning unit 108, and an in-store control unit 119. These are computer programs that realize predetermined functions. The storage device 102 (storage unit) stores a plurality of images to which meta-information (details will be described later) is attached.

[0023] When the processor 111 executes these computer programs, each function is realized. Therefore, these units can be read as the corresponding computer programs, and the processor 111 realizes the processing and functions of each unit described later in cooperation with each computer program. However, these units may be realized by dedicated hardware, FPGA (Field Programmable Gate Array), or the like. Furthermore, these computer programs may be configured with a number less than the number shown in the figure. In this case, each unit can be configured as a computer module.

[0024] As described above, the processor 111 operates as a functional unit that provides predetermined functions by executing processing according to a computer program. For example, the processor 111 functions as a refrigerator information processing unit 103 by executing processing according to a refrigerator information processing program (for example, by executing a learning data selection method). The same applies to other computer programs. Furthermore, the processor 111 also operates as a functional unit that provides the functions of each of the multiple processes executed by each computer program. In this embodiment, one processor 111 executes the computer program, but multiple processors may execute it.

[0025] The refrigerator information processing unit 103 acquires refrigerator information. This refrigerator information mainly consists of environmental information and resolution information. Environmental information refers to information about the environment of the storage area that affects the difficulty of recognizing food items. More specifically, it includes information such as the door opening angle obtained from the refrigerator's sensors, the color of the interior walls obtained from the refrigerator's model, the brightness inside and outside the refrigerator, the time of imaging, and the density inside the refrigerator. Detail information refers to information about the image quality, resolution, format (jpg, png, etc.), etc.

[0026] The image data processing unit 104 acquires images from the refrigerator body 118 via the input / output interface 110 and stores the acquired images in the storage device 102. Furthermore, if the image data processing unit 104 acquires images captured with a fisheye or wide-angle lens, it may convert them into planar images (images with a standard field of view and no distortion). Techniques for converting images captured with a fisheye or wide-angle lens into planar images are well known.

[0027] The recognition and inference processing unit 105 recognizes food items within a storage area from an image of the storage area. At this time, the recognition and inference processing unit 105 inputs the image into a machine learning model, such as one that has been pre-programmed with deep learning, and has the model output the type of food item. The recognition and inference processing unit 105 also creates inference information (described immediately below). Furthermore, the recognition and inference processing unit 105 associates environmental information, detail information, and / or inference information with the image. Incidentally, environmental information, detail information, and inference information are all metadata. Metadata is not the data or information itself, but data or information that describes the attributes and related information that represent that data or information. In other words, the recognition and inference processing unit 105 is an "information assignment unit" that assigns environmental information, detail information, and / or inference information to the image as metadata for the image.

[0028] The inference information includes the model output itself and the results calculated by the recognition inference processing unit 105 based on the model output, and includes, for example, the following: • The model inferred the types of ingredients contained in the image. • The accuracy of the type of food inferred by the model. • Precision: The percentage of data that was predicted to be food ingredient A but actually is food ingredient A. • Recall rate: The percentage of items that were actually food ingredient A that were correctly predicted to be food ingredient A. • F-score: Harmonic mean or weighted mean of precision and recall • Uncertainty: Variance in predictions due to stochastic factors occurring both inside and outside the model. • Detection frame: Image size, shape, aspect ratio, etc.

[0029] More specifically, the recognition inference processing unit 105 associates the precision, recall, F-score, and uncertainty from this inference information with a group of images that were the subject of inference for a model trained on the same group of training data. The recognition inference processing unit 105 associates the other inference information with each individual image that was the subject of inference.

[0030] When the goal is to reduce false positives, precision is emphasized. When the goal is to reduce undetected results, recall is emphasized. Although this embodiment assumes that one inference is performed for each image, it is also possible to perform a single inference on multiple images at once.

[0031] The recognition inference processing unit 105 of this embodiment recognizes the food items in the refrigerator 100 using images, but is not limited to this, as described above. When information other than images is used, the image data processing unit 104 may be omitted. Instead, a separate configuration is provided for use in recognition by the recognition inference processing unit 105. For example, when a weight sensor is used, a functional unit is provided that performs processing to associate weight with food items.

[0032] Generally, "recognition" refers to outputting the type of item. "Learning" refers to optimizing the model when it performs recognition. "Inference" refers to the trained model automatically outputting the type of item. In this embodiment, since it is an example of using a model, there is no essential difference between "recognition" and "identification." The recognition inference processing unit 105 is named in this way because it is a unit where a trained model performs recognition.

[0033] The image selection unit 106 selects images according to environmental information, resolution information, and / or inference information. In other words, the image selection unit 106 selects images from the storage unit, which stores multiple images with metadata attached, according to the metadata. The selected images become training data for machine learning the model. The dataset update unit 107 adds and updates the training dataset with the images selected by the image selection unit 106. The training dataset is the largest possible set of images that can be used as training data for machine learning a model that recognizes food ingredients. The dataset update unit 107 may also increase the amount of image data by performing data augmentation such as rotation, shifting, or horizontal flipping on the images selected by the image selection unit 106 and add and update them to the training dataset.

[0034] The model learning unit 108 uses a training dataset, for example, by deep learning, to train a model that recognizes food items inside a refrigerator. The model learning unit 108 may increase the number of training iterations by cross-validation to improve the generalization performance of the trained model and suppress overfitting. The model learning unit 108 may further fine-tune (adjust) the trained model and train the model again using unused portions of the training dataset to improve generalization performance.

[0035] The internal control unit 119 controls a motor and a compressor (not shown) to control the temperature and humidity inside the refrigerator 100.

[0036] (Training data selection process procedure) Figure 2 is a flowchart of the training data selection process. In step S201, the refrigerator information processing unit 103 acquires environmental information and / or resolution information from the refrigerator's sensors, etc.

[0037] In step S202, the image data processing unit 104 acquires the image captured by the camera and converts the fisheye image into a planar image if necessary.

[0038] In step S203, the recognition inference processing unit 105 calculates an F-score such that "weighted precision = weighted recall". In other words, the recognition inference processing unit 105 does not necessarily need to calculate an F-score such that "precision = recall", but may calculate an F-score such that "precision ≥ recall" or "precision ≤ recall". Precision, recall, and F-score may be replaced with accuracy, specificity, uncertainty, AUC (Area Under the Curve), etc. Note that "predetermined precision" is a concept that includes precision, recall, F-score, accuracy, specificity, uncertainty, and AUC.

[0039] Furthermore, the recognition inference processing unit 105 associates the environmental information, resolution information, and / or the inference information it has calculated in step S201 with the image acquired in step S202.

[0040] In step S204, the recognition inference processing unit 105 determines whether the F-value is less than or equal to the threshold c. If the F-value is less than or equal to the threshold c (bad image) (step S204 "yes"), the process proceeds to S205. If the F-value is greater than the threshold c (step S204 "no"), the training data selection process is terminated. The threshold c is used to determine whether an image is a bad image and contributes to effective image selection with low overhead. In recognizing the type (item) of food ingredients, if the recognition accuracy for a particular type is low, the recognition inference processing unit 105 may prioritize selecting images containing that type.

[0041] Steps S203 and S204 (indicated by dashed lines) correspond to the recognition inference process S209, which will be described later in Figure 3.

[0042] In step S205, the image selection unit 106 selects images according to environmental information, resolution information, and / or inference information. For example, the image selection unit 106 selects images from many candidate images that match the combination of "potato" as inference information and "the wall is brown" as environmental information. The image selection unit 106 makes it easier to select defective images by using at least one of the environmental information, resolution information, and inference information as an indicator (selection criterion) for selection. Details of step S205 will be described later.

[0043] In step S206, the dataset update unit 107 adds the images selected in step S205 to the training dataset.

[0044] In step S207, the model learning unit 108 uses the training dataset added and updated in step S206 to train a model that recognizes food ingredients. The training dataset used in this stage is labeled with "correct labels" by the user and / or manufacturer. The model learning unit 108 may perform the fine tuning described above at this stage. After processing in step S207, the process returns to step S203.

[0045] After step S204 "yes" and going through step S205, but before the learning data selection process is completed, the image selection unit 106 may display some or all of the images (defective images) selected in step S205 on any device, including the output device (not shown) of the learning data selection device 101.

[0046] In the iterative processing of steps S203 to S207, the F value corresponds to a "predetermined accuracy." Therefore, the image sorting unit 106 repeats the process of sorting training data from the images until the result of the machine learning model recognizing an item satisfies the predetermined accuracy. Accordingly, the model learning unit 108 can build a model that can recognize even defective images with high accuracy by repeatedly machine learning the model until the image recognition accuracy reaches a certain level or higher.

[0047] The recognition inference processing unit 105 can update the model to the latest version in response to changes or additions to food packaging, etc. The recognition inference processing unit 105 may, for example, update the model it uses with a model received from a server (not shown).

[0048] As described above, this embodiment can improve recognition accuracy even for images with patterns that could not be recognized before, by selecting defective images according to environmental information, resolution information, and / or inference information, and adding the selected images to the training dataset. In other words, this embodiment provides a new metric that can be used to select data that is effective for training, and enables effective data selection with low overhead.

[0049] (Details of S205) Figure 3 is a diagram illustrating the details of S205 in Figure 2. The image selection unit 106 performs image selection according to environmental information, resolution information, and / or inference information, for example, by at least one of the state change detection process S301, combination generation process S302, unknown class detection process S303, and human detection process S304. The recognition inference process S209 corresponds to steps S203 and S204 in Figure 2, and processes S301 to S305 correspond to step S205 in Figure 2.

[0050] In the recognition inference processing S209, the recognition inference processing unit 105 associates environmental information, resolution information, and / or inference information with the image, as described above.

[0051] In the state change detection process S301, the image sorting unit 106 calculates state change information P based on the amount of change in position information obtained from environmental information, resolution information and / or inference information. Defective images that can be sorted in the state change detection process S301 include time-series shift images from composite correction, dark area images, images with reduced image quality originating from the in-cabinet fisheye camera, miscellaneous and unspecified images, and images that have become whitish.

[0052] The time-series shift in composite correction images occurs when only one door of a double-door refrigerator is opened or closed, resulting in a time-series shift between the left and right sides of the refrigerator compartment. This is caused by only the food positions in one side of the refrigerator compartment being updated to the latest state. When time-series shift images occur in composite correction images, information about food items may be missing, or unclear objects such as dividers may appear, potentially leading to false detections or failure to detect items. Time-series shift images in composite correction images are filtered based on door opening / closing information, differences in image quality within the image, etc.

[0053] Dark area images are images that tend to occur when the room is dark and are prone to reducing recognition accuracy. Dark area images are filtered based on factors such as the brightness inside and outside the storage area, the time of imaging (rooms tend to be darker at night), and the brightness information of the image.

[0054] Image degradation originating from the in-refrigerator fisheye camera is more likely to occur in images showing the lower section of the refrigerator compartment. When the fisheye camera is mounted at the top of the refrigerator compartment, the lower section is far from the camera, resulting in insufficient distortion correction when the fisheye camera image is converted to a flat image, easily leading to image degradation. Images with image degradation originating from the in-refrigerator fisheye camera are selected based on information obtained from the refrigerator model, differences in image quality within the image, and other factors.

[0055] Miscellaneous and unspecified images are those where the refrigerator is overloaded with food items, making it difficult to identify the types of food. Miscellaneous and unspecified images are filtered based on factors such as food density. Images that appear whitish may be caused by factors such as the camera being located inside the storage area, where high humidity can cause the camera lens to fog up, resulting in a whitish appearance. These whitish images are filtered based on information such as door opening / closing status and humidity sensor data.

[0056] The image sorting unit 106 can sort out defective images with large state change information P. Note that state change information P corresponds to an "indicator showing the degree to which an image is defective." Specific examples of state change information P are given below. P = Area where images from different time points overlap / Total area of ​​the image P = Area of ​​the portion of the image where the brightness is below a predetermined threshold / Total area of ​​the image

[0057] In the combination generation process S302, the image selection unit 106 calculates a rarity R, which is the degree to which individual events from the environmental information, detail information, and / or inference information occur simultaneously. The image selection unit 106 increases the rarity of a combination of events that has not been used in training before. The combination generation process S302 calculates the rarity R based on the misrecognized combination information, selects images with a high rarity R, and applies them to additional training, thereby improving recognition accuracy for similar patterns of misrecognition. The defective images that can be selected in the combination generation process S302 include time-series shifted images from composite correction, images with reduced image quality originating from the in-warehouse fisheye camera, miscellaneous and unspecified images, and images that have become washed out.

[0058] The image selection unit 106 can select defective images with a large R value. The rarity R value corresponds to an "indicator showing the rarity of the learning frequency." Specific examples of rarity R are given below. R(x,y) = Total number of frequencies / Frequency of the event that the wall color is x and the recognition result (type of food) is y occurring simultaneously.

[0059] For example, if the brown color of the refrigerator interior is rare, there is a high probability that "R(brown, potato) > R(white, potato)".

[0060] In the unknown class detection process S303, the image sorting unit 106 determines the unknown class based on uncertainty calculated according to environmental information, resolution information, and / or inference information, and calculates the unknownness level U. Details of the unknown class detection process S303 will be described later. Furthermore, specific examples of the unknownness level U will also be described later.

[0061] In the person detection process S304, the image selection unit 106 detects people according to environmental information, resolution information, and / or inference information, and calculates person detection information H. Since people are not food ingredients, the image selection unit 106 excludes the entire image containing people from the training image selection process, or performs processing such as deleting the part of the image that contains people.

[0062] The image sorting unit 106 can sort out defective images with a large H value. A specific example of human detection information H is given below. H = Area of ​​people in the image / Total area of ​​the image H=0 or 1 (0 if no person is visible, 1 if even a part of a person is visible)

[0063] In the image sorting process S305, the image sorting unit 106 uses at least one combination of state change information P, rarity R, unknownness U, and human detection information H as an indicator (sorting criterion) and sorts images that meet that indicator. This makes it easier to sort out defective images that have the quality desired by the user. It is not mandatory to combine all of state change information P, rarity R, unknownness U, and human detection information H. The image sorting unit 106 may sort images using at least one of these.

[0064] (Details of S303) Figure 4 is a diagram illustrating the details of the process in S303 of Figure 3. In uncertainty calculation process S401, the image selection unit 106 calculates the uncertainty of an image according to the inference information. Uncertainty is the variance of the inference results output by the model for images of the same type of food ingredient. For example, suppose that when an image of a "potato" is input, the model has previously inferred many different types, such as "potato," "taro," "sweet potato," "ginger," "egg," etc., and the number of types is n. In this case, n itself becomes the uncertainty. The image selection unit 106 may also calculate uncertainty using other statistical methods. The model outputs different inference results each time for an image of an unknown food ingredient. In other words, the greater the uncertainty, the higher the possibility that this type of food ingredient has never been stored in the refrigerator before, or that it is a new type of food ingredient.

[0065] In the uncertainty determination process S402, the image processing unit 106 converts the uncertainty calculated in S401 into an uncertainty U, according to environmental information and / or detail information (such as the refrigerator model and the position in the image based on the door angle). It has been empirically observed that, depending on the model or door angle, the uncertainty of the inference result output by the model will be calculated to be larger (or smaller) than expected, regardless of the type of food image that is input. Therefore, the image processing unit 106 corrects the uncertainty according to a predetermined rule and sets the result as the uncertainty U.

[0066] Another example of the degree of unknown U is the degree of "fluctuation" in the inference result when the propagation parameters of the model are slightly changed (Monte Carlo dropout). Suppose the image selection unit 106 slightly changes the propagation parameters of a model that previously inferred an image of a "potato" with 100% probability. As a result, suppose the probability of the model inferring a "potato" becomes 20%. In this case, the image selection unit 106 may set "U = 100 - 20 = 80".

[0067] Furthermore, the image processing unit 106 may assign (determine) the value of the unknownness level U to one of the predefined discrete unknown classes (UC = 1, 2, 3, ...). By determining the unknown class, the image processing unit 106 can detect new products, etc., whose types have not been registered in advance, and this can lead to the addition of new types, etc. Note that the unknownness level U corresponds to "an index indicating the degree to which an item is not recognized as a known item."

[0068] <Example 2> In Example 1, the learning data selection device 101, which is part of the refrigerator 100, performs the learning data selection process. However, the learning data selection device 101, which is in a separate enclosure independent of the refrigerator 100, may also perform the learning data selection process. Below, a modified example in which the mobile terminal 113 performs the learning data selection process will be described.

[0069] Figure 5 is a diagram illustrating the configuration of the mobile terminal 113. The mobile terminal 113 comprises a processor 502, a storage device 504, a touch panel 501, and a communication unit 503. The mobile terminal 113 is a computer such as a smartphone.

[0070] The processor 502 and storage device 504 have the same functions as the processor 111 and storage device 102 shown in Figure 1. The touch panel 501 functions as an input / output unit. The communication unit 503 is connected to the network 116. This connection can be wireless or wired.

[0071] The storage device 504 stores the learning image selection program 511 that performs the processing of Embodiment 2. The learning image selection program 511 consists of a refrigerator information processing module 505, an image data processing module 506, a recognition inference processing module 507, an image selection module 508, a dataset update module 509, a model learning module 510, and a refrigerator internal control module 512. Parts of several of these modules may be combined to form a larger module.

[0072] For example, the refrigerator information processing module 505 in Figure 5 performs the same functions as the refrigerator information processing unit 103 in Figure 1. The same applies to the other modules. However, it is desirable that the internal control module 512 also manages the usage status of food items inside the refrigerator. For example, the internal control module 512 acquires information read from user input information and food item codes, and the recognition inference processing module 507 recognizes the relevant food item.

[0073] The learning image selection program 511 is preferably distributed to the mobile terminal 113 via the network 116. Therefore, the network 116 will be implemented via the internet.

[0074] As described above, according to Examples 1 and 2, by using inference information and refrigerator information to select defective images and adding the selected defective images to the training dataset, it is possible to improve recognition accuracy even for patterns that could not be recognized before. In other words, it provides a new metric that can be used to select data that is effective for training, and it becomes possible to perform effective data selection with low overhead using this metric.

[0075] (Effects of this embodiment) The effects of the learning data selection device of this embodiment are as follows: (1) The training data selection device can select images to be used as training data based on environmental information and other information associated with the images. (2) The training data selection device can keep the training dataset for machine learning the model up to date. (3) The learning data selection device can select defective images.

[0076] (4) The training data selection device can select images that are used in training less frequently. (5) The learning data selection device can select images of unknown food ingredients. (6) The training data selection device can train the model using machine learning. (7) The training data selection device can repeat the image selection process until a predetermined level of accuracy is achieved. (8) The learning data selection device can display the selected images.

[0077] It should be noted that the present invention (learning data sorting device, storage device, and learning data sorting method) is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0078] Furthermore, each of the aforementioned configurations, functions, processing units, processing means, etc., may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. Alternatively, each of the aforementioned configurations, functions, etc., may be implemented in software by having the processor interpret and execute programs that realize each function. Information such as programs, tables, and files that realize each function can be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD. Furthermore, the control lines and information lines shown are those deemed necessary for explanatory purposes and do not necessarily represent all control lines and information lines in the actual product. In practice, it can be assumed that almost all components are interconnected. The various information of inventory management device 1 may reside in the cloud. [Explanation of Symbols]

[0079] 100 Refrigerator 101 Training Data Selection Device 102 Storage device (storage unit) 103 Refrigerator Information Processing Unit 104 Image Data Processing Unit 105 Recognition Inference Processing Unit (Information Assignment Unit) 106 Image sorting section 107 Dataset Update Section 108 Model Learning Department 109 Storage medium 110 Input / Output Interfaces 111 processors 112 Communications Department 113 Mobile devices 115 Calculator 116 Network 119 Internal Control Unit 501 Touch Panel 502 Processors 503 Communications Department 504 Storage device 505 Refrigerator Information Processing Module 506 Image Data Processing Module 507 Recognition Inference Processing Module 508 Image sorting module 509 Dataset Update Module 510 Model Learning Module 511 Training Image Selection Program 512 Internal Control Module

Claims

1. An information assignment unit assigns to an image, as metadata to the image, environmental information of the storage area in which an item is stored, which affects the difficulty of recognizing the item, detail information relating to the resolution of the image, and inference information based on the output of a model that recognizes the item. The storage unit, which stores multiple images to which the metadata has been attached, is equipped with an image selection unit that selects images according to the metadata. The aforementioned inference information is, The precision rate, which indicates the proportion of data that the model predicted to be a certain item that actually is that item, The recall rate, which indicates the proportion of actual items that the model correctly predicted, and The F-value includes the harmonic mean or weighted mean of the precision and recall, The aforementioned image sorting unit, The process of selecting images is repeated until the F-value, which is such that the precision is equal to the recall, becomes larger than a predetermined threshold. A learning data selection device characterized by the following.

2. The system includes a dataset update unit that adds the selected images to a training dataset for machine learning the model. A learning data sorting device according to claim 1, characterized by the following:

3. The aforementioned image sorting unit, Select the images according to an indicator that shows the degree to which the images are defective. A learning data sorting device according to claim 1, characterized by the following:

4. The aforementioned image sorting unit, Select the images according to an indicator showing the rarity of learning frequency. A learning data sorting device according to claim 1, characterized by the following:

5. The aforementioned image sorting unit, Selecting images according to an index indicating the degree to which the aforementioned item is not recognized as a known item, A learning data sorting device according to claim 1, characterized by the following:

6. The system includes a model learning unit that performs machine learning on the model using the aforementioned training dataset. The learning data sorting device according to claim 2, characterized by the following:

7. The aforementioned image sorting unit, Display the selected images on any device until the F-value, which is such that the precision is equal to the recall rate, becomes greater than a predetermined threshold. A learning data sorting device according to claim 1, characterized by the following:

8. An information assignment unit assigns to an image, as metadata to the image, environmental information of the storage area in which an item is stored, which affects the difficulty of recognizing the item, detail information relating to the resolution of the image, and inference information based on the output of a model that recognizes the item. The storage unit, which stores multiple images to which the metadata has been attached, is equipped with an image selection unit that selects images according to the metadata. The aforementioned inference information is, The precision rate, which indicates the proportion of data that the model predicted to be a certain item that actually is that item, The recall rate, which indicates the proportion of actual items that the model correctly predicted, and The F-value includes the harmonic mean or weighted mean of the precision and recall, The aforementioned image sorting unit, The process of selecting images is repeated until the F-value, which is such that the precision is equal to the recall, becomes larger than a predetermined threshold. A storage facility characterized by [this feature].

9. The information assignment unit of the learning data selection device is: Environmental information of the storage area in which an item is stored, which affects the difficulty of recognizing the item in the image, detail information regarding the resolution of the image, and inference information based on the output of a model that recognizes the item are added to the image as metadata. The image sorting unit of the aforementioned learning data sorting device is: The images stored in the storage unit, to which the aforementioned metadata has been attached, are selected according to the metadata. The aforementioned inference information is, The precision rate, which indicates the proportion of data that the model predicted to be a certain item that actually is that item, The recall rate, which indicates the proportion of actual items that the model correctly predicted, and The F-value includes the harmonic mean or weighted mean of the precision and recall, The aforementioned image sorting unit, The process of selecting images is repeated until the F-value, which is such that the precision is equal to the recall, becomes larger than a predetermined threshold. A training data selection method characterized by the following.

Images