Identification system for packaged food products and method of applying food product labels to packages

Abstract

To identify a new commodity from other registered commodities by detecting that it is difficult to identify the new commodity by an identification system and proposing the change of a commodity package.CONSTITUTION: The identification system is equipped with a camera 8, a computer 4 which identifies the kind of a commodity according to an image from the camera, and a display device 10 which displays the identification result of the commodity. The computer 4 includes a feature amount extraction unit 22 that extracts a feature amount from an image, a product database 26 that stores a feature amount of a product image, and an identification unit 24 that identifies a type of a product by comparing the extracted feature amount with a feature amount of a product image registered in the product database 26. When a new commodity is registered in a commodity database 26, if an extracted feature quantity is similar to a feature quantity of another commodity registered in the commodity database 26 by a threshold or more, it is displayed on a display device 10 that the new commodity cannot be registered because a similar commodity is already registered, and change of a package is requested.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] This invention relates to determining the type of packaged food by image recognition so that labels in accordance with the Food Labeling Act can be affixed. [Background technology]

[0002] Convenience stores, supermarkets, and other stores process food, package it, affix a food label, and sell it. Food labels are based on the Food Labeling Act and display the product name (black pork cutlet, vegetable dumplings, chirashi sushi, etc.), ingredients, expiration date or best-before date, information about allergens, storage instructions, nutritional information, etc. Note that food packages may have labels unrelated to the Food Labeling Act, such as "XX specialty," "XX farm," or "XX tomato," but this specification refers to these as stickers and distinguishes them from food labels. Furthermore, when this specification simply refers to "label," it generally refers to a food label.

[0003] Food labels are linked to 3-5 digit numbers or barcodes, printed by a label printer, and then manually affixed to food packages. Food labels are typically applied by visually identifying the food type and entering the number assigned to each food item into a label printer or similar device. Alternatively, barcodes printed in a barcode book along with the food's name are manually scanned with a barcode scanner and entered into a label printer, resulting in the food label. However, when entering numbers, it is difficult to eliminate input errors. When scanning barcodes, there is also the risk of accidentally scanning adjacent barcodes. Entering the wrong label type not only poses health risks related to allergens, but also damages a store's credibility due to incorrect product names.

[0004] Related prior art is shown below. Patent Document 1 (JP 2023-64548A) proposes printing food labels by image-recognizing the type of food element. In the inventor's opinion, it is difficult to distinguish between "meat dumplings" and "vegetable dumplings," or between simple "tonkatsu" and "tonkatsu" made with branded meat, based on the image characteristics of the food itself. Hereinafter, in this specification, "product" refers to a packaged food product, and "food" refers to the packaged food itself. Patent Document 2 (JP 7250275B) discloses a method for identifying food on a tray in a cafeteria, etc., by identifying the tableware, thereby limiting food candidates and facilitating food identification. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2023-64548A [Patent Document 2] Patent 7250275B Summary of the Invention [Problem to be solved by the invention]

[0006] The objective of this invention is to provide an identification system that detects when it is difficult to distinguish between new products and previously registered products and suggests changing the product packaging. This invention identifies the overall image features of the package and the food inside, so even if the food inside is indistinguishable, it is possible to identify the product by changing the packaging. This makes it possible to distinguish between new products and previously registered products, and to register new products in an identifiable manner. [Means for solving the problem]

[0007] The packaged food identification system of the present invention comprises: a camera for capturing an image of a product consisting of a package and the food inside; a computer that identifies the type of product based on the product image obtained from the camera; a display that displays the product identification result by the computer, The computer a feature extraction unit that extracts features from product images obtained by a camera; a product database for registering products by storing product names, data specifying food labels to be attached to packages, and feature amounts of product images; a classification unit that identifies the type of product by comparing the feature amount extracted by the feature amount extraction unit with feature amounts of product images registered in a product database; When registering a new product in the product database, if the feature values ​​extracted from the new product image are similar to the feature values ​​of other products registered in the product database by more than a threshold value, the new product registration unit displays on the display that the new product cannot be registered because a similar product is already registered.

[0008] The method of attaching a food label to a package of the present invention includes: a camera for capturing an image of a product consisting of a package and the food inside; a computer that identifies the type of product based on the product image obtained from the camera; a display that displays the product identification result by the computer, The computer a feature extraction unit that extracts features from product images obtained by a camera; a product database for registering products by storing product names, data specifying food labels to be attached to packages, and feature amounts of product images; a classification unit that identifies the type of product by comparing the feature amount extracted by the feature amount extraction unit with feature amounts of product images registered in a product database; When registering a new product in the product database, if the feature quantities extracted from the new product image are similar to the feature quantities of other products registered in the product database by more than a threshold value, the system uses a packaged food identification system that is equipped with a new registration unit that displays on the display that the new product cannot be registered because a similar product is already registered. In this invention, when the type of product is identified, the identification result is transferred to a label printer, whereby a food label corresponding to the identified product is printed and affixed to the product package. When a new product is to be registered in the product database, if the display indicates that the new product cannot be registered because a similar product has already been registered, the product package is changed.

[0009] Preferably, when the new product cannot be registered, the new registration unit suggests on the display that a sticker be affixed to the package to enable identification of the product. It is particularly preferred that the display suggest a design for the sticker. In this way, the new product can be easily distinguished from previously registered products without changing the package itself or the food inside.

[0010] Preferably, when the new registration unit is unable to register a new product, the new product image obtained by the camera and an image of a previously registered product with similar features are displayed on a display device. This display allows the worker to understand why identification is difficult and provides a hint as to how the package can be changed to enable identification.

[0011] Preferably, the product database is configured to store feature amounts of product images of multiple products for one type of product, and when the type of product is identified by the identification unit, to update the stored feature amounts of the product images by adding a newly identified feature amount, If the type of product can be identified but another type of product with a similar feature amount of the product image is registered in the product database, an inquiry as to whether it is OK to update the storage of the feature amount of the product image is displayed on the display.

[0012] Each time product A already registered in the product database is cooked and identified, it is preferable to additionally learn the features in the image and expand the range in which it can be identified as product A. However, additional learning of features close to the features of other product B increases the possibility of misidentifying product B as product A. Therefore, preferably, when the type of product can be identified but other types of products with close product image features are registered in the product database, a query is displayed on the display asking whether it is OK to update the memory of the product image features, and additional learning is performed after waiting for confirmation from the operator.

[0013] Preferably, a code such as a barcode corresponding to the identified product type is displayed on the display. When the code is read by a barcode scanner or the like, the food label to be printed can be instructed to the label printer in the same way as reading a barcode from a barcode book.

[0014] This invention uses an identification system to detect when a new product is difficult to identify. This invention identifies the entire package and the food inside, so even if the food inside is indistinguishable, the product can be identified by changing the packaging. This makes it possible to identify the product and affix the correct food label. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a block diagram of an identification system according to an embodiment of the present invention; [Figure 2] A diagram showing the display of the recognition result and the reading of the output barcode. [Figure 3] Block diagram of the main parts of the identification system [Figure 4] A schematic diagram of the product database in the identification system. [Figure 5] Flowchart showing product identification algorithm [Figure 6] A diagram showing the process when the reliability of the identification is low. [Figure 7] A diagram showing the data in the log memory [Figure 8] Flowchart showing the process for new registration DETAILED DESCRIPTION OF THE INVENTION

[0016] The scope of the present invention should be determined based on the claims, taking into account the description in the specification and well-known techniques in this field, and in accordance with the understanding of those skilled in the art. [Example]

[0017] An embodiment is shown in Figures 1 to 8. As shown in Figure 1, the identification system 2 comprises a stand 3 on which a packaged product 5 is placed, an identification computer 4, a light source 6, a camera 8, and a display 10 such as an LCD touch panel. The display 10 is not limited to a touch panel, and any device capable of displaying images and processing instructions on the screen as input is sufficient. To facilitate imaging, it is preferable to provide a polarizing plate that processes the light from the light source 6 and another polarizing plate that processes the light incident on the camera 8.

[0018] As shown by the dashed line in Figure 1, the processing in computer 4 may be transferred to server 12, and multiple identification systems 2 may be connected to server 12. Reference numeral 14 denotes a code scanner such as a barcode scanner, which reads the barcode, QR code (registered trademark), etc. displayed on display 10, transfers the data to label printer 15, prints a food label corresponding to product 5, and affixes it manually, for example, to the package of product 5. Barcode scanner 14 and label printer 15 are not part of the system of the present invention.

[0019] As shown in Figure 2, for example, products 5 prepared in the store are placed on the machine 3 in their respective packages, and an image is taken with a camera, and the type of product 5 is determined from the feature quantities of the product image. Products 5 are food items stored in a package, and even wrapped rice balls are included in the products of this invention, with the wrap being considered as the package. For example, all products 5 prepared in the store are image-identified, and food labels are attached.

[0020] The features of a product image are not just the features of the food inside, but also the features of the entire image including the packaging. Features include the shape and size of the package, the pattern printed on the package, the size and pattern of the sticker attached to the package, and the text on the package or sticker. Image recognition may or may not involve character recognition. The shape, size, color, and pattern of the food inside the package are also included in the features of a product image. Features that can be used for image recognition can be used as features regardless of whether it is the package or the food inside.

[0021] The captured product image, the product name, and a barcode corresponding to the food label are displayed on the display 10. The displayed barcode is read by the barcode scanner 14, transferred to a label printer, and a food label is printed and attached to the product 5, for example, by hand.

[0022] Figure 3 shows the configuration of the computer 4. An image memory 20 stores captured product images, and a pre-processing unit 21 performs pre-processing such as rotating the product images, adjusting their orientation, and removing noise as necessary. A feature extraction unit 22 extracts features from the product images, and a classification unit 24 compares the extracted features with features for each registered product in a product database 26 and extracts products whose extracted features are similar to the features of the registered products at a threshold or higher. Details of the processing by the classification unit 24 are shown in Figure 5.

[0023] The data in the product database 26 is shown in Figure 4. Data for one type of product consists of the product name, codes such as barcodes, a collection of feature values ​​for each individual product across multiple products, a representative image of the product, etc. Since one type of product is identified and learned multiple times, there are feature values ​​for multiple products, and it is possible to store the average value and variance of the feature values, or to store multiple feature values ​​as they are.

[0024] Returning to Fig. 3, display control 27 controls the display on display 10 and processes inputs made by the operator to display 10. In this embodiment, each time an item registered in product database 26 is identified, the feature amount of the item is added to product database 26 and stored as additional learning. At this time, if the feature amount is close to that of another item already registered, learning control unit 28 inquires of the operator via display 10 whether or not additional learning should be performed.

[0025] The new registration unit 30 performs processing when registering a new product in the product database 26, and details of this processing are shown in FIG. 8. Each time a product is identified, the image is stored together with the name of the worker, the date and time of the image capture, etc., and stored in the log memory 31. The data in the log memory 31 is shown schematically in FIG. 7.

[0026] Figure 5 shows the product identification algorithm. In step 1, an image of each packaged product is taken, and in step 2, feature amounts are extracted from the image of the package and the food inside. In step 3, a product database is searched to extract products whose feature amounts are similar enough to meet a threshold value. If there is only one type of product whose feature amounts are similar enough to meet the threshold value and the identification is highly reliable, in step 5 the captured product image, product name, and barcode are displayed on the display (see Figure 2). Even if the identification is highly reliable, it is preferable to display a button such as "Cancel" on display 10 so that the user can jump to step 9 in Figure 5.

[0027] In step 6, the barcode is read by a scanner and transferred to a label printer. In Figure 2, the horizontal mark in the center of the barcode indicates that the barcode has not been read, and the horizontal mark disappears when the barcode is read. The horizontal mark does not have to be displayed. Also, instead of reading the barcode with a scanner, the barcode data may be transferred from computer 4 to label printer 15. In step 6, the printed food label is affixed to product 5, for example, by hand.

[0028] If the reliability of the identification is low, for example, if there are multiple products with feature values ​​similar to the threshold or if there are no products with feature values ​​similar to the threshold, step 9.10 is executed. For example, as shown on the left side of Figure 6, the captured image is marked with a color or other mark to draw attention, and a candidate product name (in this case, "Yukari" rice balls) and an icon indicating that the identification is questionable (in this case, a "?" mark) are displayed. If the worker selects the product name "Yukari" here, the product name is set to "Yukari." If the worker touches the marked product or the "?" mark, the names of the top few product names with the highest feature similarity are displayed (center of Figure 6). If the worker selects "Wakame" from this, the process jumps to step 5, where the display in the upper right corner of Figure 6 is displayed, the barcode is read with a barcode scanner, and a food label is printed.

[0029] As a general rule, additional learning is performed each time a product is identified (step 8), and the features of the identified product are added to the product database 26. When an operator selects a product from multiple candidates, as in Figure 6, learning the identification results can be very meaningful, but it may also cause confusion in the identification. Therefore, it is preferable to ask the operator whether or not to perform additional learning, as shown in the lower right of Figure 6, and perform additional learning only if the operator selects additional learning. A similar problem occurs when one type of candidate is identified, but the difference between the features of other products is small, so that additional learning may cause the difference in features between products in the product database 26 to become too small. In such cases, the operator is also asked whether or not to perform additional learning. If the reliability of the identification is determined to be high in step 4 of Figure 5, there is no need to confirm whether or not additional learning is necessary.

[0030] 7, the log memory 33 stores the date and time, the name of the worker, the name of the identified product, the captured image, etc., each time a product is identified, as the identification result. The log memory 33 is useful for verifying the validity of the identification at a later date.

[0031] Figure 8 shows the process for new registration. The process in Figure 8 is performed when registering a new product in the product database, or when registering a product that is not completely new but is treated as a different product from already registered products. In step 11, an image of each package of the product to be newly registered is taken, and features are extracted in step 2. In step 13, the features are compared with those of other already registered products to check whether there are any other already registered products that are difficult to distinguish (step 14). "Difficult to distinguish" means, for example, that the features are similar to a predetermined value or more, and this predetermined value is different from the discrimination threshold in step 4 in Figure 5. If there are no already registered products that are difficult to distinguish, in step 15 the product name, the barcode corresponding to the food label, the product image, and features are registered in the product database, and the process ends.

[0032] If there are any registered products that are difficult to distinguish, an image of the new product obtained from the camera and an image of a registered product with similar features are displayed on the display along with their product names, allowing the worker to understand why they are difficult to distinguish. This display provides hints on how the packaging can be changed to enable identification. It is particularly preferable to suggest package changes on the display 10 (step 16). For example, it is suggested to attach a sticker to the package that clearly indicates the difference from similar products, or to change the shape or color of the package. An example of a sticker that clearly indicates the difference may be created using AI or the like and displayed on the display. Once it becomes easy to distinguish the new product from the registered product by attaching a sticker to the package, the process returns to step 11, and the registration of the new product is completed. [Explanation of symbols]

[0033] 2. Identification System 3 units 4. Computer 5 items 6 light source 8. Camera 10 Display 12 Servers 14. Barcode Scanner 15. Label printer 20 Image Memory 21 Pretreatment section 22 Feature extraction unit 24 Identification unit 26 Product Database 27 Display control unit 28 Learning control unit 30 New Registration Section 31 Log Memory

Claims

1. a camera for capturing an image of a product consisting of a package and the food inside; a computer that identifies the type of product based on the product image obtained from the camera; a display that displays the product identification result by the computer, The computer a feature extraction unit that extracts features from product images obtained by a camera; a product database for registering products by storing product names, data specifying food labels to be attached to packages, and feature amounts of product images; a classification unit that identifies the type of product by comparing the feature amount extracted by the feature amount extraction unit with feature amounts of product images registered in a product database; A packaged food identification system that includes a new registration unit that, when registering a new product in the product database, if the feature quantities extracted from the new product image are similar to the feature quantities of other products registered in the product database by more than a threshold value, displays on the display a message that the new product cannot be registered because a similar product is already registered.

2. 2. The packaged food identification system of claim 1, wherein the new registration unit, when unable to register a new product, causes the display to display a suggestion to affix a sticker to the package that enables product identification.

3. 3. The packaged food identification system of claim 1, wherein the new registration unit, when unable to register a new product, causes the display to display a new product image obtained from the camera and an image of a previously registered product having similar features.

4. the product database is configured to store feature amounts of product images of a plurality of products for one type of product, and to update the stored feature amounts of the product images when the type of product is identified by the identification unit; A packaged food identification system according to any one of claims 1 to 3, characterized in that if the type of product can be identified but another type of product with similar features of the product image is registered in the product database, a message is displayed on the display asking whether it is okay to update the memory of the feature values ​​of the product image.

5. 5. The packaged food identification system according to claim 1, wherein a code corresponding to the type of the identified product is displayed on the display.

6. a camera for capturing an image of a product consisting of a package and the food inside; a computer that identifies the type of product based on the product image obtained from the camera; a display that displays the product identification result by the computer, The computer a feature extraction unit that extracts features from product images obtained by a camera; a product database for registering products by storing product names, data specifying food labels to be attached to packages, and feature amounts of product images; a classification unit that identifies the type of product by comparing the feature amount extracted by the feature amount extraction unit with feature amounts of product images registered in a product database; When registering a new product in the product database, if the feature quantity extracted from the new product image is similar to the feature quantity of another product registered in the product database by more than a threshold value, the new product cannot be registered because a similar product is already registered. Once the product type is identified, the identification results are sent to a label printer, which prints a food label corresponding to the identified product and affixes it to the product package. When registering a new product in the product database, if the display indicates that the new product cannot be registered because a similar product has already been registered, the product packaging is changed. How to apply food labels to packaging.

Images