Image storage device for learning and product retrieval device

Abstract

The present invention provides a learning image storage device and a product retrieval device that can automatically remove abnormal images containing noise from multiple captured images and store the resulting captured images as training images. [Solution] The learning image storage device 10 is a learning image storage device that stores learning images for learning images of an object, and comprises an imaging unit 16 that images the object, a determination unit 11 that determines whether the image of the object is an abnormal image, and a storage unit 12 that stores at least a part of the image as a learning image. The determination unit 11 determines whether the image is an abnormal image based on a trained model of the object and the features of the image. The storage unit 12 stores the image that is determined not to be an abnormal image from among a plurality of images as a learning image.

Description

Technical Field

[0001] The present invention relates to a learning image storage device and a product calling device.

Background Art

[0002] In Patent Document 1, in an article processing device, when a new article is conveyed by a conveying unit and when the characteristics of the article change and cannot be handled by a pre-configured learned model, a technique for automatically collecting teacher data related to the article by associating and storing an image of the article with the article information is described.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When automatically collecting learning images (for example, teacher data) related to an article as in the above prior art, it is desirable to remove abnormal images including noise from a large number of images including the article. However, when there are a large number of images including the article, it is not easy for an operator to determine whether an image is an abnormal image.

[0005] An object of the present invention is to provide a learning image storage device and a product calling device that can store, as learning images, captured images from which abnormal images including noise have been automatically removed from a plurality of captured images.

Means for Solving the Problems

[0006] (1) A learning image storage device according to one aspect of the present invention is a learning image storage device that stores learning images for learning images of an article, comprising: an imaging unit that images an article; a determination unit that determines whether or not the image of the captured article is an abnormal image; and a storage unit that stores at least a part of the captured image as a learning image, wherein the determination unit determines whether or not the image is an abnormal image based on a pre-configured trained model of the article and the feature quantities of the captured image, and the storage unit stores the image from among the plurality of captured images that is determined not to be an abnormal image as a learning image.

[0007] In a learning image storage device according to one aspect of the present invention, a determination unit determines whether an image is an abnormal image based on a trained model relating to an item and the features of the captured image. A storage unit stores the captured images that are determined not to be abnormal from among a plurality of captured images as learning images. As a result, for example, each time an item is captured by the imaging unit, the captured images that are determined not to be abnormal are automatically stored sequentially as learning images. Therefore, according to a learning image storage device according to one aspect of the present invention, it is possible to store captured images from which abnormal images containing noise have been automatically removed as learning images.

[0008] (2) In (1) above, the determination unit determines that an image is an abnormal image if the degree of abnormality calculated based on the trained model and the features of the captured image is equal to or greater than a predetermined abnormality determination threshold, and the degree of abnormality may be calculated to be smaller the more recent the image was captured. In this case, the more recent the image was captured, the less likely it is to be determined that the image is an abnormal image, so it is possible to suppress the erroneous determination of an image that does not actually contain noise as an abnormal image, for example, when only a part of an item has been changed to the latest specifications.

[0009] (3) In (1) or (2) above, the determination unit determines that an image is an abnormal image based on a comparison between an abnormality score calculated based on the trained model and the features of the captured image and a predetermined abnormality determination threshold. The abnormality determination threshold may be an abnormality score value at which a predetermined percentage of the captured images are determined to be abnormal. In this case, for example, if the rate of abnormalities in which noise is present in multiple captured images is known in advance, the abnormality determination threshold can be set automatically.

[0010] (4) Another embodiment of the present invention provides a product calling device comprising the learning image storage device described in (1) or (2) above, an imaging unit for imaging products, a product information storage unit for storing product information relating to the type of product, and a calling unit for calling product information. The calling unit may identify the type of product by applying the feature quantities of the imaged product to a trained model trained using the learning images, and call product information corresponding to the identified type of product. In this case, for example, each time the imaging unit images a product, images that are determined not to be abnormal are automatically stored sequentially as learning images, and the type of product can be identified using the trained model trained using the learning images. [Effects of the Invention]

[0011] According to some aspects of the present invention, it is possible to automatically remove abnormal images containing noise from multiple captured images and store the resulting captured image as a training image. [Brief explanation of the drawing]

[0012] [Figure 1] This is an external perspective view showing a weighing and packaging device equipped with a learning image storage device and a product retrieval device according to an embodiment. [Figure 2] This is a block diagram showing the configuration of a learning image storage device and a product retrieval device according to the embodiment. [Figure 3] (a) is a diagram illustrating multiple training candidate images taken at different times. (b) is a diagram showing an example of an anomaly score. (c) is the judgment result when using this example of anomaly score. [Figure 4] (a) is a reproduction of the candidate images for training shown in Figure 3(a). (b) is a diagram showing other examples of anomaly scores. (c) is the judgment result when using other examples of anomaly scores. [Figure 5] (a) is a time-series graph showing the degree of anomaly when the image was taken at time t2. (b) is a time-series graph showing the degree of anomaly when the image was taken at time t4, which is later than time t2. [Figure 6] This is a flowchart illustrating the processing operation of a learning image storage device. [Figure 7] This is a flowchart illustrating the processing operation of a weighing and packaging device equipped with a product calling device. [Modes for carrying out the invention]

[0013] Hereinafter, one embodiment will be described in detail with reference to the attached drawings. In the description of the drawings, the same or equivalent elements will be denoted by the same reference numeral, and redundant descriptions will be omitted.

[0014] The weighing and packaging device 1 shown in Figures 1 and 2 is a device for weighing and packaging goods G (articles). By incorporating the pricing unit 4 described later into the weighing and packaging device 1, it is configured as a weighing, packaging, and pricing device. Goods G are articles to be weighed, packaged, and priced by the weighing and packaging device 1. Goods G include, for example, articles such as food products and a container in which the articles are placed or stored. If goods G are articles contained in a container, both the articles and the container may be packaged by the weighing and packaging device 1.

[0015] The weighing and packaging device 1 comprises a weighing and loading mechanism 2, a packaging section 3, a pricing unit 4, a display operation unit 5, and a control unit 6. Each mechanism included in the weighing and packaging device 1 is housed, for example, in the main body 1a of the weighing and packaging device 1 and in a casing 1b that houses film rolls, etc.

[0016] The weighing and loading mechanism 2 includes a weighing unit 13, a conveying unit 14, and an imaging unit 16. The weighing unit 13 is a weighing device that weighs the product G and outputs the weighing result (weight value) of the product G to the control unit 6. The control unit 6 subtracts the tare weight of a container or the like from the stable weighing value to calculate the net weight of the product G. The control unit 6 outputs an imaging command to the imaging unit 16. The weighing unit 13 has a placement part (not shown) such as a weighing pan on which the product G is placed.

[0017] The conveying unit 14 is a member that conveys the product G placed on the weighing unit 13 to the packaging unit 3 and includes, for example, a pusher conveyor, a belt conveyor, etc. The conveying unit 14 conveys, for example, the product G placed on the weighing unit 13 to the back side. Thereby, the product G is sent to the lifter mechanism 32 (described later) of the packaging unit 3.

[0018] The imaging unit 16 images the product G placed on the weighing unit 13 and acquires an imaging image of the product G. The imaging unit 16 is, for example, a CCD camera, a CMOS camera, etc. The imaging unit 16 is mounted on the bottom surface of the discharge table ES, for example, above the place where the product G of the weighing unit 13 is placed. The imaging image acquired by the imaging unit 16 is output to the learning image storage device 10 or the image processing unit 18.

[0019] The learning image storage device 10 is a device that stores learning images for learning images related to the product G. The learning image storage device 10 is directly or indirectly connected to the weighing and packaging device 1. When the learning image storage device 10 is indirectly connected to the weighing and packaging device 1, for example, the learning image storage device 10 and the weighing and packaging device 1 are connected via Ethernet (registered trademark). In this case, the learning image storage device 10 may be a cloud computer or the like arranged at a location different from the weighing and packaging device 1. In the example of FIG. 2, the learning image storage device 10 is a device separate from the weighing and packaging device 1, but may be built into the weighing and packaging device 1.

[0020] The learning image storage device 10 acquires a feature amount (hereinafter referred to as "first feature amount") indicating the features of a subject included in a captured image of a learning candidate product, which is the product G to be imaged for learning the image related to the product G (hereinafter referred to as "learning candidate captured image"). The subject of the learning candidate captured image is not limited to the product G and may include an object other than the product G imaged by the imaging unit 16. The learning image storage device 10 acquires the first feature amount including the features of the learning candidate product. The first feature amount includes, for example, features such as the features of the article itself, the features of the container on which the object is placed, the features of an object other than the learning candidate product, and the like. The features of the article itself are, for example, the shape, area ratio, color tone of each region distinguished by color, and the like. The features of the container include features such as the size, shape, and color of the container. The features of an object other than the product G include features that contribute to machine learning noise, such as a situation where an object of another product G is captured, a situation where an object such as a user's hand and scissors is captured, and a situation where at least a part of the product G is not captured. The first feature amount may be, for example, a vector including a predetermined number of numbers obtained through a neural network corresponding to the feature extraction layer of the learned model related to the article.

[0021] Functionally, the learning image storage device 10 includes a determination unit 11 and a storage unit 12. The determination unit 11 determines whether the learning candidate captured image is an abnormal image based on the learned model related to the article and the first feature amount of the learning candidate captured image. The learned model related to the article is a prediction model generated by machine learning for each article related to the learning candidate captured image, and is an inference program incorporated with parameters (learned parameters) obtained as a result of machine learning. The storage unit 12 stores at least a part of the learning candidate captured image as a learning image. The storage unit 12 stores the learning candidate captured image determined not to be an abnormal image among the plurality of learning candidate captured images as a learning image.

[0022] The trained model is pre-generated using machine learning from previously captured images, and can be updated using machine learning with newly captured candidate images. It is desirable to use normal images that do not contain features that would be considered noise in machine learning, rather than abnormal images containing such features, when updating the trained model. If objects other than the candidate products mentioned above are included in the candidate images, they can become noise in machine learning. However, when the number of candidate images becomes very large, it becomes practically difficult for users to visually determine whether a candidate image is abnormal or not.

[0023] Therefore, the determination unit 11 determines whether each of the multiple candidate images for training is an abnormal image, for example, based on the trained model generated up to the time of capturing the candidate images for training and the first feature quantities of each of the multiple candidate images for training captured after the generation or update of the trained model.

[0024] The determination unit 11, for example, obtains a first feature of the subject from each of the multiple candidate images for training. The determination unit 11 calculates an anomaly score based on the trained model generated up to the time the candidate images for training were captured and the first feature of the candidate images for training. The anomaly score is an index for determining whether or not a candidate image for training is an anomaly image. The determination unit 11, for example, applies the obtained first feature to the trained model relating to the product G that is the target of training, and uses an anomaly detection algorithm as a separation layer to calculate the anomaly score for each of the multiple candidate products for training. The anomaly score is calculated such that if the value is less than the anomaly judgment threshold, the product is normal, and if the value is greater than or equal to the anomaly judgment threshold, the product is not normal (it is anomaly).

[0025] The determination unit 11 determines that an image is an abnormal image if, for example, the abnormality score calculated based on the trained model and the first feature of the candidate image is greater than or equal to a predetermined abnormality determination threshold. The abnormality determination threshold is an abnormality score threshold used to determine whether or not a candidate image is an abnormal image. As an example, the determination unit 11 may set the abnormality determination threshold using a pre-set parameter. For example, the abnormality determination threshold may be 0. In this case, the determination unit 11 determines that a candidate image is an abnormal image if the calculated abnormality score is 0 or greater.

[0026] Figure 3(a) shows several training candidate images IM1 to IM6, arranged from newest to oldest, as examples of multiple training candidate images taken at different times. In the example in Figure 3(a), the product (e.g., meat on a tray) in the oldest training candidate image IM6 is in a normal state at the time of the oldest image. The product in training candidate image IM5 is different from the product in training candidate image IM6, and was in an abnormal state at the time training candidate image IM5 was taken. The product in training candidate image IM4 is the same as in training candidate image IM6 and is in a normal state. The product in training candidate image IM3 is the same as in training candidate image IM6, but part of the product is not visible, so it is in an abnormal state as a training candidate image.

[0027] Figure 3(b) shows the anomaly scores corresponding to the training candidate images IM1 to IM6 in Figure 3(a). The anomaly scores for training candidate images IM3 to IM6 are calculated to be 0.7, -0.9, 0.3, and -0.9, respectively. As shown in Figure 3(c), when the anomaly determination threshold is 0, the determination unit 11 determines that training candidate images IM4 and IM6, whose calculated anomaly scores are less than 0, are normal images, and determines that training candidate images IM3 and IM5, whose calculated anomaly scores are 0 or greater, are abnormal images.

[0028] In this way, the determination unit 11 automatically determines whether a candidate image is an abnormal image based on the trained model related to product G and the first feature of the candidate image. The storage unit 12 automatically stores the candidate images that are determined not to be abnormal as training images. Therefore, even if the number of candidate images becomes very large, the user does not need to visually determine whether a candidate image is abnormal, and the storage unit 12 can automatically store appropriate training images.

[0029] Here, even if no objects other than product G are included in the training candidate image, as described above, if the anomaly score calculated based on the trained model and the first feature of the training candidate image is used as is, there is a possibility that the training candidate image may be incorrectly judged as an anomaly image, for example, if the specifications of product G have recently been changed. Examples of changes in the specifications of product G include changes in the shape or size of the container on which the item is placed or stored, changes in the arrangement of the item in the container, and changes in the color of the item.

[0030] For example, in the example shown in Figure 3(a), the product shown in the most recently captured candidate image IM1 is assumed to have only the tray color changed compared to the product in candidate image IM6. The product shown in the next most recently captured candidate image IM2 is assumed to have the tray shape changed to be longer in the longitudinal direction and the arrangement of the meat changed compared to the product in candidate image IM6. Based on the trained model and the first feature of the candidate images, the anomaly scores are calculated to be 0.1 and 0.2, respectively, as shown in Figure 3(b), and are greater than or equal to 0. In this case, if the anomaly scores are used as they are, the determination unit 11 will incorrectly determine that candidate images IM1 and IM2 are abnormal images, even though they were actually normal at the time they were captured, as shown in Figure 3(c).

[0031] Therefore, the determination unit 11 calculates a smaller anomaly correction value for learning candidate images that are more recently captured. The anomaly correction value is a weighted value of the anomaly that takes into account the possibility that learning candidate images captured more recently are not anomaly images. The determination unit 11 uses the calculated anomaly correction value to calculate the corrected anomaly, which is corrected according to the capture time. In Figure 4(b), as another example of anomaly, the corrected anomaly corresponding to the learning candidate images IM1 to IM6 in Figure 4(a) is shown on the right side. The corrected anomaly is, for example, the sum of the uncorrected anomaly in Figure 3(b) and the anomaly correction value.

[0032] In the example shown in Figure 4(b), the abnormality correction value is set to -0.5 for the most recently acquired training candidate image IM1. The abnormality correction values ​​are set to -0.4 for training candidate image IM2, -0.3 for training candidate image IM3, -0.2 for training candidate image IM4, -0.1 for training candidate image IM5, and 0.0 for training candidate image IM6 as the acquisition date becomes progressively older. By correcting the abnormality in this way, as shown in Figure 4(c), the determination unit 11 can determine that training candidate images IM1 and IM2 are not abnormal images (they are normal images). Therefore, even for training candidate images IM1 and IM2 that were recently acquired, it is possible to suppress the incorrect determination that training candidate images IM1 and IM2 are abnormal images and to automatically store them in the storage unit 12 as appropriate training images.

[0033] "Most recent imaging time" refers to the imaging time when the latest candidate learning product was imaged. At the imaging time when the latest candidate learning product was imaged, the abnormality correction value used by the determination unit 11 to determine whether the candidate learning image is an abnormal image is set to the abnormality correction value for the most recent imaging time (e.g., -0.5). As shown in Figure 5(a), if a candidate learning product is imaged at time t1 to obtain the candidate learning image IM6, and the latest candidate learning product is imaged at time t2 to obtain the candidate learning image IM1, the abnormality correction value graph W1 will increase linearly from time t2 to time t1 with a predetermined slope, according to the value shown in the example in Figure 4(b), and will be 0.0 before time t1. As shown in Figure 5(b), if, after time t2, a candidate learning product is imaged at time t3 to obtain the candidate learning image, and the latest candidate learning product is imaged at time t4 to obtain the candidate learning image, the abnormality correction value will be smallest at time t4, when the imaging time is most recent (e.g., -0.5), and will increase to 0 as the imaging times become progressively older. The graph W2 of the anomaly correction value increases linearly with a predetermined slope from time t4 to time t3, and is 0.0 before time t3.

[0034] The anomaly correction value is not limited to these examples. For example, it may be increased non-linearly from the value of the most recent imaging time, or it may not be kept constant at 0.0 even when going back in time. The anomaly correction value does not have to be increased in steps; it may be given in a step-like manner over time, such as correcting the anomaly only within a predetermined time range past the most recent imaging time, and not correcting the anomaly before that range.

[0035] The memory unit 12 stores the first feature quantities of training candidate images that have been determined not to be abnormal (i.e., to be normal images). Each of the multiple first feature quantities stored in the memory unit 12 is associated with a corresponding product G. By automatically storing training images, the memory unit 12 can update the trained model using various methods.

[0036] The timing for the memory unit 12 to update the trained model may be at a prior time, separate from the weighing and packaging of the products using the weighing and packaging device 1. Alternatively, the weighing and packaging of the products using the weighing and packaging device 1 may be performed simultaneously with the updating of the trained model, by treating a portion of the target products G, which are the products to be weighed and packaged by the weighing and packaging device 1, as trained candidate products.

[0037] The image processing unit 18 acquires feature quantities (hereinafter referred to as "secondary feature quantities") that represent the characteristics of the target product G. The image processing unit 18 is connected to the imaging unit 16. The image processing unit 18 may be, for example, a processor computer such as a CPU (Central Processing Unit) installed in the imaging unit 16. The image processing unit 18 outputs the acquired secondary feature quantities to the control unit 6. The secondary feature quantities of the target product G include, for example, the characteristics of the target product G itself, and the characteristics of the container on which the target product G is placed. The characteristics of the target product G itself include, for example, the shape of each region distinguished by color, the area ratio, and the color tone of each region. The characteristics of the container include the size, shape, and color of the container.

[0038] The packaging unit 3 wraps the target product G, which has been transported by the transport unit 14, with film. The packaging unit 3 covers the target product G, which has been transported from the weighing unit 13, with film. Specifically, the packaging unit 3 has a film transport mechanism 31, a lifter mechanism 32, a folding mechanism 33, and a sealing mechanism 34.

[0039] The film transport mechanism 31 is a mechanism that pulls out the film (not shown) to be used to package the target product G from the film roll and transports it to the packaging position. The film pulled out from a single film roll by the film transport mechanism 31 is held under tension at the packaging position.

[0040] The lifter mechanism 32 is a mechanism that raises the target product G to the packaging position, pushing the target product G upward against the film that is held taut by the film transport mechanism 31. The folding mechanism 33 folds the periphery of the film that protrudes from the target product G into the bottom surface of the target product G. The sealing mechanism 34 heat-seals the overlapping portion of the film folded by the folding mechanism 33. The packaged target product G is discharged toward the discharge platform ES.

[0041] The pricing unit 4 issues a pricing label for the target product G and affixes it to the packaged target product G. The label printer LP prints information about the item (product information) on the label. Product information includes, for example, the product name, unit price, and additives, and this information is read from the product master 64 (product information storage unit) of the control unit 6. The label application mechanism LI affixes the label printed by the label printer LP to the packaged target product G.

[0042] The display operation unit 5 displays product information, imaging results from the imaging unit 16, etc., and is a mechanism (interface) that accepts user operations on the weighing and packaging device 1. The display operation unit 5 may also have a display operation unit 51 and a notification unit 52.

[0043] The notification unit 52 may issue a warning if the feature quantity transmitted from the learning image storage device 10 (hereinafter referred to as the "third feature quantity") and the second feature quantity obtained from the target product G placed on the weighing unit 13 do not match.

[0044] The control unit 6 is located inside the main body 1a and controls the operation of each of the above mechanisms. Therefore, the control unit 6 is composed of a computer having a storage medium 61 such as a ROM (Read Only Memory) that stores programs and information, a RAM (Random Access Memory) that temporarily stores data, an HDD (Hard Disk Drive), a CPU, and communication circuits. The control unit 6 has a storage medium 61, a call unit 62, a control unit 63, and a product master 64. The product master 64 is a storage section that stores product information related to the type of product. The product master 64 stores data such as the unit price and name of multiple types of products, and product information such as the size, shape, material, and tare weight of multiple types of trays.

[0045] The weighing and packaging device 1 is configured to allow the target product G to be retrieved by the product retrieval device 100. The product retrieval device 100 comprises the learning image storage device 10 described above, the imaging unit 16 described above, the product master 64, and the retrieval unit 62 for retrieving product information.

[0046] The storage medium 61 stores packaging parameters for packaging the target product G. The storage medium 61 also temporarily stores the third feature quantity transmitted from the learning image storage device 10.

[0047] The calling unit 62 identifies the type of product G (hereinafter referred to as "calling product") based on the feature quantities (hereinafter referred to as "fourth feature quantity") of the captured image of the product G for calling product information, and calls the product information corresponding to the identified type of product G. The calling product may be a product dedicated to calling, or a part of the target products (for example, the first target product in a series of target products) may be used for calling.

[0048] The fourth feature is a feature that indicates the characteristics of the subject contained in the captured image of the item to be called. The fourth feature may be acquired by the storage unit 12 using the captured image of the item to be called captured by the imaging unit 16. The fourth feature may be a vector containing a predetermined number of digits obtained through a neural network corresponding to the feature extraction layer of a trained model.

[0049] The calling unit 62 identifies the type of product to be called by, for example, applying the acquired fourth feature to a trained model trained using training images, and outputting the type of product to be called using a product discrimination algorithm as a separation layer. The calling unit 62 identifies the product corresponding to the identified type of product to be called as the current target product, reads the product information of the target product from the product master 64, and displays it on the display operation unit 51.

[0050] The control unit 63 is the main part of the control unit 6 and controls the weighing unit 13, the transport unit 14, and the packaging unit 3, etc. The control unit 63 sets the product corresponding to the type of product to be called, which has been identified by the calling unit 62, as the target product at the moment, and retrieves various data related to the set target product from the storage medium 61 and the product master 64, and outputs it to the display operation unit 5, the label printer LP of the pricing unit 4, and the packaging unit 3. Based on the weighing value obtained from the weighing unit 13, the control unit 63 calculates the net weight and price of the target product G and outputs this as print information to the label printer LP.

[0051] The control unit 63 determines, based on the third feature and the second feature, whether the placed target product G matches the target product that was called at the present time. The control unit 63 may also determine, based on the comparison result between the third feature and the second feature, whether or not to transport the target product G to the packaging unit 3 by the transport unit 14.

[0052] Next, the processing operation of the learning image storage device 10 will be explained with reference to Figure 6. First, images of multiple learning candidate products (items) are taken (step S11). For example, the user manually places multiple products G sequentially onto the weighing unit 13 as learning candidate products. The multiple products G may also be automatically supplied sequentially to the weighing unit 13. The control unit 63 instructs the imaging unit 16 to sequentially start imaging each product G. The imaging unit 16 sequentially outputs the captured images of the multiple products G to the determination unit 11. At this time, the user may specify the product call number corresponding to the learning candidate product.

[0053] Next, the system acquires first features from multiple candidate images for training (step S12). The determination unit 11 acquires features that indicate the characteristics of the subjects included in the candidate images for training. Next, the system calculates the anomaly score for multiple candidate products for training based on the trained model for the items and the first features (step S13). For example, the determination unit 11 applies the acquired first features to the trained model for product G corresponding to the call number and calculates the anomaly score for each of the multiple candidate products for training using an anomaly detection algorithm as a separation layer.

[0054] Next, the abnormality correction value is calculated and the abnormality is corrected according to the acquisition time of multiple training candidate images (step S14). The determination unit 11 calculates a smaller abnormality correction value for training candidate images that were acquired more recently. The determination unit 11 uses the calculated abnormality correction value to calculate the corrected abnormality according to the acquisition time. Next, the abnormality judgment threshold is set (step S15). For example, the determination unit 11 sets the abnormality judgment threshold to 0.

[0055] Next, it is determined whether the corrected abnormality level is equal to or greater than a predetermined abnormality determination threshold (step S16). The determination unit 11 determines whether the training candidate image is an abnormal image by, for example, comparing the corrected abnormality level with the abnormality determination threshold.

[0056] For example, if the degree of abnormality after correction is greater than or equal to the abnormality detection threshold (step S16: YES), the determination unit 11 determines that the training candidate image is an abnormal image (step S17). Then, the storage unit 12 does not store the training candidate image as a training image (step S17). After that, the training image storage device 10 terminates the processing operation shown in Figure 6. On the other hand, if the degree of abnormality after correction is less than the abnormality detection threshold (step S16: NO), the determination unit 11 determines that the training candidate image is not an abnormal image (it is normal) (step S19). Then, the storage unit 12 stores the training candidate image as a training image (step S20). After that, the training image storage device 10 terminates the processing operation shown in Figure 6.

[0057] Next, the processing operation of the product calling device 100 will be explained with reference to Figure 7. First, the product to be called is placed on the weighing unit and an image of the product to be called is taken (step S31). For example, the user manually places the product to be called on the weighing unit 13. Next, the type of product to be called is identified by applying the fourth feature of the image of the product to a trained model that has been trained using training images (step S32). The image processing unit 18 acquires the fourth feature, which indicates the characteristics of the subject contained in the image of the product to be called. The calling unit 62 identifies the type of product to be called by applying the acquired fourth feature to a trained model that has been trained using training images, for example, and outputting the type of product to be called using a product discrimination algorithm as a separation layer.

[0058] Next, the third feature quantity and product information corresponding to the identified type of product to be called are retrieved and set (displayed) (step S33). The third feature quantity corresponding to the type of product to be called identified by the calling unit 62 is read from the learning image storage device 10.

[0059] Next, the target product is placed on the weighing unit and the target product is imaged (step S34). For example, the user manually places multiple target products on the weighing unit 13 in sequence. The control unit 63 inputs the weighed value of the target product output from the weighing unit 13 (step S35), and based on this weighed value, if it determines that the target product is placed on the weighing unit 13 or that the weighed value of the target product is stable, the control unit 63 instructs the imaging unit 16 to start imaging the target product.

[0060] Next, the image processing unit acquires the second feature of the captured image of the target product (step S36). The imaging unit 16 outputs the captured image of the target product to the image processing unit 18. The image processing unit 18 acquires the second feature of the target product from the captured image acquired by the imaging unit 16.

[0061] Next, the control unit 63 compares the third feature and the second feature (step S37). If the control unit 63 determines that the third feature and the second feature do not match (step S38: NO), a warning is displayed by the display operation unit 5 (step S39). After that, the operation of the weighing and packaging device 1 is terminated.

[0062] If it is determined that the third feature and the second feature match (step S38: YES), the control unit 63 determines whether the measured value is stable or not. If the measured value of the target product is not stable (step S40: NO), the measured value is entered again (step S41).

[0063] When the weighing value stabilizes (step S40: YES), the control unit 63 outputs the net weight and pricing data such as price, calculated based on the stable weighing value, to the label printer LP (step S42). The label printer LP creates print information based on the product information and pricing data entered in step S33, prints this print information on a label, and issues it.

[0064] The control unit 63 drives the transport unit 14 to transport the target product placed on the weighing unit 13 to the packaging unit 3 and package the target product (step S43). The label printer LP issues a printed label, and when the packaged target product is discharged to the discharge table ES, the label application mechanism LI attaches the issued label to the packaged target product (step S44).

[0065] As explained above, with the learning image storage device 10, the determination unit 11 determines whether a candidate image is an abnormal image based on the trained model related to product G and the first feature quantity of the candidate image. The storage unit 12 stores the candidate images that are determined not to be abnormal from among the multiple candidate images as learning images. As a result, for example, each time the imaging unit 16 captures a candidate product, candidate images that are determined not to be abnormal are automatically stored sequentially as learning images. Therefore, with the learning image storage device 10, candidate images that have had abnormal images containing noise automatically removed from a plurality of candidate images can be stored as learning images.

[0066] The determination unit 11 determines that a candidate image is an abnormal image if the abnormality score calculated based on the trained model and the first feature of the candidate image is greater than or equal to a predetermined abnormality determination threshold. The determination unit 11 calculates a smaller abnormality score the more recent the image was captured. As a result, the more recent the image was captured, the less likely it is to be determined that the candidate image is an abnormal image. This prevents the system from mistakenly determining a candidate image that does not actually contain noise as an abnormal image, for example, when only a part of the candidate product has been updated to the latest specifications.

[0067] The product retrieval device 100 comprises the above-mentioned learning image storage device 10, an imaging unit 16 for imaging product G, a product master 64 (product information storage unit) for storing product information related to the type of product G, and a retrieval unit 62 for retrieving product information. The retrieval unit 62 identifies the type of product to be retrieved by applying the fourth feature quantity of the image of the retrieved product to a trained model trained using the learning images, and retrieves product information corresponding to the identified type of product to be retrieved. As a result, for example, each time the imaging unit 16 images a product to be retrieved, candidate images determined not to be abnormal are automatically stored sequentially as learning images, and the type of product to be retrieved can be identified using the trained model trained using the learning images.

[0068] Although one embodiment relating to one aspect of the present invention has been described above, the aspect of the present invention is not limited to the above embodiment.

[0069] For example, in the above embodiment, the determination unit 11 determined that the training candidate image was an abnormal image when the degree of abnormality calculated based on the trained model and the first feature of the training candidate image was greater than or equal to a predetermined abnormality determination threshold, but the unit is not limited to this example. For example, the determination unit may determine that the training candidate image is an abnormal image when the degree of normality calculated based on the trained model and the first feature of the training candidate image is less than or equal to a predetermined normality determination threshold. In this case, the more recent the image was captured, the higher the degree of normality calculated may be.

[0070] In the above embodiment, the determination unit 11 used a preset parameter as the abnormality determination threshold, but is not limited to this example. For example, the abnormality determination threshold may be an abnormality score value at which a predetermined percentage of the training candidate images are determined to be abnormal. The determination unit 11 may set the abnormality determination threshold based on the calculated abnormality score and the number of training candidate images, such that the abnormality score of a predetermined percentage of the training candidate images is equal to or greater than the abnormality determination threshold. In this case, for example, if the rate of abnormality occurrence where noise is present in multiple training candidate images is known in advance, the determination unit 11 can automatically set the abnormality determination threshold.

[0071] In the above embodiment, the learning image storage device 10 was configured as part of the product calling device 100 in the weighing and packaging device 1, but the invention is not limited to this example. For example, the learning image storage device may be configured as part of a product calling device in a POS system, weighing machine, label issuing device, or other device. Alternatively, the learning image storage device may be used independently to store images obtained by automatically removing abnormal images containing noise from a plurality of captured images as learning images. [Explanation of symbols]

[0072] 10...Image storage device for learning, 11...Determination unit, 12...Storage unit, 16...Imaging unit, 62...Calling unit, 64...Product master (product information storage unit), 100...Product calling device.

Claims

1. A learning image storage device that stores learning images for learning images related to objects, An imaging unit for imaging the aforementioned article, A determination unit that determines whether the captured image of the captured article is an abnormal image, The system includes a storage unit that stores at least a portion of the captured image as the learning image, The determination unit determines whether the captured image is the abnormal image based on the trained model relating to the article and the feature quantities of the captured image. The memory unit is a learning image storage device that stores, among a plurality of captured images, the captured images that have been determined not to be abnormal images as the learning images.

2. The determination unit, If the degree of abnormality calculated based on the trained model and the features of the captured image is equal to or greater than a predetermined abnormality determination threshold, the captured image is determined to be an abnormal image. The learning image storage device according to claim 1, wherein the degree of abnormality is calculated to be smaller the more recent the time the captured image was taken.

3. The determination unit, Based on the comparison result between the anomaly score calculated using the trained model and the features of the captured image and a predetermined anomaly detection threshold, it is determined that the captured image is an anomaly image. The learning image storage device according to claim 1 or 2, wherein the abnormality determination threshold is the value of the degree of abnormality at which a predetermined proportion of the captured images are determined to be abnormal.

4. A learning image storage device according to claim 1 or 2, An imaging unit that takes images of the product, A product information storage unit that stores product information relating to the type of product, It comprises a calling unit for calling the aforementioned product information, The calling unit identifies the type of product by applying the feature quantities of the captured image of the product to the trained model trained using the training images, and calls up product information corresponding to the identified type of product.

Images