Learning image storage device and product calling device

Abstract

The invention provides a learning image storage device and a commodity calling device. The learning image storage device stores a learning image for learning an image related to an article. The learning image storage device includes: an image capturing unit that captures an image of an article; a determination unit that determines whether or not the captured image of the captured article is an abnormal image; and a storage unit that stores at least a portion of the captured image as a learning image. The determination unit determines whether or not the captured image is an abnormal image based on a learning completion model relating to the article and a feature amount of the captured image, and the storage unit stores, as a learning image, a captured image determined not to be an abnormal image among the plurality of captured images.

Description

Technical Field

[0001] This invention relates to an image storage device for learning and a product retrieval device. Background Technology

[0002] Patent Document 1 describes the following technology: In an article processing device, when a new article is conveyed by a conveyor unit, or when the characteristics of the article change and cannot be handled by a pre-built, learned model, the image of the article is associated with the article information and stored, thereby automatically collecting training data related to the article.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent No. 7368834 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] As with the prior art described above, when automatically collecting images related to items for learning purposes (e.g., training data), it is desirable to remove noisy and anomalous images from a large number of images containing items. However, when there are a large number of images containing items, it is not easy for the operator to determine whether an image is anomalous.

[0008] The purpose of this invention is to provide a learning image storage device and a product retrieval device, which can store a captured image, after automatically removing noisy abnormal images from multiple captured images, as a learning image.

[0009] Methods for solving problems

[0010] (1) A learning image storage device according to one aspect of the present invention, which stores learning images for learning images related to an object, comprises: a shooting unit that shoots an object; a determination unit that determines whether the shot image of the object is an abnormal image; and a storage unit that stores at least a portion of the shot images as learning images, wherein the determination unit determines whether the shot image is an abnormal image based on a pre-constructed learning model related to the object and feature quantities of the shot image, and the storage unit stores the shot images that are determined not to be abnormal images from a plurality of shot images as learning images.

[0011] In one aspect of the learning image storage device of the present invention, the determination unit determines whether a captured image is an anomalous image based on a learned model associated with an object and feature quantities of the captured image. The storage unit stores the captured images determined not to be anomalous images from a plurality of captured images as learning images. Thus, for example, whenever an object is captured by the capturing unit, the captured images determined not to be anomalous images are automatically stored sequentially as learning images. Therefore, according to one aspect of the learning image storage device of the present invention, it is possible to store captured images from a plurality of captured images after automatically removing anomalous images containing noise as learning images.

[0012] (2) In (1) above, the determination unit may determine that the captured image is an anomalous image if the anomalousness calculated based on the learned model and the feature values ​​of the captured image is above a predetermined anomalousness determination threshold. The later the captured image was taken, the smaller the anomalousness will be calculated. In this case, the later the captured image was taken, the more difficult it is to determine that the captured image is an anomalous image. Therefore, for example, in cases where only a part of the item has been changed to the latest specifications, it is possible to suppress the situation where a captured image that does not actually contain noise is mistakenly determined to be an anomalous image.

[0013] (3) In (1) or (2) above, the determination unit may determine whether a captured image is an anomalous image based on a comparison between the anomaly degree calculated based on the features of the learned model and the captured image and a predetermined anomaly determination threshold. The anomaly determination threshold is the anomaly degree value of a predetermined proportion of captured images that are determined to be anomalous. In this case, for example, if the anomaly occurrence rate of noise in multiple captured images is known in advance, the anomaly determination threshold can be automatically set.

[0014] (4) Alternatively, the product retrieval device of the present invention may include: the learning image storage device described in (1) or (2) above; a shooting unit for shooting a product; a product information storage unit for storing product information related to the type of the product; and a retrieval unit for retrieving product information. The retrieval unit applies the feature values ​​of the captured product image to a learned model that has been trained using the learning image, thereby determining the type of product and retrieving product information corresponding to the determined type of product. In this case, for example, whenever a product is shot by the shooting unit, images that are determined not to be abnormal images are automatically stored sequentially as learning images, and the type of product can be determined using the learned model that has been trained using the learning image.

[0015] Invention Effects

[0016] According to some aspects of the present invention, it is possible to store images taken from multiple images after automatically removing anomalous images containing noise as learning images. Attached Figure Description

[0017] Figure 1 This is a perspective view showing the external appearance of a metering and packaging device having an image storage device for learning and a product recall device according to an embodiment.

[0018] Figure 2 This is a block diagram illustrating the structure of the learning image storage device and the product retrieval device according to the embodiments.

[0019] Figure 3 (a) is a diagram illustrating multiple learning candidate images taken at different times.

[0020] Figure 3 (b) is a graph showing an example of anomaly.

[0021] Figure 3 (c) is a graph showing the judgment result in an example case where anomaly degree was used.

[0022] Figure 4 (a) is shown again Figure 3 (a) is a picture of the learning candidate shooting image.

[0023] Figure 4 (b) is a graph showing other examples of anomalies.

[0024] Figure 4 (c) is a graph showing the judgment results in other cases where anomaly degree was used.

[0025] Figure 5 (a) is a time series diagram showing the anomaly degree at time t2 when the shooting period is t2.

[0026] Figure 5 (b) is a time series diagram showing the anomaly at time t4 after the shooting period t2.

[0027] Figure 6 This is a flowchart used to illustrate the processing actions of an image storage device for learning.

[0028] Figure 7 This is a flowchart illustrating the processing actions of a metering and packaging device equipped with a product dispensing device. Detailed Implementation

[0029] Hereinafter, one embodiment will be described in detail with reference to the accompanying drawings. Furthermore, in the description of the drawings, the same or equivalent elements are labeled with the same reference numerals, and repeated descriptions are omitted.

[0030] Figure 1 and Figure 2The measuring and packaging apparatus 1 shown is a device for measuring and packaging commodity G (article). By introducing the pricing unit 4 (described later) into the measuring and packaging apparatus 1, it is configured as a measuring and packaging pricing device. Commodity G is a measured article that is measured, packaged, and priced by the measuring and packaging apparatus 1. Commodity G may include, for example, food items and containers for holding or storing those items. When commodity G is an article contained in a container, the measuring and packaging apparatus 1 may package both the article and the container.

[0031] The metering and packaging device 1 includes a metering and feeding mechanism 2, a packaging section 3, a pricing unit 4, a display and operation unit 5, and a control unit 6. The various mechanisms included in the metering and packaging device 1 are housed, for example, within the main body 1a of the metering and packaging device 1 and the housing 1b for storing film rolls, etc.

[0032] The metering and feeding mechanism 2 includes a metering unit 13, a conveying unit 14, and a photographing unit 16. The metering unit 13 is a metering device for measuring the commodity G, outputting the metering result (metered value) of the commodity G to the control unit 6. The control unit 6 calculates the net weight of the commodity G by subtracting the tare weight of the container, etc., from the measured value in a stable state. The control unit 6 outputs a photographing command to the photographing unit 16. The metering unit 13 includes a mounting section (not shown) for holding the commodity G, such as a metering tray.

[0033] The conveying unit 14 is a component that transports the product G placed in the metering unit 13 to the packaging unit 3, and may include, for example, a pusher conveyor or a belt conveyor. The conveying unit 14, for example, transports the product G placed in the metering unit 13 inwards. Thus, the product G is transported to the lifting mechanism 32 (described later) of the packaging unit 3.

[0034] The imaging unit 16 captures an image of the product G placed on the measuring unit 13. The imaging unit 16 may be, for example, a CCD camera or a CMOS camera. The imaging unit 16 is, for example, mounted on the bottom surface of the discharge table ES above the position where the product G is placed on the measuring unit 13. The image captured by the imaging unit 16 is output to the learning image storage device 10 or the image processing unit 18.

[0035] The learning image storage device 10 is a device for storing learning images related to the product G. The learning image storage device 10 is directly or indirectly connected to the measuring and packaging device 1. In the case where the learning image storage device 10 is indirectly connected to the measuring and packaging device 1, for example, it is connected via Ethernet (registered trademark). In this case, the learning image storage device 10 may also be a cloud computer or the like, located in a different location from the measuring and packaging device 1. Figure 2 In the example, the learning image storage device 10 is a different device from the metering and packaging device 1, but it can also be built into the metering and packaging device 1.

[0036] The learning image storage device 10 acquires a feature quantity (hereinafter referred to as "first feature quantity") representing the features of the subject contained in the captured image of product G (hereinafter referred to as "learning candidate product image") taken for learning images related to product G. The subject of the learning candidate product image is not limited to product G, and may include objects other than product G captured by the imaging unit 16. The learning image storage device 10 acquires the first feature quantity including the features of the learning candidate product. The first feature quantity includes, for example, features of the item itself, features of the container holding the object, features of objects other than the learning candidate product, etc. Features of the item itself include, for example, the shape, area ratio, and hue of each area distinguished by color. Features of the container include features such as the size, shape, and color of the container. Features of objects other than product G include, for example, features that capture other objects of product G, capture objects such as the user's hand and scissors, or capture at least a part of product G, which are factors that contribute to noise in machine learning. The first feature quantity may also be, for example, a vector containing a predetermined number of numbers obtained by the neural network corresponding to the feature extraction layer of the model learned in relation to the item.

[0037] The learning image storage device 10 has a determination unit 11 and a storage unit 12 as functional structures. The determination unit 11 determines whether a learning candidate image is an anomalous image based on a completed learning model associated with an item and a first feature quantity of a learning candidate image. The completed learning model associated with an item is a predictive model generated by machine learning for each item associated with the learning candidate image; it is an inference program that incorporates parameters (complete learning parameters) obtained from the results of machine learning. The storage unit 12 stores at least a portion of the learning candidate images as learning images. The storage unit 12 stores learning candidate images that are determined not to be anomalous images from a plurality of learning candidate images as learning images. The determination unit 11 and the storage unit 12 can be implemented, for example, by one or more processors executing programs stored in memory. Alternatively, these functional units can also be configured as dedicated hardware circuits for performing each function.

[0038] The learned model is pre-generated using machine learning based on past captured images and can be updated using machine learning with newly captured candidate images. The candidate images used to update the learned model aim to avoid using anomalous images containing features that contribute to machine learning noise, and instead use normal images that do not contain features that contribute to machine learning noise. If a candidate image contains objects other than the candidate product mentioned above, it may become noise for machine learning. If the number of candidate images becomes very large, it becomes difficult for a user to visually determine whether a candidate image is anomalous.

[0039] Therefore, the determination unit 11 determines, for example, whether each of the plurality of learning candidate images is an abnormal image based on the first feature value of each of the learned model generated before the learning candidate images are captured and the plurality of learning candidate images captured after the learned model is generated or updated.

[0040] The determination unit 11, for example, obtains a first feature value of the subject from each of a plurality of candidate images for learning. Based on the completed learning model generated before capturing the candidate images and the first feature value of the candidate images, the determination unit 11 calculates anomaly scores. Anomaly scores are an indicator used to determine whether a candidate image is an anomalous image. The determination unit 11, for example, applies the obtained first feature value to the completed learning model related to the product G, which is the object of learning, and uses an anomaly detection algorithm as a separation layer to calculate anomaly scores for each of the plurality of candidate products for learning. The anomaly scores are calculated; if the score is less than an anomaly determination threshold, the product is normal; if the score is greater than or equal to the anomaly determination threshold, the product is abnormal (abnormal).

[0041] For example, if the anomaly degree calculated based on the learned model and the first feature value of the candidate image being learned is above a predetermined anomaly determination threshold, the determination unit 11 determines that the captured image is an anomalous image. The anomaly determination threshold is a threshold used to determine whether the candidate image being learned is an anomalous image. As an example, the determination unit 11 can use a pre-set parameter to set the anomaly determination threshold. For example, the anomaly determination threshold can be 0. In this case, the determination unit 11 determines that the candidate image being learned is an anomalous image if the calculated anomaly degree is 0 or above.

[0042] exist Figure 3 In (a), multiple learning candidate images from different shooting periods are shown in chronological order from latest to earliest, representing learning candidate images IM1 to IM6. Figure 3 In example (a), the product captured in the earliest learning candidate image IM6 (e.g., lean meat on a tray) was in a normal state at the early stage of the shooting. In learning candidate image IM5, the product captured was different from that in learning candidate image IM6, and was in an abnormal state at the time learning candidate image IM5 was taken. In learning candidate image IM4, the product captured was the same as that in learning candidate image IM6, and was in a normal state. In learning candidate image IM3, although the product captured was the same as that in learning candidate image IM6, a part of the product was not captured, so as a learning candidate image, it was in an abnormal state.

[0043] exist Figure 3 (b) shows the Figure 3 (a) The anomalies corresponding to the learning candidate images IM1–IM6. The anomalies corresponding to the learning candidate images IM3–IM6 were calculated as 0.7, -0.9, 0.3, and -0.9, respectively. Figure 3 As shown in (c), when the anomaly determination threshold is 0, the determination unit 11 determines that the learning candidate images IM4 and IM6 with a calculated anomaly degree of less than 0 are normal images, and determines that the learning candidate images IM3 and IM5 with a calculated anomaly degree of more than 0 are abnormal images.

[0044] Thus, the determination unit 11 automatically determines whether a learning candidate image is an anomalous image based on the completed learning model related to the product G and the first feature value of the learning candidate image. The storage unit 12 automatically stores the learning candidate images that are determined not to be anomalous images from among the multiple learning candidate images as learning images. Therefore, even if the number of learning candidate images becomes very large, the process of the user visually judging whether the learning candidate images are anomalous images can be eliminated, and the storage unit 12 can automatically store appropriate learning images.

[0045] Here, if the candidate images being studied do not contain objects other than the aforementioned product G, for example, if the specifications of product G have recently been changed, directly using the anomaly degree calculated based on the first feature value of the learned model and the candidate images might mistakenly classify the candidate images as anomalous. Changes in the specifications of product G could include, for example, changes in the shape or size of the container holding or storing the items, changes in the arrangement of the items within the container, or changes in the color of the items.

[0046] For example, in Figure 3 In example (a), compared to the product in the learning candidate image IM6, the latest-period image taken during the learning candidate period, the only difference is the color of the tray. Compared to the product in the learning candidate image IM6, the product in the next-latest-period image taken during the learning candidate period, IM2, has a tray that is longer in the longitudinal direction, and the arrangement of the lean meat is different. If the calculation is based on the first feature quantity of the learned model and the learning candidate images, then as follows... Figure 3 As shown in (b), the anomaly scores were calculated to be 0.1 and 0.2, respectively, both above 0. In this case, if the anomaly score is used directly, the candidate images IM1 and IM2 would actually not be in an abnormal state at the time of capture, but... Figure 3 As shown in (c), the determination unit 11 may mistakenly determine that the learning candidate images IM1 and IM2 are abnormal images.

[0047] Therefore, the later the shooting date of the candidate image being studied, the smaller the anomaly correction value calculated by the determination unit 11. The anomaly correction value is a weighted value that considers the possibility that a candidate image shot later in time is less likely to be an anomalous image. The determination unit 11 uses the calculated anomaly correction value to calculate the corrected anomaly degree based on the shooting date. Figure 4 In (b), as another example of anomaly, the right side shows... Figure 4 (a) The corrected anomalies corresponding to candidate images IM1 to IM6. The corrected anomalies are, for example,... Figure 3 (b) is the sum of the anomaly degree before correction and the anomaly degree correction value.

[0048] exist Figure 4 In example (b), the anomaly correction value is assigned -0.5 for the latest-captured candidate image IM1. As the capture date becomes earlier, the anomaly correction value is assigned -0.4 for candidate image IM2, -0.3 for candidate image IM3, -0.2 for candidate image IM4, -0.1 for candidate image IM5, and 0.0 for candidate image IM6. This anomaly correction, as... Figure 4 As shown in (c), the determination unit 11 can determine that the learning candidate images IM1 and IM2 are not abnormal images (but normal images). Therefore, for the learning candidate images IM1 and IM2 that were captured later, it is possible to suppress the situation where the learning candidate images IM1 and IM2 are mistakenly determined to be abnormal images, and the storage unit 12 automatically stores them as appropriate learning images.

[0049] "Latest shooting period" refers to the shooting period in which the latest learning candidate product was captured. During the shooting period in which the latest learning candidate product was captured, the anomaly correction value used by the determination unit 11 to determine whether the captured image of the learning candidate is an anomalous image is set to the anomaly correction value for the latest shooting period (e.g., -0.5). Figure 5 As shown in (a), when a learning candidate product is photographed at time t1 to obtain learning candidate image IM6, and the latest learning candidate product is photographed at time t2 to obtain learning candidate image IM1, the anomaly correction value curve W1 becomes the following curve: According to Figure 4 The value in example (b) increases linearly with a prescribed slope from time t2 to time t1, and is 0.0 before time t1. Figure 5As shown in (b), if a learning candidate product is photographed at time t3 after time t2 to obtain a learning candidate image, and the latest learning candidate product is photographed at time t4 to obtain a learning candidate image, the anomaly correction value is the smallest (e.g., -0.5) at the latest shooting time t4, and increases to 0 as the shooting time becomes earlier. The anomaly correction value curve W2 becomes the following curve: it increases linearly with a specified slope from time t4 to time t3, and is 0.0 before time t3.

[0050] Furthermore, the anomaly correction value is not limited to such examples. For instance, it can increase non-linearly from the latest shooting period, or it can be non-fixed at 0.0 even when tracing back to the shooting period. The anomaly correction value also does not have to increase in stages; it can be assigned in a step-like manner according to the time series, such as: the anomaly is corrected to be smaller only within a specified time range tracing back from the latest shooting period, and anomalies further back in time are not corrected.

[0051] Storage unit 12 stores first feature values ​​of candidate images for learning that are determined to be normal images (not abnormal images). Each of the multiple first feature values ​​stored in storage unit 12 is associated with a corresponding product G. By automatically storing learning images, storage unit 12 can update the learned model using various methods.

[0052] The timing for updating the learned model in the storage unit 12 can be a pre-emptive timing separate from the metering and packaging of goods using the metering and packaging device 1. Alternatively, it can be done as follows: simultaneously with the metering and packaging of goods using the metering and packaging device 1, a portion of the target goods (i.e., target goods G) in the metering and packaging device 1 is processed as learning candidate goods, and the metering and packaging and the updated learned model are performed in parallel.

[0053] The image processing unit 18 acquires a feature quantity (hereinafter referred to as "second feature quantity") representing the characteristics of the object product G. The image processing unit 18 is connected to the imaging unit 16. The image processing unit 18 may be, for example, a processor computer such as a CPU (Central Processing Unit) installed in the imaging unit 16. The image processing unit 18 outputs the acquired second feature quantity to the control unit 6. The second feature quantity of the object product G may be, for example, a feature of the object product G itself, a feature of the container holding the object product G, etc. The features of the object product G itself may be, for example, the shape, area ratio, and hue of each area distinguished by color. The features of the container may be, for example, the size, shape, and color of the container.

[0054] The packaging section 3 uses a film to package the product G transported by the conveying section 14. The packaging section 3 covers the product G transported from the metering section 13 with a film for packaging. Specifically, the packaging section 3 has a film conveying mechanism 31, a lifting mechanism 32, a folding mechanism 33, and a sealing mechanism 34.

[0055] The film conveying mechanism 31 is a mechanism that pulls the film (not shown) of the product G to be packaged from the film roll and conveys it to the packaging position. The film pulled from the film roll by the film conveying mechanism 31 is tensioned and held at the packaging position.

[0056] The lifting mechanism 32 raises the product G to the packaging position, lifting it relative to the film held and tensioned by the film conveying mechanism 31. The folding mechanism 33 folds the periphery of the film exposed from the product G into the bottom surface of the product G. The sealing mechanism 34 heat-seals the overlapping portion of the film folded in by the folding mechanism 33. The packaged product G is then discharged towards the discharge table ES.

[0057] Price labeling unit 4 issues price tags related to the target product G and affixes them to the packaged target product G. Label printer LP prints product information (product information) onto the labels. Product information includes, for example, the product name, unit price, and additives; this information is read from the product master file 64 (product information storage unit) of control unit 6. Label affixing mechanism LI affixes the labels printed by label printer LP onto the packaged target product G.

[0058] The display operation unit 5 is a mechanism (interface) that displays product information, the shooting results of the shooting unit 16, etc., and accepts user operations on the measuring and packaging device 1. The display operation unit 5 may include a display operation unit 51 and a notification unit 52.

[0059] The notification unit 52 can issue a warning when the feature quantity (hereinafter referred to as "third feature quantity") sent from the learning image storage device 10 is inconsistent with the second feature quantity obtained from the object product G placed in the measurement unit 13.

[0060] The control unit 6 is disposed within the main body 1a and controls the operation of the aforementioned mechanisms. Therefore, the control unit 6 is composed of a computer having a storage medium 61 such as a ROM (Read Only Memory) storing programs and information, a RAM (Random Access Memory) temporarily storing data, and an HDD (Hard Disk Drive), a CPU, and communication circuitry. The control unit 6 may include one or more circuits configured to implement the functions of the calling unit 62 and the control unit 63. The control unit 6 has a storage medium 61, a calling unit 62, a control unit 63, and a product master file 64. The product master file 64 is a storage section that stores product information related to the type of product. The product master file 64 stores data such as the unit price and name of various products, and product information related to the size, shape, material, and tare weight of various pallets.

[0061] The metering and packaging device 1 is configured to retrieve the target product G via the product retrieval device 100. The product retrieval device 100 includes the aforementioned learning image storage device 10, the aforementioned imaging unit 16, the product master file 64, and the retrieval unit 62 for retrieving product information.

[0062] Storage medium 61 stores packaging parameters for the packaged object commodity G. Storage medium 61 temporarily stores a third feature value sent from the learning image storage device 10.

[0063] The calling unit 62 determines the type of product G based on the feature quantity (hereinafter referred to as "fourth feature quantity") of the captured image of the product G (hereinafter referred to as "calling product") used to call product information, and calls the product information corresponding to the determined type of product G. The calling product may be a product dedicated to calling, or it may be used for a part of the target products (e.g., the initial target product in a series of target products).

[0064] The fourth feature is a feature representing the features of the subject contained in the captured image of the product to be used. The fourth feature may also be obtained by the storage unit 12 using the captured image of the product to be used captured by the imaging unit 16. The fourth feature may also be a vector including a predetermined number of numbers obtained by the neural network corresponding to the feature extraction layer of the learned model.

[0065] The calling unit 62, for example, applies the acquired fourth feature quantity to the learned model that has been trained using the learning image, uses a product discrimination algorithm as a separation layer, and outputs the type of product to be called, thereby determining the type of product to be called. The calling unit 62, for example, reads the product information of the target product from the product master file 64 as the target product of the current time, and displays it on the display operation unit 51.

[0066] The control unit 63 is the main part of the control unit 6, controlling the metering unit 13, the conveying unit 14, and the packaging unit 3, etc. The control unit 63 sets the product corresponding to the type of product to be retrieved determined by the retrieval unit 62 as the target product at the current time, and retrieves various data related to the set target product from the storage medium 61 and the product master file 64, and outputs it to the display operation unit 5, the label printer LP of the pricing unit 4, and the packaging unit 3. Based on the measurement value obtained from the metering unit 13, the control unit 63 calculates the net weight and price of the target product G, and outputs it as printing information to the label printer LP.

[0067] Based on the third and second characteristic quantities, the control unit 63 determines whether the loaded object product G is consistent with the object product at the current time of being invoked. The control unit 63 may also determine whether to transport the object product G to the packaging unit 3 via the transport unit 14 based on the comparison result of the third and second characteristic quantities.

[0068] Next, refer to Figure 6 The processing operation of the learning image storage device 10 will be explained. First, multiple learning candidate items (articles) are photographed (step S11). For example, the user manually loads multiple items G as learning candidate items into the measuring unit 13 in sequence. Multiple items G can also be automatically supplied to the measuring unit 13 in sequence. The control unit 63 causes the imaging unit 16 to start photographing each item G in sequence. The imaging unit 16 outputs the photographed images of the multiple items G in sequence to the determination unit 11. At this time, the user can specify the call number of the item corresponding to the learning candidate item.

[0069] Next, the first feature values ​​of multiple candidate images for learning are obtained (step S12). The determination unit 11 obtains feature values ​​representing the features of the subjects contained in the candidate images for learning. Next, based on the completed learning model associated with the item and the first feature values, the anomaly degree of multiple candidate items for learning is calculated (step S13). For example, the determination unit 11 applies the obtained first feature values ​​to the completed learning model associated with the item G corresponding to the call number, and uses an anomaly detection algorithm as a separation layer to calculate the anomaly degree of each candidate item for learning.

[0070] Next, an anomaly correction value is calculated based on the shooting dates of the multiple candidate images, and the anomaly is corrected (step S14). The later the shooting date of the candidate image, the smaller the anomaly correction value is calculated by the determination unit 11. The determination unit 11 uses the calculated anomaly correction value to calculate the corrected anomaly based on the shooting date. Next, an anomaly determination threshold is set (step S15). For example, the determination unit 11 sets the anomaly determination threshold to 0.

[0071] Next, it is determined whether the corrected anomaly level is above the specified anomaly determination threshold (step S16). For example, the determination unit 11 determines whether the learning candidate image is an anomalous image by comparing the corrected anomaly level with the anomaly determination threshold.

[0072] For example, if the anomaly level after correction is above the anomaly determination threshold (step S16: Yes), the determination unit 11 determines that the learning candidate image is an anomalous image (step S17). Then, the storage unit 12 does not store the learning candidate image as a learning image (step S18). After that, the learning image storage device 10 ends. Figure 6 The processing steps are as follows. On the other hand, if the anomaly degree after correction is less than the anomaly determination threshold (step S16: No), the determination unit 11 determines that the learning candidate image is not an abnormal image (it is normal) (step S19). Then, the storage unit 12 stores the learning candidate image as a learning image (step S20). After that, the learning image storage device 10 ends. Figure 6 The processing actions.

[0073] Next, refer to Figure 7 The processing operation of the product retrieval device 100 will be explained. First, the product to be retrieved is placed on the measuring unit, and the product to be retrieved is photographed (step S31). For example, the user manually places the product to be retrieved on the measuring unit 13. Next, the fourth feature value of the photographed image of the product to be retrieved is applied to the learned model that has been learned using the learning image, thereby determining the type of product to be retrieved (step S32). The image processing unit 18 obtains a fourth feature value representing the features of the subject contained in the photographed image of the product to be retrieved. The retrieval unit 62, for example, applies the obtained fourth feature value to the learned model that has been learned using the learning image, uses a product discrimination algorithm as a separation layer, and outputs the type of product to be retrieved, thereby determining the type of product to be retrieved.

[0074] Next, the third feature value corresponding to the determined type of product to be used and the product information are retrieved and set (displayed) (step S33). The third feature value corresponding to the type of product to be used as determined by the retrieval unit 62 is read from the learning image storage device 10.

[0075] Next, the target product is placed in the measuring unit and photographed (step S34). For example, the user manually places multiple target products in the measuring unit 13 in sequence. The control unit 63 inputs the measurement value of the target product output from the measuring unit 13 (step S35). When it is determined based on the measurement value that the target product has been placed in the measuring unit 13 or the measurement value of the target product has stabilized, the control unit 63 causes the photographing unit 16 to start photographing the target product.

[0076] Next, the image processing unit acquires the second feature value of the captured image of the target product (step S36). The capturing unit 16 outputs the captured image of the target product to the image processing unit 18. The image processing unit 18 acquires the second feature value of the target product from the captured image acquired by the capturing unit 16.

[0077] Next, the control unit 63 compares the third characteristic quantity with the second characteristic quantity (step S37). If the control unit 63 determines that the third characteristic quantity and the second characteristic quantity are inconsistent (step S38: no), the display operation unit 5 displays a warning (step S39). After that, the operation of the metering and packaging device 1 ends.

[0078] If the third characteristic quantity is determined to be consistent with the second characteristic quantity (step S38: Yes), the control unit 63 determines whether the measurement value is stable. If the measurement value of the target product is unstable (step S40: No), the measurement value is input again (step S41).

[0079] When the measurement value is stable (step S40: Yes), the control unit 63 outputs the net weight and price, etc., calculated based on the stable measurement value, to the label printer LP (step S42). The label printer LP generates printing information based on the product information and price data input in step S33, prints the printing information on the label, and issues it.

[0080] The control unit 63 drives the conveyor unit 14 to transport the goods placed in the metering unit 13 to the packaging unit 3 and package the goods (step S43). The label printer LP issues printed labels, and when the packaged goods are discharged to the discharge table ES, the label affixing mechanism LI affixes the issued labels to the packaged goods (step S44).

[0081] As explained above, according to the learning image storage device 10, the determination unit 11 determines whether a learning candidate image is an abnormal image based on the completed learning model related to the product G and a first feature value of the learning candidate image. The storage unit 12 stores the learning candidate images that are determined not to be abnormal images from among the multiple learning candidate images as learning images. Thus, for example, whenever a learning candidate product is photographed by the shooting unit 16, the learning candidate images that are determined not to be abnormal images are automatically stored as learning images in sequence. Therefore, according to the learning image storage device 10, it is possible to store the learning candidate images after automatically removing abnormal images containing noise from multiple learning candidate images as learning images.

[0082] If the anomaly score calculated by the determination unit 11 based on the first feature value of the learned model and the learned candidate image is above a predetermined anomaly determination threshold, then the determination unit 11 determines that the learned candidate image is an anomalous image. The later the time period of the learned candidate image, the smaller the anomaly score calculated by the determination unit 11. Therefore, the later the time period of the learned candidate image, the more difficult it is to determine that the learned candidate image is an anomalous image. Thus, for example, in cases where only a portion of the learned candidate products has been changed to the latest specifications, it is possible to suppress the situation where a learned candidate image that does not actually contain noise is mistakenly determined to be an anomalous image.

[0083] The product retrieval device 100 includes: the aforementioned learning image storage device 10; an imaging unit 16 that captures images of product G; a product master file 64 (product information storage unit) that stores product information related to the type of product G; and a retrieval unit 62 that retrieves product information. The retrieval unit 62 applies a fourth feature value of the captured image of the product to a learned model that has been trained using the learning images, thereby determining the type of product to be retrieved. The retrieval unit 62 retrieves product information corresponding to the determined type of product to be retrieved. Thus, for example, whenever the imaging unit 16 captures an image of a product to be retrieved, learning candidate images that are determined not to be abnormal images are automatically stored sequentially as learning images, and the type of product to be retrieved can be determined using the learned model trained using the learning images.

[0084] The above describes one embodiment of one aspect of the present invention, but one aspect of the present invention is not limited to the above embodiment.

[0085] For example, in the above embodiment, the determination unit 11 determines that the learning candidate image is an anomalous image if the anomalousness calculated based on the first feature value of the learned model and the learning candidate image is above a predetermined anomalousness determination threshold, but this is not limited to this example. For example, the determination unit may also determine that the learning candidate image is an anomalous image if the normality calculated based on the first feature value of the learned model and the learning candidate image is below a predetermined normality determination threshold. In this case, the later the shooting period of the learning candidate image, the higher the normality is calculated.

[0086] In the above embodiment, the determination unit 11 uses a preset parameter as the anomaly determination threshold, but is not limited to this example. For example, the anomaly determination threshold may be an anomaly degree value in which a predetermined proportion of the learning candidate images are determined to be abnormal. Alternatively, the determination unit 11 may set the anomaly determination threshold based on the calculated anomaly degree and the number of learning candidate images captured, such that the anomaly degree of a predetermined proportion of the learning candidate images captured is above the anomaly determination threshold. In this case, for example, if the anomaly occurrence rate of noise in the multiple learning candidate images is known in advance, the determination unit 11 can automatically set the anomaly determination threshold.

[0087] In the above embodiments, the learning image storage device 10 is configured as part of the product recall device 100 in the metering and packaging device 1, but is not limited to this example. For example, the learning image storage device may also be configured as part of the product recall device in a POS, a meter, a label issuing device, or other devices. Alternatively, the learning image storage device may be used independently to store images taken from multiple captured images after automatically removing abnormal images containing noise as learning images. Furthermore, at least some of the constituent elements of the embodiments described above can be arbitrarily combined with each other. For example, features described in one embodiment may be combined with features described in other embodiments.

Claims

1. A learning image storage device, which stores learning images for learning images related to objects, characterized in that, This learning image storage device has the following features: The photography department, which photographs the items described; The determination unit determines whether the captured image of the item is an abnormal image; as well as The storage unit stores at least a portion of the captured images as the learning images. The determination unit determines whether the captured image is an abnormal image based on the learned model associated with the item and the feature values ​​of the captured image. The storage unit stores the captured images that are determined not to be abnormal images from among the multiple captured images as the learning images.

2. The image storage device for learning according to claim 1, characterized in that, If the anomaly degree calculated by the determination unit based on the learned model and the feature values ​​of the captured image is above a predetermined anomaly determination threshold, then the determination unit determines that the captured image is an anomalous image. The later the time period of the captured image, the smaller the anomaly degree will be calculated.

3. The image storage device for learning according to claim 1 or 2, characterized in that, The determination unit determines whether the captured image is an anomalous image by comparing the anomaly degree calculated based on the features of the learned model and the captured image with a predetermined anomaly determination threshold. The anomaly determination threshold is the anomaly degree value of a specified proportion of the captured images that is determined to be abnormal.

4. A product dispensing device, characterized in that, The product dispensing device has the following features: The image storage device for learning as described in claim 1 or 2; The photography department, which photographs the products; The product information storage unit stores product information related to the type of the product; as well as The calling unit retrieves the product information. The calling unit applies the feature values ​​of the captured image of the product to the learned model that has been trained using the learning image, thereby determining the type of the product and calling the product information corresponding to the determined type of the product.

Images