Commodity image storage device and program

Abstract

To provide an imaging apparatus and a program that, in a case where a package of a commodity has been changed, enable a commodity image of the changed commodity to be easily distinguished from a commodity image before being changed and to be grouped.SOLUTION: A commodity image storage device includes: an interface; a storage unit that stores group information in which, based on commodity images of commodities imaged by an imaging unit and input from the interface and commodity specifying information that specifies the commodities in the commodity images, commodity images related to an identical type of commodity specifying information are classified into one or more groups, in association with the commodity specifying information; and a processor. The processor calculates, for each of the groups, a similarity between a commodity image input together with the commodity specifying information and a commodity image stored in the storage unit for each of the groups and associated with the commodity specifying information based on the commodity specifying information input from the interface, and stores the input commodity image in a group determined in accordance with the calculated similarity.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] An embodiment of the present invention relates to a product image storage device and a program. [Background technology]

[0002] In stores such as shopping centers, mass retailers, and department stores, there are POS systems that perform product registration and payment processes for products sold in the stores. Recently, some of these POS systems recognize the package of a product by capturing an image of the product using an imaging device.

[0003] However, even for the same product, the design of the product's package may change (hereinafter referred to as package change) depending on, for example, the season, etc. In such a case, in package recognition, in order to recognize that a product with a changed package is the same product as the product before the change, it was necessary to train a machine learning model in advance based on images of the product with the changed package. Summary of the Invention [Problem to be solved by the invention]

[0004] The problem that the present invention aims to solve is to provide an imaging device and a program that, when the packaging of a product is changed, can group the product images of the changed product into a different group from the product images before the change, even if the product is the same. [Means for solving the problem]

[0005] The product image storage device of the embodiment has an interface, a storage unit that stores group information, in which product images related to the same product identification information are classified into one or more groups based on a product image of a product captured by an imaging unit and product identification information identifying the product of the product image, input from the interface, in association with the product identification information, and a processor, and the processor calculates, based on the product identification information input from the interface, a similarity for each group between the product image input together with the product identification information and a product image associated with the product identification information stored by group in the storage unit, and stores the input product image in a group determined according to the calculated similarity. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram showing a system according to an embodiment. [Diagram 2] FIG. 2 is a perspective view showing the appearance of the POS terminal and the imaging device. [Diagram 3] FIG. 3 is a block diagram showing a hardware configuration of the imaging apparatus. [Figure 4] FIG. 4 is a block diagram showing a hardware configuration of the product image storage device. [Diagram 5] FIG. 5 is a diagram showing an example of extracting a scanned image. [Figure 6] FIG. 6 is an explanatory diagram showing a process of grouping product images based on captured images. [Figure 7] FIG. 7 is a memory map showing the configuration of the group information section. [Figure 8] FIG. 8 is a flowchart showing the flow of control processing of the imaging device. [Figure 9] FIG. 9 is a functional block diagram showing the functional configuration of the product image storage device. [Figure 10] FIG. 10 is a flowchart showing the flow of a control process related to grouping of commodity images by the commodity image storage device. [Figure 11]FIG. 11 is a flowchart showing the flow of a control process related to training of a machine learning model by the product image storage device. [Figure 12] FIG. 12 is a flowchart showing the flow of a control process related to training of a machine learning model using an image selection method by a product image storage device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following embodiment, the product image storage device will be described as a server installed in a store and also serving as a store server for managing sales information of the store, but the product image storage device may be a headquarters server installed in the headquarters of a company that operates multiple stores, or a cloud server installed on the cloud. In addition, the present invention is not limited to the embodiments described below.

[0008] FIG. 1 is a diagram showing a system according to an embodiment. In FIG. 1, a system 10 is installed in a store T such as a supermarket, a mass retailer, a convenience store, or a specialty store. The system 10 is installed in the store T and includes one or more POS (Point of Sales) terminals 1 (product sales data processing devices), an imaging device 3, an edge device 5, and a product image storage device 7, which are connected to each POS terminal 1, for example. Each POS terminal 1, the edge device 5, and the product image storage device 7 are connected to a communication line L (for example, a LAN (Local Area Network)) that allows mutual communication. Furthermore, each imaging device 3 is connected to each POS terminal 1, and is connected to the POS terminal 1, for example, by a USB (Universal Serial Bus) cable U. Furthermore, each imaging device 3 is connected to the edge device 5 and the product image storage device 7 by a communication line L.

[0009] In the embodiment, the edge device 5 is described as a device separate from the product image storage device 7, but the product image storage device 7 may also have the function of the edge device 5. In this case, the edge device 5 is deleted from the configuration of the system 10.

[0010] The system 10 performs product registration processing, payment processing, and sales management for products sold in the store T. The system 10 executes the product registration processing using package recognition technology.

[0011] Package recognition is a technology for recognizing products (identifying one product or extracting multiple candidate products) by combining a technology for identifying a product by capturing and analyzing symbols such as barcodes or two-dimensional codes attached to the product (hereinafter referred to as "code recognition") with a known general object recognition technology for recognizing a product based on feature quantities indicating the product's external characteristics such as the product's texture, shape, and package (design, etc.) based on a captured image of the product. Package recognition is a process performed by the edge device 5 and / or the product image storage device 7.

[0012] General object recognition is described, for example, in "Yanai Keiji, 'The Current State and Future of General Object Recognition,' Transactions of the Information Processing Society of Japan, Vol. 48, No. SIG16 [searched August 10, 2010], Internet <URL: http: / / mm.cs.uec.ac.jp / IPSJ-TCVIM-Yanai.pdf>".

[0013] In code recognition, when an image of a symbol attached to a product is captured, barcode information is obtained by decoding and analyzing the symbol. The barcode information includes product identification information that identifies the product to which the symbol is affixed. The product identification information is, for example, a product code that indicates the product, but it does not have to be a product code as long as it is information that identifies the product. Code recognition is a process performed by the imaging device 3.

[0014] The imaging device 3 executes code recognition based on images captured by the first imaging unit 37 and the second imaging unit 38 (see Figs. 2 and 3) and generates barcode information. The imaging device 3 then outputs image information relating to the images captured by the first imaging unit 37 and the second imaging unit 38 and the generated barcode information to the product image storage device 7.

[0015] In the embodiment, the edge device 5 is stored inside the main body 42 (see FIG. 2). The edge device 5 executes generic object recognition based on the image information input from the imaging device 3. Then, the edge device 5 outputs, to the imaging device 3, product identification information of the identified product or product identification information of multiple candidate products.

[0016] The imaging device 3 transmits to the POS terminal 1 either the product identification information included in the generated barcode information or the product identification information input from the edge device 5. Specifically, when the imaging device 3 is able to generate barcode information based on code recognition, it transmits to the POS terminal 1 either the product identification information included in the barcode information or the product identification information of the identified product input from the edge device 5. When the imaging device 3 is unable to generate barcode information, it transmits to the POS terminal 1 the product identification information of the product input from the edge device 5.

[0017] The product image storage device 7 groups and stores the product images by product identification information (i.e., by product) based on the product images and product identification information in which the product is captured among the captured images input from the imaging device 3. Specifically, the product image storage device 7 creates groups by product images with the same product identification information and similar feature amounts, and groups the product images. However, even for the same product, the package may be changed depending on the season, etc. Such products are assigned the same product identification information, but the feature amount extracted from the product image differs depending on the package. In such a case, even if the product identification information is the same, the product image storage device 7 groups the product with the package before the change and the product with the package after the change into different groups by the feature amount. In this way, product images with the same package are grouped in the same group, and product images with different packages are grouped in different groups. Note that even if the package is the same, the feature amount may change (become lower) depending on the conditions under which the product is captured (such as the brightness of the lighting or the image around the product). Therefore, products with the same package may be grouped into different groups.

[0018] Furthermore, the product image storage device 7 of the embodiment stores time information in association with each group. In theory, this time information is the time when the product image was most recently stored in the group, but in reality, it may be either the time when the product image was captured by the imaging device 3 (the time when the product registration operation was performed by the customer) or the time when the barcode information was generated based on the captured symbol. Each time the product images captured by the imaging device 3 are grouped, new time information is stored in the group in which the product image is stored (the time information is updated).

[0019] This time information may be information indicating a specific time, but does not have to be precise time information, and may be information indicating a period, such as information indicating a specific date.

[0020] Furthermore, based on the time information, it is possible to check whether a product image has been stored in a group within a predetermined period going back from a predetermined time (for example, the current time (which may be the current date)), so the time information is an example of performance information indicating that the product image has been stored within a predetermined period going back from the present. Since the performance information is information that allows checking whether a product image has been stored within a predetermined period going back from the present, it may be information (for example, flag information) indicating that a product image has been stored within a predetermined period (for example, the period from the previous training (training will be described later) to the present time), and the flag information or the like may be associated with a group in which a product image has been stored within a predetermined period. In this way, information indicating that a product image has been stored within a predetermined period is also an example of performance information indicating that the product image has been stored within a predetermined period going back from the present. Hereinafter, in the embodiment, the time information will be described as an example of performance information.

[0021] When a group of similar features exists for a product image related to certain product feature information input from the imaging device 3 (for example, when the product is the same product with the same (unchanged) packaging), the product image storage device 7 adds and stores the input product image to that group and updates the time information for that group.

[0022] Furthermore, when there is no group of similar features for a product image related to certain product feature information input from the imaging device 3 (for example, in the case of the same product but with a different package), the product image memory device 7 creates a group with a new group number related to the product identification information, stores the input product image in the new group, and also stores the time information in the new group.

[0023] Furthermore, when certain product feature information input from the imaging device 3 is not stored (for example, in the case of a new product), the product image memory device 7 creates a new group with group number "1" related to the new product identification information, stores the input product image in the new group, and also stores the time information of the group.

[0024] In addition, the product image storage device 7 of the embodiment trains the machine learning model 745 (see FIG. 4). In the embodiment, the machine learning model 745 is stored in the product image storage device 7. However, the machine learning model 745 may be stored in a device other than the product image storage device 7.

[0025] The product image storage device 7 stores a machine learning model 745 (see FIG. 4). The machine learning model 745 is a model used, for example, when identifying one product based on a product image captured by the imaging device 3 or extracting multiple candidate products.

[0026] As described above, the product image storage device 7 trains the machine learning model 745 to enable product recognition in response to the release of a new product or changes to the packaging of an existing product. When training for a product, the product image storage device 7 uses a stored group (associated with product identification information) corresponding to the product. Specifically, the product image storage device 7 selects a group that stores time information included in a predetermined period going back from the present, from among the time information associated with each group. Then, the product image storage device 7 executes training for the product based on the product images included in the selected group.

[0027] The product image storage device 7 estimates the relationship between the product and the camera when each product image included in the selected group was captured, for example, using a technique such as Structure from Motion. When estimating, a region such as a person's hand (hand holding a product) that appears in the image is identified and removed using a technique such as semantic segmentation. Then, the product image storage device 7 selects product images so that the estimated positions and orientations are uniform. Training is performed using the selected product images. If there are other groups selected for the product, the product image storage device 7 performs the same process on the other groups and trains the product. In this way, the product image storage device 7 trains the machine learning model 745. Then, the product image storage device 7 outputs the machine learning model 745 after training to the edge device 5. The edge device 5 performs general object recognition on the product captured by the imaging device 3 using the machine learning model 745 input from the product image storage device 7.

[0028] During training, a 3D image (3D model) may be generated based on the product images included in the selected group using a technique such as photogrammetry. Training can be performed on the product from multiple angles by changing the orientation of the generated 3D image to various angles.

[0029] Next, the POS terminal 1 and the imaging device 3 will be described. Fig. 2 is a perspective view showing the appearance of a self-service POS terminal 1 that is operated by a customer, and the imaging device 3. As shown in Fig. 2, the POS terminal 1 has a main body 42 and a basket placement stand 4. The main body 42 incorporates a circuit board and a power source for controlling the POS terminal 1. The basket placement stand 4 is attached to one side of the main body 42, and is a stand on which a basket containing products that a customer will purchase is placed.

[0030] The top surface of the main body 42 has a flat product placement section 41. The main body 42 has a pair of support posts 45 extending upward at the rear of the product placement section 41. A printer 22 that issues receipts is attached to the support posts 45 on the side farther from the basket placement platform 4. A card reader 21 that reads information from cards such as credit cards and point cards is attached to the top of the printer 22.

[0031] Additionally, an imaging device 3 is attached to the support post 45 on the side closer to the basket placement platform 4. The imaging device 3 includes a first imaging section 37 (imaging section) and a second imaging section 38 (imaging section), and is a device that captures an image of a commodity scanned in front of the imaging device 3. The first imaging section 37 and the second imaging section 38 are, for example, cameras in which a large number of CCDs (Charge Coupled Devices: imaging elements) are arranged in a two-dimensional form (planar shape).

[0032] The imaging device 3 is installed with its reading surface facing the customer, and a first imaging section 37 provided inside the imaging device 3 captures an image of a product by scanning the product that the customer has taken out of the basket. Furthermore, since customers tend to aim the symbol S (see FIG. 6) attached to the product at the imaging device 3 to scan the product, the first imaging section 37 often simultaneously captures an image of the symbol S attached to the product. In the following embodiment, it is assumed that the first imaging section 37 captures an image of the symbol S.

[0033] Further, the second imaging section 38, which is a part of the imaging device 3, is attached above the imaging device 3. The second imaging section 38 is installed such that its reading surface faces diagonally downward from the imaging device 3, and images the scanned commodity from a different angle than the first imaging section 37. In other words, the second imaging section 38 images a surface of the commodity different from the surface imaged by the first imaging section 37.

[0034] It is desirable to image the product from multiple angles in the imaging device 3. In order to have the symbol S imaged toward the first imaging unit 37, the customer searches for the position of the symbol S in front of the first imaging unit 37 (i.e., changes the orientation of the product in various ways), so it is possible to image the product from multiple angles in the first imaging unit 37 and the second imaging unit 38. However, in order to image the product from more multiple angles (for example, uniformly over the entire surface of the product), this can be achieved by using an imaging device that images the product from all sides, such as that described in JP 2020-047051 A.

[0035] A display 18 is provided between the pair of support columns 45. The display 18 is a display made of, for example, liquid crystal, and is installed facing a customer who operates the POS terminal 1. The display 18 is located slightly above the imaging device 3 and the printer 22, and at the same height as the card reader 21 and the second imaging unit 38.

[0036] The display 18 displays product information (product name, product price, etc.) of the registered product and payment information (total amount, payment amount, change amount, etc.) of the payment processed. An operation unit 17 consisting of a touch panel is provided on the display surface of the display 18.

[0037] In addition, between the pair of support columns 45 and below the display 18, there are provided a temporary placement table 44 for temporarily placing products, and a pair of attachment parts 43 for attaching handles of a shopping bag for containing products that have undergone product registration processing. The products placed in the shopping bag are placed on the product placement part 41.

[0038] Furthermore, a substantially cylindrical pole 6 extending upward is attached to the support 45 on the side closer to the car placement platform 4. A patrol lamp 61 is attached to the pole 6.

[0039] The patrol lamp 61 lights up or flashes when an abnormality occurs in the POS terminal 1 (for example, when the receipt paper runs out) or when a customer calls an attendant.

[0040] One or more POS terminals 1 (three in the first embodiment) are installed in the checkout area in store T. The POS terminal 1 is a product sales data processing device that allows customers to perform product registration and accounting operations themselves. When a customer performs a product registration operation (for example, an operation of scanning a symbol S such as a barcode attached to a product and the appearance of the product and having the imaging device 3 capture an image of the product), the POS terminal 1 executes product registration processing related to the product to be sold based on the product identification information received from the imaging device 3. The POS terminal 1 also executes payment processing based on the customer's accounting operation (an operation of ending the product registration operation and starting the accounting process).

[0041] The product registration process refers to a process of acquiring a product code (product identification information) that identifies a product sold at store T, displaying product information (product name, product price, etc.) such as the product name and price of the product on display 18 based on the acquired product code, and storing the product information in the product information section. The settlement process refers to a closing process related to the transaction based on the product information stored in the product information section in conjunction with the product registration process, specifically, displaying the total amount on display 18, settlement processing by media such as cash, credit card, electronic money, etc., calculating the change based on the deposit in the case of cash settlement and displaying it on display 18, issuing a receipt from printer 22 on which product information and accounting information (total amount, deposit amount, change amount, point information, etc.) of the settled product are printed.

[0042] The POS terminal 1 also transmits product information and transaction information (collectively referred to as "sales data") of the processed products to the product image storage device 7, which also functions as a store server. In this embodiment, the sales data is transmitted from the POS terminal 1 to the product image storage device 7 when the transaction process is completed, but it may be transmitted at any timing, such as transmitting all data for one day at the closing time of the store T.

[0043] In such a system 10, a customer performs a product registration operation by removing a product from a basket placed on the basket placement table 4 and capturing an image of the symbol S affixed to the product by passing (scanning) in front of the imaging device 3 (first imaging unit 37 and second imaging unit 38). The customer then places the product for which the product registration operation has been performed in the shopping bag attached to the mounting unit 43. The POS terminal 1 performs product registration processing for the product for which the product registration operation has been performed based on the product identification information received from the imaging device 3. The customer who has performed the product registration operation for all products then performs a checkout operation by operating the checkout button. When the checkout operation has been performed, the POS terminal 1 performs a payment process for the products for which the product registration operation has been performed.

[0044] The product image storage device 7 is installed, for example, in the back yard of the store T. The product image storage device 7 collects sales data received from the POS terminal 1. The product image storage device 7 manages the sales of products in the store T based on the collected sales data.

[0045] Next, the hardware configuration of the imaging device 3 will be described. FIG. 3 is a block diagram showing the hardware configuration of the imaging device 3. As shown in FIG. 3, the imaging device 3 includes a CPU 31, which is an example of a processor, a ROM 32, a RAM 33, a memory unit 34, and the like. The CPU 31 is the main controller of the imaging device 3. The ROM 32 stores various programs. The RAM 33 expands programs and various data. The memory unit 34 stores various programs. The CPU 31, the ROM 32, the RAM 33, and the memory unit 34 are connected to one another via a bus 35. The CPU 31 executes a control process of the imaging device 3, which will be described later, by operating according to a control program stored in the ROM 32 or the memory unit 34 and expanded in the RAM 33. The processor may be a processor other than the CPU 31.

[0046] The RAM 33 has an image storage unit 331, a product identification information unit 332, and a barcode information unit 333. The image storage unit 331 stores an image captured by the first imaging unit 37 and an image captured by the second imaging unit 38. The product identification information unit 332 stores product identification information input from the edge device 5. The barcode information unit 333 stores barcode information obtained by decoded (code-recognized) the symbol S included in the image captured by the first imaging unit 37.

[0047] The memory unit 34 is configured with a non-volatile memory such as an HDD or a flash memory in which stored information is retained even when the power is turned off, and has a control program unit 341. The control program unit 341 stores a control program for driving the imaging device 3.

[0048] In addition, the CPU 31 is connected to a first imaging section 37 and a second imaging section 38 via a bus 35 and a controller 36 .

[0049] The CPU 31 is also connected to a communication unit 39 via a bus 35. The communication unit 39 is communicatively connected to the edge device 5 and the product image storage device 7 via a communication line L. The communication unit 39 is also communicatively connected to the POS terminal 1 via a USB cable U.

[0050] Next, the hardware configuration of the commodity image storage device 7 will be described. FIG. 4 is a block diagram showing the hardware configuration of the commodity image storage device 7. As shown in FIG. 4, the commodity image storage device 7 includes a CPU 71, which is an example of a processor, a ROM 72, a RAM 73, a memory unit 74, and the like. The CPU 71 is the main controller of the commodity image storage device 7. The ROM 72 stores various programs. The RAM 73 expands programs and various data. The memory unit 74 stores various programs. The CPU 71, the ROM 72, the RAM 73, and the memory unit 74 are connected to each other via a bus 75. The CPU 71 executes the control process of the commodity image storage device 7, which will be described later, by operating according to the control program stored in the ROM 72 or the memory unit 74 and expanded in the RAM 73. The processor may be other than the CPU 71.

[0051] The RAM 73 has a sales information section 731 and a buffer 732. The sales information section 731 stores sales data received from the POS terminal 1 and manages sales of the store T. The buffer 732 stores captured images and barcode information input from the imaging device 3.

[0052] The memory unit 74 is composed of a non-volatile memory (HDD in the embodiment) such as an HDD (hard disk drive) or a flash memory in which stored information is retained even when the power is turned off, and has a control program unit 741, an object detection model 742, a feature extraction model 743, a group information unit 744 (storage unit), and a machine learning model 745. The control program unit 741 stores a control program for driving the product image storage device 7.

[0053] The object detection model 742 is a dictionary for identifying an area (range) of an object (product) captured in a captured image. In other words, the object detection model 742 is a dictionary for identifying an area in which an object (product) is captured from a captured image based on the input captured image. In other words, the object detection model 742 refers to a series of product images in which an object has been detected from the captured images input from the imaging device 3 as a scanned image K (described later in FIG. 5). The object detection model 742 is a dictionary for extracting an image of a product (hereinafter referred to as an "object image") surrounded by a identified area in the scanned image K.

[0054] The feature extraction model 743 is a dictionary for calculating feature amounts effective for identifying a product shown in a specified area (object image). That is, the feature extraction model 743 outputs a feature vector indicating feature amounts indicating the features of a product whose area is specified, based on the input object image.

[0055] The group information section 744 stores, for each item of product specifying information, product images of the items specified by the item specifying information, in one or more groups. The group information section 744 will be described later with reference to FIG.

[0056] In addition, the CPU 71 is connected to a display unit 77 and an operation unit 78 via a bus 75 and a controller 76. The display unit 77 is, for example, a liquid crystal display, and displays information to an operator who operates the product image storage device 7. The operation unit 78 is, for example, a keyboard, and is operated by the operator of the product image storage device 7.

[0057] The CPU 71 is also connected to a communication unit 79 (interface (I / F)) via a bus 75. The communication unit 79 is communicatively connected to the POS terminal 1, the imaging device 3, and the edge device 5 via a communication line L. The communication unit 79 may be composed of three physical interfaces, an image I / F that inputs the captured image output from the imaging device 3, a barcode information I / F that inputs the barcode information output from the imaging device 3, and a time information I / F that inputs the time information output from the imaging device 3, or may be a single interface that inputs the captured image, barcode information, and time information output from the imaging device 3.

[0058] Next, the scanned image will be described. The captured image of the product scanned (scanning operation) by the customer is sent from the imaging device 3 to the product image storage device 7. The images sent from the imaging device 3 include a mixture of images showing the product (product images) and images not showing the product. The product image storage device 7 extracts a scanned image showing the object by detecting the object from the input captured image. The object may be detected using the object detection model 742, or may be detected simply using an infrared proximity sensor or may utilize a technology such as SSD (Single Shot Multibox Detector).

[0059] Fig. 5 is an explanatory diagram of extraction of scan image K from the imaging information input from the imaging device 3. As shown in Fig. 5, the CPU 71 of the product image storage device 7 extracts scan image K (Frame 0002 to Frame 0008) which is a series of product images in which object B (product) is detected from a series of images (Frame 0001 to Frame 0009) obtained when a customer scans one product. A symbol S (barcode) is shown in the scan image K, and the imaging device 3 decodes (recognizes the code) this symbol S to generate barcode information and outputs it to the product image storage device 7. In the following description, images showing products including the scan image K will be collectively referred to as product images.

[0060] Next, a process will be described in which the CPU 71 groups product images (scanned images) based on the captured image and barcode information input from the imaging device 3. Fig. 6 is an explanatory diagram showing the process of grouping scanned images K based on the captured image.

[0061] As shown in Fig. 6, first, the CPU 71 extracts the scan image K shown in Fig. 5 from the captured image input from the imaging device 3. At that time, the CPU 71 uses the object detection model 742 to perform object B detection to identify the area of ​​the product showing the product shown in the extracted scan image K (step A). ​​Next, the CPU 71 uses the feature extraction model 743 to calculate the feature amount of the detected product and generate a feature vector (step B). Meanwhile, the CPU 71 searches the group information section 744 based on the barcode information input at the same time, and calculates the feature amount related to the product image included in each group for all groups associated with the barcode information, and generates a feature vector for each group.

[0062] Next, the CPU 71 calculates the similarity between the feature vector generated based on the input captured image and the feature vector for each group generated by searching the group information section 744 (step C). The similarity is calculated, for example, by using a Convolutional Neural Network (CNN) used for product recognition to generate feature vectors for the input scanned image K and the scanned images K stored by group, and then calculating the cosine similarity between the generated feature vectors. Alternatively, local features such as SIFT for each scanned image K may be calculated, and the identity of the packages may be determined by matching the calculated features to calculate the similarity.

[0063] Next, the CPU 71 groups the scan images K according to the similarity (step D). If there is a group with a high similarity to the generated feature vector, the CPU 71 adds the scan image K to the group and stores it. If there is no group with a high similarity to the generated feature vector, the CPU 71 creates a new group related to the barcode information in the group information section 744, and stores the scan image K in the created group. If the input barcode information is not stored in the group information section 744 as a result of searching the group information section 744, a new group associated with the input barcode information is created in the group information section 744, and stores the scan image K in the created group. In this way, the input scan images K are grouped.

[0064] Next, the group information section 744 will be described. Fig. 7 is a memory map showing the configuration of the group information section 744. Note that, in the example of Fig. 7, group information a of barcode information 1 indicating two types of products and group information b of barcode information 2 are shown, but in reality, there is as much group information related to barcode information as there are products sold at store T.

[0065] 7, group information section 744 stores group information a of a product identified by barcode information 1, and group information b of a product identified by barcode information 2. As described above, group information section 744 also stores group information other than group information a and group information b, but this will not be described here.

[0066] Group information a and group information b each have a barcode information section 7441, a group number section 7442, a time information section 7443, and an image information section 7444. The barcode information section 7441 stores barcode information for identifying a product. In the embodiment, barcode information 1 is stored in group information a, and barcode information 2 is stored in group information b. The group number section 7442 stores a group number indicating a group belonging to the barcode information stored in the barcode information section 7441. In the example of FIG. 7, three groups (group 1, group 2, group 3) related to barcode information 1 belong to group information a. Four groups (group 1, group 2, group 3, group 4) related to barcode information 2 belong to group information b.

[0067] The time information section 7443 stores, for each group, time information (an example of performance information) indicating the time (or date, etc.) when a product image was most recently stored in the group, in correspondence with each group number stored in the group number section 7442.

[0068] The image information section 7444 stores the product images according to their mutual similarity in association with each group number stored in the group number section 7442. Specifically, the similarity between the feature amount (feature vector) of the scan image K extracted based on the image input from the imaging device 3 and the feature amount (feature vector) calculated from the product images stored in each group is calculated, and if there is a group with high similarity (for example, a group with a calculated similarity of 95% or more), the scan image K is additionally stored in the group. If there is no group with high similarity, a new group related to the barcode information is created, a new group number is generated and stored in the group number section 7442, the current time is stored in the time information section 7443 as time information of the created group, and the scan image K is stored in the image information section 7444.

[0069] When flag information is used as the performance information, the flag information is stored in the time information section 7443. In this case, when a scanned image K is stored in the image information section 7444 after the previous training is performed, the flag information is stored in the time information section 7443 in association with the stored group.

[0070] From here, the control of the imaging device 3 will be described. Fig. 8 is a flowchart showing the flow of the control process of the imaging device 3. As shown in Fig. 8, the CPU 31 of the imaging device 3 judges whether an imaging start signal has been received (S11). The imaging start signal is received from the POS terminal 1, for example, when a customer operates a button to start an operation in the POS terminal 1. The CPU 31 waits until the imaging start signal is received (No in S11), and if it is determined that the imaging start signal has been received (Yes in S11), it activates the first imaging unit 37 and the second imaging unit 38 to start imaging (S12).

[0071] Next, the CPU 31 determines whether an imaging end signal has been received (S13). The imaging end signal is received from the POS terminal 1, for example, when a customer has performed a payment process at the POS terminal 1. If it is determined that an imaging end signal has been received (Yes in S13), the CPU 31 ends imaging by the first imaging unit 37 and the second imaging unit 38 (S16). Then, the CPU 31 returns to S11.

[0072] If it is determined that the imaging end signal has not been received (No in S13), the CPU 31 stores the captured images whose imaging was started in S13 in the image storage unit 331 (S14). Then, the CPU 31 outputs the stored captured images to the edge device 5 and the product image storage device 7 (S15). In S14, the CPU 31 stores all images captured from the start to the end of imaging.

[0073] Next, the CPU 31 judges whether the symbol S is detected from the captured image stored in the image storage unit 331 (S21). If it is judged that the symbol S is detected (Yes in S21), the CPU 31 performs code recognition to decode the detected symbol S (S22). Then, the CPU 31 stores the barcode information generated by decoding and the time when the barcode information was generated as time information in the barcode information unit 333 (S23). Then, the CPU 31 outputs the barcode information stored in the barcode information unit 333 (i.e., the barcode information for the captured image stored in S14) and the time information to the commodity image storage device 7 (S24). In this way, when the stored captured image contains the symbol S, the CPU 31 outputs the captured image, the decoded barcode information, and the time information to the commodity image storage device 7 in succession at or close to the same time. Note that the CPU 31 may also transmit the barcode information stored in the barcode information unit 333 to the POS terminal 1. Then, the CPU 31 returns to S13.

[0074] Furthermore, in S21, if it is determined that the symbol S has not been detected (No in S21), the CPU 31 determines whether or not the commodity identification information has been input from the edge device 5 (S25). If it is determined that the commodity identification information has not been input (No in S25), the CPU 31 returns to S13, and if it is determined that the commodity identification information has been input from the edge device 5 (Yes in S25), the CPU 31 stores the commodity identification information in the commodity identification information section 332 (S26). Then, the CPU 31 transmits the commodity identification information stored in the commodity identification information section 332 to the POS terminal 1 (S27). Then, the CPU 31 returns to S13. In this way, the imaging device 3 outputs the captured images captured by the first imaging section 37 and the second imaging section 38, the barcode information obtained by decoding the symbol S, and the time information to the commodity image storage device 7.

[0075] From here, we will explain the functional configuration of the product image storage device 7. Fig. 9 is a functional block diagram showing the functional configuration of the product image storage device 7. The CPU 71 of the product image storage device 7 functions as a calculation unit 711, a determination unit 712, a storage control unit 713, a performance storage unit 714, a selection unit 715, a 3D image generation unit 716, and a training unit 717 by following the control programs stored in the ROM 72 and the control program unit 741 of the memory unit 74.

[0076] Based on the product identification information included in the barcode information input from the communication unit 79 (interface), the calculation unit 711 calculates the similarity for each group between the scan image K input together with the product identification information and the product image associated with the product identification information stored by group in the group information unit 744. Specifically, based on the product identification information included in the barcode information input from the communication unit 79, the calculation unit 711 generates a feature vector representing the feature amount of the scan image K input together with the product identification information and the product image associated with the product identification information stored by group in the group information unit 744, and calculates the similarity related to the generated feature vector for each group.

[0077] The determination unit 712 determines a group in which the input scan image K is to be stored according to the calculated similarity. Specifically, when there is a group with a high similarity, the determination unit 712 determines the group (existing group) as the group in which the input scan image K is to be stored (sets the group number of the group). When there is no group with a high similarity (when the similarity with all groups is low), the determination unit 712 determines a new group related to the product identification information (creates a new group related to the product identification information and sets the group number of the group). When the product identification information input by the calculation unit 711 is not in the group information unit 744, the determination unit 712 determines a new group created based on the input product identification information (creates a new group related to the new product identification information and sets the group number "1" of the group).

[0078] The storage control unit 713 stores the product image input in the group determined according to the calculated similarity. In an embodiment, the storage control unit 713 stores the scan image K input in the group determined by the determination unit 712. Specifically, when the determination unit 712 determines an existing group with a high similarity, the storage control unit 713 additionally stores the scan image in the existing group. Furthermore, when the determination unit 712 determines a new group related to the same barcode information, the storage control unit 713 stores the scan image in the new group. Furthermore, when the determination unit 712 determines a new group related to new barcode information, the storage control unit 713 stores the scan image in the new group.

[0079] The result storage unit 714 stores the result information for the group in which the scan image K is stored. Specifically, the result storage unit 714 stores the time information input from the imaging device 3 for the group in which the scan image K is stored.

[0080] The selection unit 715 selects a group for which performance information is stored. Specifically, the selection unit 715 selects a group for which time information indicating a period within a predetermined period going back from the current time, which is a predetermined time, is stored (i.e., a group of products that have been frequently purchased recently). The predetermined period is, for example, the range of dates and times going back from the current time to the date and time when the previous training was performed. The predetermined time can be set by the store T according to the circumstances of the store T.

[0081] The 3D image generating unit 716 generates, for each group, a 3D image of the product based on the product images (accumulated scan images K) stored in each group, using a technique such as photogrammetry.

[0082] The training unit 717 trains the machine learning model 745 using the 3D image generated by the 3D image generation unit 716. Specifically, the training unit 717 uniformly trains the machine learning model 745 using various faces of the 3D image generated by the 3D image generation unit 716.

[0083] From here, the control of the commodity image storage device 7 will be described. FIG. 10 is a flowchart showing the flow of the control process when the commodity image storage device 7 groups and stores the scan image K input from the imaging device 3. As shown in FIG. 10, the CPU 71 of the commodity image storage device 7 judges whether the captured image output from the imaging device 3 has been input (acquired) (S31). The CPU 71 waits until the captured image is input (No in S31), and if it is judged that the captured image has been input (acquired) (Yes in S31), it performs an object detection process (detection of a commodity) based on the acquired captured image (S32). Then, the CPU 71 judges whether an object (commodity) has been detected from the input captured image (S33). If it is judged that a commodity has been detected (Yes in S33), the CPU 71 extracts the scan image K from the object image and stores the extracted scan image K in the buffer 732 (S34).

[0084] Next, the CPU 71 judges whether the barcode information and time information related to the input captured image have been acquired from the imaging device 3 (S35). If it is judged that the barcode information and time information related to the input captured image have not been acquired (No in S35), the CPU 71 returns to S31, and if it is judged that the barcode information and time information related to the input captured image have been acquired (Yes in S35), the CPU 71 stores the input barcode information and time information in the buffer 732 (S36). The captured image is associated with the buffer information and time information related to the symbol S shown in the captured image by the processes of S34 and S36. Then, the CPU 71 returns to S31.

[0085] Moreover, if it is determined that a product has not been detected (No in S33; for example, a product is not detected when the customer has finished scanning one product), the CPU 71 determines whether barcode information and time information are stored in the buffer 732 (S41). If it is determined that barcode information and time information are not stored in the buffer 732 (No in S41), the CPU 71 clears the buffer 732 (S42). That is, the CPU 71 clears the captured images stored in the buffer 732. The CPU 71 then returns to S31.

[0086] Furthermore, if it is determined that barcode information and time information are stored in buffer 732 (Yes in S41), CPU 71 searches group information section 744 based on the barcode information (S43). Then, CPU 71 determines whether the same barcode information is stored in group information section 744 (S44).

[0087] If it is determined that the same barcode information is stored (Yes in S44), the calculation unit 711 calculates a similarity based on the feature vector calculated based on the scan image K stored in the buffer 732 and the feature vector for each group calculated based on the product image stored in the image information section 7444 of each group associated with the same barcode information in the group information section 744 (S45).

[0088] Then, the CPU 71 judges whether there is a group having a feature vector with high similarity to the feature vector calculated based on the scan image K stored in the buffer 732 based on the similarity calculated by the calculation unit 711 (S46). If it is judged that there is a group having a feature vector with high similarity (Yes in S46), the determination unit 712 determines the group number stored in the group number section 7442 corresponding to the group as the group of the scan image K stored in the buffer 732 (S47). Then, the storage control unit 713 adds the scan image K to the image information section 7444 associated with the determined group number and stores it (S48). Also, the performance storage unit 714 overwrites the time information stored in the buffer 732 to the time information section 7443 associated with the determined group number and stores it (S48). By the process of the performance storage unit 714 in S48, the latest time information when the scan image K was stored in the group is stored, and the time information is updated. Then, the CPU 71 ends the process and returns to S31.

[0089] Also, when it is determined that there is no group having a feature vector with high similarity (No in S46), the determination unit 712 generates a new group number belonging to the barcode information, and determines the group number as the group of the scanned image K stored in the buffer 732 (S51). Then, the storage control unit 713 stores the determined new group number in the group number section 7442 (S52). The storage control unit 713 also stores the scanned image K in the group generated in the image information section 7444 in association with the new group number (S52). Also, the performance storage unit 714 stores the time information stored in the buffer 732 in the time information section 7443 of the group associated with the determined new group number (S52). The process of the performance storage unit 714 in S52 stores the latest time information when the scanned image K was stored in the new group. Then, the CPU 71 ends the process and returns to S31.

[0090] Also, in S44, when it is determined that the same barcode information is not stored (No in S44), the determination unit 712 generates a new group number (group number "1") belonging to the barcode information stored in the buffer 732, and determines the group number as the group of the scan image K stored in the buffer 732 (S53). Then, the storage control unit 713 generates group information (for example, group information c) related to the new barcode information, and stores the new group number related to the group information in the group number section 7442 (S54). The storage control unit 713 also stores the scan image K in the image information section 7444 associated with the new group number (S54). Furthermore, the record storage unit 714 stores the time information stored in the buffer 732 in the time information section 7443 associated with the determined new group number "1" (S54). The process of the record storage unit 714 in S54 stores the latest time information when the scan image K was stored in the group with the group number "1". Then, the CPU 71 ends the process and returns to S31.

[0091] The product image storage device 7 of this embodiment stores product images of products specified by barcode information in one or more groups obtained by grouping images with high similarity. The input scanned image K is added to and stored in a group with high similarity, and if the similarity is not high with any of the groups, a new group is created and the scanned image K is stored therein.

[0092] According to such an embodiment, when the packaging of a product is changed, it is possible to store the product images of the same product that have been changed in a separate group from the product images before the change.

[0093] From here, the control of the product image storage device 7 operated to start training when training the machine learning model 745 will be described. FIG. 11 is a flowchart showing the flow of the control process of training the machine learning model 745 by the product image storage device 7. As shown in FIG. 11, the selection unit 715 selects, in the group information related to each barcode information stored in the group information unit 744, a group whose time information stored in the time information unit 7443 is within a predetermined period going back from the current time, as a group to be used for training the machine learning model 745 (S61). Therefore, a group of new products or products whose packaging has been recently changed within a predetermined period is selected. On the other hand, a group of products whose sales have been discontinued or products with old packages (products with packages that are not currently on sale) is not selected.

[0094] Next, the 3D image generation unit 716 generates a 3D image for the selected group based on the product image stored in the image information unit 7444 (S62). The training unit 717 then uses the generated 3D image to perform training for the product identified by the barcode information (S63). The training in S63 is performed by slightly changing the orientation (angle) of the generated 3D image each time, thereby uniformly training the product on various sides. Similar training is also performed for different groups. Similar training is also performed for products with different barcode information.

[0095] To end the training, for example, an operation to end the training is performed. The CPU 71 judges whether or not the operator has performed an operation to end the training (S64). If it is judged that the operation to end the training has not been performed (No in S64), the process returns to S63 since further training continues, and the training unit 717 changes the orientation of the 3D image and performs further training (S63). The training unit 717 also performs training for different groups. In addition, similar training is performed for products with different barcode information.

[0096] On the other hand, if it is determined that an operation to end training has been performed (Yes in S64), the CPU 71 ends the process.

[0097] From here, the training of the machine learning model 745 using a technique such as Structure from Motion will be described. FIG. 12 is a flowchart showing the flow of control processing related to the training of the machine learning model using a Structure from Motion-based image selection method. As shown in FIG. 12, the selection unit 715 selects, in the group information related to each barcode information stored in the group information unit 744, a group whose time information stored in the time information unit 7443 is within a predetermined period going back from the current time, as a group to be used for training the machine learning model 745 (S71). Therefore, a group of new products or products whose packaging has been recently changed within a predetermined period is selected. On the other hand, a group of products whose sales have been discontinued or products with old packages (products with packages that are not currently on sale) is not selected.

[0098] Next, the 3D image generation unit 716 estimates the position and orientation of the product and the camera for the selected group based on the product images stored in the image information unit 7444 (S72). Then, the 3D image generation unit 716 selects product images so that the estimated positions and orientations are uniform (S73). Then, the training unit 717 uses the selected product images to perform training on the product identified by the barcode information (S74). The training unit 717 performs similar training for different groups. Also, performs similar training for products with different barcode information.

[0099] To end the training, for example, an operation to end the training is performed. The CPU 71 judges whether or not the operator has performed an operation to end the training (S75). If it is judged that the operation to end the training has not been performed (No in S75), the process returns to S64 since further training continues, and the training unit 717 executes further training (S74). The training unit 717 also executes training for different groups. In addition, similar training is executed for products with different barcode information.

[0100] On the other hand, if it is determined that an operation to end training has been performed (Yes in S75), the CPU 71 ends the process.

[0101] In such an embodiment, when training the machine learning model 745, the product image storage device 7 can select a group in which time information is stored within a specified period (e.g., a group in which the packaging has been changed) and can train without selecting a group in which time information is not stored within the specified period (e.g., a group in which old packaging is not included in the specified period), thereby improving the accuracy of the training.

[0102] Although the embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

[0103] For example, in the embodiment, the product image storage device 7 performs grouping (classification) of the product images and training of the machine learning model 745. However, the present invention is not limited to this, and the product image storage device 7 may be a device that performs grouping of the product images. In this case, a training device that trains the machine learning model may be separately provided, and the training device may use information on the groups grouped by the product image storage device 7 to perform training of the machine learning model. In this case, the 3D image generation unit 716 and the training unit 717 in FIG. 9 are functional configurations of the training device, and the control in FIG. 11 is executed by the training device. In this case, the time information may be possessed by the product image storage device 7 or the training device.

[0104] In the embodiment, the machine learning model is trained using the group information section 744 grouped by the product image storage device 7. However, the present invention is not limited to this, and the product image storage device 7 may use the group information section 744 for purposes other than training the machine learning model.

[0105] In the embodiment, the training unit 717 generates a 3D image to train the machine learning model. However, the present invention is not limited to this. The training unit 717 may use the product images stored in each group to train the machine learning model without generating a 3D image.

[0106] In the embodiment, the edge device 5 has been described as a standalone device. However, the present invention is not limited to this, and the edge device 5 may be incorporated in, for example, the product image storage device 7. In other words, the product image storage device 7 may have the functions of the edge device 5.

[0107] In addition, in the embodiment, the current time is used as an example of a predetermined time, but the predetermined time may be a time other than the current time as long as it is not too far from the current time, such as the closing time of a store yesterday.

[0108] In addition, the program executed by the product image storage device 7 of the embodiment is provided in an installable or executable format file recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk).

[0109] The program executed by the product image storage device 7 of the embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program executed by the product image storage device 7 of the embodiment may be provided or distributed via a network such as the Internet.

[0110] Furthermore, the program executed by the product image storage device 7 of the embodiment may be configured to be provided by being pre-installed in the ROM 72 or the like. [Explanation of symbols]

[0111] 1 POS terminal 3. Imaging device 5. Edge Devices 7 Product image storage device 10. System 31 CPU 34 Memory section 37 First Imaging Unit 38 Second Imaging Unit 39 Communications Department 71 CPU 74 Memory section 79 Communications Department 131 Product Information Department 331 Image storage unit 332 Product Specific Information Department 333 Barcode Information Department 711 Calculation section 712 Decision Section 713 Memory control unit 714 Performance memory section 715 Selection Section 716 3D Image Generation Unit 717 Training Department 732 Buffer 742 Object Detection Model 743 Feature Extraction Model 744 Group Information Department 745 Machine Learning Models 7441 Barcode Information Department 7442 Group number part 7443 Time information section 7444 Image Information Department [Prior art documents] [Patent documents]

[0112] [Patent Document 1] JP 2020-47051 A

Claims

1. An interface; A storage unit that stores group information that classifies the product images related to the same product identification information into one or more groups based on the product images of the products captured by the imaging unit and product identification information that identifies the products of the product images, which are input from the interface, in association with the product identification information; A processor; having The processor, Calculating, for each group, a similarity between the product image input together with the product identification information and a product image associated with the product identification information stored in the storage unit by group, based on the product identification information input from the interface; storing the input product image in a group determined according to the calculated similarity; The product image storage device according to the present invention is characterized in that

2. The processor, If the similarity is equal to or greater than a predetermined value, the product image input from the interface is added to the group and stored; If the similarity is less than a predetermined value, the product images input from the interface are stored as a new group related to the product identification information. The product image storage device according to claim 1 .

3. The processor, When the product identification information input from the interface is not stored in the storage unit, the product images are stored in the storage unit as a new group related to the product identification information input from the interface. The product image storage device according to claim 1 .

4. The storage unit stores, for each group, performance information indicating that the product image has been stored within a predetermined period of time, The processor, The performance information is newly stored for the group in which the product image is stored. The product image storage device according to claim 1 .

5. The performance information is time information indicating a time when the product image was stored within a predetermined period prior to a predetermined time, The processor, overwriting and storing time information relating to the time when the product image was stored for the group; The product image storage device according to claim 4 .

6. The processor, Selecting a group within the predetermined period based on the performance information, and training a machine learning model based on the product images stored in the selected group.

6. The product image storage device according to claim 4 or 5.

7. The processor, For each of the selected groups, a position and orientation of the product relative to the camera at the time of photographing the product is estimated based on the product images stored in the group, the product images are selected according to the estimation results, and a machine learning model is trained using the selected product images. The product image storage device according to claim 6 .

8. The processor, generating, for each of the selected groups, a 3D image relating to the product based on the product images stored in the group; Train a machine learning model using the generated 3D images; The product image storage device according to claim 6 .

9. a computer as a product image storage device having an interface, a storage unit that stores group information in which the product images related to the same product identification information are classified into one or more groups based on a product image of a product captured by an imaging unit and product identification information that is input from the interface, in association with the product identification information, and a processor; Calculating, for each group, a similarity between the product image input together with the product identification information and a product image associated with the product identification information stored in the storage unit by group, based on the product identification information input from the interface; storing the input product image in a group determined according to the calculated similarity; A program to make it function like this.

Images