Product image storage device and program

Abstract

To provide an imaging apparatus and a program that, in a case where a package of a commodity has been changed, enable a commodity image of the changed commodity to be easily distinguished from a commodity image before being changed and to be grouped.SOLUTION: A commodity image storage device includes: an interface; a storage unit that stores group information in which, based on commodity images of commodities imaged by an imaging unit and input from the interface and commodity specifying information that specifies the commodities in the commodity images, commodity images related to an identical type of commodity specifying information are classified into one or more groups, in association with the commodity specifying information; and a processor. The processor calculates, for each of the groups, a similarity between a commodity image input together with the commodity specifying information and a commodity image stored in the storage unit for each of the groups and associated with the commodity specifying information based on the commodity specifying information input from the interface, and stores the input commodity image in a group determined in accordance with the calculated similarity.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] FIELD An embodiment of the present invention relates to a product image storage device and a program. [Background technology]

[0002] POS systems exist that perform product registration and payment processes for products sold in stores such as shopping centers, mass retailers, and department stores. Some of these POS systems now use an imaging device to capture images of products and then recognize the packages to identify the products.

[0003] However, even for the same product, the design of the product's packaging may change (hereinafter referred to as "packaging change"), for example, depending on the season, etc. In such cases, in package recognition, in order to recognize that a product with changed packaging is the same product as the product before the change, it was necessary to train a machine learning model in advance based on images of the product with the changed packaging. Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide an imaging device and a program that, when the packaging of a product is changed, can group the product images of the changed product into a different group from the product images before the change, even if the product is the same. [Means for solving the problem]

[0005] The product image storage device of the embodiment has an interface, a storage unit that stores group information that classifies product images related to the same product identification information into one or more groups based on product images of products captured by an imaging unit and product identification information that identifies the product of the product images, which are input from the interface, in association with the product identification information, and a processor, and the processor calculates, based on the product identification information input from the interface, a similarity for each group between the product image input together with the product identification information and the product image associated with the product identification information stored by group in the storage unit, and stores the input product image in a group determined according to the calculated similarity. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating a system according to an embodiment. [Figure 2] FIG. 2 is a perspective view showing the appearance of the POS terminal and the imaging device. [Figure 3] FIG. 3 is a block diagram showing the hardware configuration of the imaging device. [Figure 4] FIG. 4 is a block diagram showing the hardware configuration of the product image storage device. [Figure 5] FIG. 5 is a diagram showing an example of extracting a scanned image. [Figure 6] FIG. 6 is an explanatory diagram showing the process of grouping product images based on captured images. [Figure 7] FIG. 7 is a memory map showing the configuration of the group information section. [Figure 8] FIG. 8 is a flowchart showing the flow of control processing of the imaging device. [Figure 9] FIG. 9 is a functional block diagram showing the functional configuration of the commodity image storage device. [Figure 10] FIG. 10 is a flowchart showing the flow of control processing related to grouping of commodity images by the commodity image storage device. [Figure 11]FIG. 11 is a flowchart showing the flow of control processing related to training of a machine learning model by the product image storage device. [Figure 12] FIG. 12 is a flowchart showing the flow of control processing related to training of a machine learning model using an image selection method by a product image storage device. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following embodiments, the product image storage device will be described as a server installed in a store that also serves as a store server for managing sales information for the store. However, the product image storage device may also be a headquarters server installed in the headquarters of a company that operates multiple stores, or a cloud server installed on the cloud. Furthermore, the present invention is not limited to the embodiments described below.

[0008] FIG. 1 is a diagram illustrating a system according to an embodiment. In FIG. 1, a system 10 is installed in a store T, such as a supermarket, a mass retailer, a convenience store, or a specialty store. The system 10 is installed in the store T and includes one or more POS (Point of Sales) terminals 1 (product sales data processing devices), an imaging device 3 connected to each POS terminal 1, an edge device 5, and a product image storage device 7. Each POS terminal 1, the edge device 5, and the product image storage device 7 are connected to a communication line L (e.g., a LAN (Local Area Network)) that allows mutual communication. Each imaging device 3 is connected to each POS terminal 1, and is connected to the POS terminal 1 by, for example, a USB (Universal Serial Bus) cable U. Each imaging device 3 is also connected to the edge device 5 and the product image storage device 7 by the communication line L.

[0009] In the embodiment, the edge device 5 is described as a device separate from the product image storage device 7, but the product image storage device 7 may also have the function of the edge device 5. In this case, the edge device 5 is deleted from the configuration of the system 10.

[0010] The system 10 performs product registration processing, payment processing, and sales management for products sold at the store T. The system 10 performs product registration processing using package recognition technology.

[0011] Package recognition is a technology for recognizing products (identifying one product or extracting multiple candidate products) by combining a technology for identifying a product by capturing and analyzing symbols such as barcodes or two-dimensional codes attached to the product (hereinafter referred to as "code recognition") with a well-known general object recognition technology for recognizing a product based on feature quantities indicating the product's external characteristics, such as the product's texture, shape, and packaging (design, etc.), based on a captured image of the product. Package recognition is a process performed by the edge device 5 and / or the product image storage device 7.

[0012] General object recognition is described, for example, in "Yanai Keiji, 'Current Status and Future of General Object Recognition,' Transactions of the Information Processing Society of Japan, Vol. 48, No. SIG16 [searched August 10, 2010], Internet <URL: http: / / mm.cs.uec.ac.jp / IPSJ-TCVIM-Yanai.pdf>".

[0013] Code recognition is a process performed by the imaging device 3, in which a symbol attached to a product is imaged and the symbol is decoded and analyzed to obtain barcode information. The barcode information includes product identification information that identifies the product to which the symbol is attached. The product identification information is, for example, a product code that indicates the product, but it does not have to be a product code as long as it is information that identifies the product. Code recognition is a process performed by the imaging device 3.

[0014] The imaging device 3 performs code recognition based on images captured by the first imaging unit 37 and the second imaging unit 38 (see FIGS. 2 and 3 for both) and generates barcode information. The imaging device 3 then outputs image information relating to the images captured by the first imaging unit 37 and the second imaging unit 38 and the generated barcode information to the commodity image storage device 7.

[0015] In this embodiment, the edge device 5 is stored inside the main body 42 (see FIG. 2). The edge device 5 performs generic object recognition based on image information input from the imaging device 3. Then, the edge device 5 outputs product identification information of the identified product or product identification information of multiple candidate products to the imaging device 3.

[0016] The imaging device 3 transmits to the POS terminal 1 either the product identification information included in the generated barcode information or the product identification information input from the edge device 5. Specifically, if the imaging device 3 is able to generate barcode information based on code recognition, it transmits to the POS terminal 1 either the product identification information included in the barcode information or the product identification information of the identified product input from the edge device 5. If the imaging device 3 is unable to generate barcode information, it transmits to the POS terminal 1 the product identification information of the product input from the edge device 5.

[0017] The product image storage device 7 groups and stores product images by product identification information (i.e., by product) based on the product images and product identification information of the captured images input from the imaging device 3 in which the product is captured. Specifically, the product image storage device 7 creates groups for product images with the same product identification information and similar features, and groups the product images. However, even for the same product, the packaging may change depending on the season, etc. Such products are assigned the same product identification information, but the features extracted from the product images differ depending on the packaging. In such a case, the product image storage device 7 groups products with the same product identification information, but with the same packaging, into different groups based on the features. In this way, product images with the same packaging are grouped in the same group, and product images with different packaging are grouped in different groups. Note that even for the same packaging, the features may change (become lower) depending on the conditions under which the product is captured (such as the brightness of the lighting or the image around the product). Therefore, products with unchanged packaging may be grouped into different groups.

[0018] Furthermore, the product image storage device 7 of the embodiment stores time information in association with each group. In theory, this time information is the time when the most recent product image was stored in the group, but in reality, it may be either the time when the product image was captured by the imaging device 3 (the time when the product registration operation was performed by the customer) or the time when barcode information was generated based on the captured symbol. Each time product images captured by the imaging device 3 are grouped, new time information is stored in the group in which the product image is stored (the time information is updated).

[0019] This time information may be information indicating a specific time, but does not have to be precise time information, and may be information indicating a time period, such as information indicating a specific date.

[0020] Furthermore, based on the time information, it is possible to check whether a product image has been stored in a group within a predetermined period of time going back from a predetermined time (for example, the current time (which may be the current date)). Therefore, the time information is an example of performance information indicating that the product image has been stored within a predetermined period of time going back from the present. Since the performance information is information that makes it possible to check whether a product image has been stored within a predetermined period of time going back from the present, it may be information (for example, flag information) indicating that a product image has been stored within a predetermined period of time (for example, the period from the previous training (training will be described later) to the present time), and the flag information or the like may be associated with a group in which a product image has been stored within a predetermined period of time. In this way, information indicating that a product image has been stored within a predetermined period of time is also an example of performance information indicating that the product image has been stored within a predetermined period of time going back from the present. Hereinafter, in the embodiments, time information will be described as an example of performance information.

[0021] If a group of similar features exists for a product image related to certain product feature information input from the imaging device 3 (for example, if the product is the same product and the packaging is the same (unchanged)), the product image storage device 7 adds and stores the input product image to the group and updates the time information for the group.

[0022] Furthermore, if there is no group of similar features for a product image related to certain product feature information input from the imaging device 3 (for example, in the case of the same product but with a different package), the product image storage device 7 creates a group with a new group number related to the product identification information, stores the input product image in the new group, and also stores the time information in the new group.

[0023] Furthermore, when certain product feature information input from the imaging device 3 is not stored (for example, in the case of a new product), the product image storage device 7 creates a new group with group number "1" relating to new product identification information, stores the input product image in the new group, and also stores the time information of the group.

[0024] Furthermore, the product image storage device 7 of the embodiment trains a machine learning model 745 (see FIG. 4 ). In the embodiment, the machine learning model 745 is stored in the product image storage device 7. However, the machine learning model 745 may be stored in a device other than the product image storage device 7.

[0025] The product image storage device 7 stores a machine learning model 745 (see FIG. 4). The machine learning model 745 is a model used, for example, to identify one product or extract multiple candidate products based on a product image captured by the imaging device 3.

[0026] As described above, the product image storage device 7 trains the machine learning model 745 to enable product recognition in response to the release of new products and changes to the packaging of existing products. When training for a product, the product image storage device 7 uses a stored group corresponding to the product (associated with the product identification information). Specifically, the product image storage device 7 selects a group that stores time information included in a predetermined period going back from the present, from the time information associated with each group. Then, the product image storage device 7 performs training for the product based on the product images included in the selected group.

[0027] The product image storage device 7 estimates the relationship between the product and the camera when each product image included in the selected group was captured, using technology such as Structure from Motion. During the estimation, it identifies and removes areas of a person's hand (a hand holding a product) or the like that appear in the image using technology such as semantic segmentation. The product image storage device 7 then selects product images so that the estimated positions and orientations are uniform. Training is performed using the selected product images. If other groups have been selected for the product, the product image storage device 7 performs similar processing on the other groups to train the product. In this way, the product image storage device 7 trains the machine learning model 745. The product image storage device 7 then outputs the trained machine learning model 745 to the edge device 5. The edge device 5 uses the machine learning model 745 input from the product image storage device 7 to perform generic object recognition on the product captured by the imaging device 3.

[0028] Furthermore, during training, a 3D image (3D model) may be generated based on the product images included in the selected group using techniques such as photogrammetry. The orientation of the generated 3D image can be changed to various angles to perform training on the product from multiple angles.

[0029] Next, the POS terminal 1 and the imaging device 3 will be described. Fig. 2 is a perspective view showing the appearance of a self-service POS terminal 1 that is operated by customers themselves, and the imaging device 3. As shown in Fig. 2, the POS terminal 1 has a main body 42 and a basket stand 4. The main body 42 houses a circuit board and a power supply that control the POS terminal 1. The basket stand 4 is attached to one side of the main body 42, and is a stand on which a customer places a basket containing items to be purchased.

[0030] The top surface of the main body 42 has a flat product placement section 41. The main body 42 has a pair of support posts 45 extending upward at the rear of the product placement section 41. A printer 22 that issues receipts is attached to the support post 45 on the side farther from the basket placement table 4. A card reader 21 that reads information from cards such as credit cards and point cards is attached to the top of the printer 22.

[0031] Additionally, an imaging device 3 is attached to the support 45 on the side closer to the basket placement platform 4. The imaging device 3 includes a first imaging section 37 (imaging section) and a second imaging section 38 (imaging section), and is a device that captures images of products scanned in front of the imaging device 3. The first imaging section 37 and the second imaging section 38 are, for example, cameras in which a large number of CCDs (Charge Coupled Devices: imaging elements) are arranged in a two-dimensional form (plane).

[0032] The imaging device 3 is installed with its reading surface facing the customer, and a first imaging unit 37 provided inside the imaging device 3 captures an image of a product by scanning the product that the customer has taken out of the basket. Furthermore, since customers tend to scan the product by pointing the symbol S (see FIG. 6) attached to the product toward the imaging device 3, the first imaging unit 37 often simultaneously captures an image of the symbol S attached to the product. In the following embodiments, it is assumed that the first imaging unit 37 captures an image of the symbol S.

[0033] Furthermore, the second imaging unit 38, which is part of the imaging device 3, is attached above the imaging device 3. The second imaging unit 38 is installed so that its reading surface faces diagonally downward from the imaging device 3, and images the scanned commodity from a different angle than the first imaging unit 37. In other words, the second imaging unit 38 images a different side of the commodity than the side imaged by the first imaging unit 37.

[0034] It is desirable to image the product from multiple angles using the imaging device 3. The customer searches for the position of the symbol S in front of the first imaging unit 37 (i.e., changes the orientation of the product) in order to have the symbol S aimed at the first imaging unit 37 and image it, so it is possible to image the product from multiple angles using the first imaging unit 37 and the second imaging unit 38. However, to image the product from more multiple angles (for example, uniformly image the entire surface of the product), this can be achieved by using an imaging device that images the product from all sides, such as that described in JP 2020-047051 A.

[0035] A display 18 is provided between the pair of support columns 45. The display 18 is a display made of, for example, a liquid crystal display, and is installed facing the customer who operates the POS terminal 1. The display 18 is located slightly above the imaging device 3 and the printer 22, and at the same height as the card reader 21 and the second imaging unit 38.

[0036] The display 18 displays product information (product name, product price, etc.) of the registered product and payment information (total amount, payment amount, change amount, etc.) of the payment processed. An operation unit 17 made up of a touch panel is provided on the display surface of the display 18.

[0037] In addition, between the pair of support columns 45 and below the display 18, there are provided a temporary stand 44 for temporarily placing products and a pair of attachment sections 43 for attaching handles of a shopping bag for placing products that have undergone product registration processing. The products placed in the shopping bag are placed on the product placement section 41.

[0038] Furthermore, a substantially cylindrical pole 6 extending upward is attached to the support 45 on the side closer to the car mounting base 4. A patrol lamp 61 is attached to the pole 6.

[0039] The patrol lamp 61 lights up or flashes when an abnormality occurs in the POS terminal 1 (for example, when the receipt paper runs out) or when a customer calls an attendant.

[0040] One or more POS terminals 1 (three in the first embodiment) are installed in the checkout area within store T. The POS terminal 1 is a product sales data processing device that allows customers to register products and perform checkout operations themselves. When a customer performs a product registration operation (for example, scanning a symbol S such as a barcode attached to a product and the product's exterior and having the imaging device 3 capture an image), the POS terminal 1 performs product registration processing for the product to be sold based on the product identification information received from the imaging device 3. The POS terminal 1 also performs payment processing based on the customer's checkout operation (operation to end the product registration operation and start the checkout process).

[0041] The product registration process refers to the process of acquiring a product code (product identification information) that identifies a product sold at store T, displaying product information (product name, product price, etc.) such as the product name and price of the product on display 18 based on the acquired product code, and storing the product information in the product information section. The settlement process refers to the closing process for the transaction based on the product information stored in the product information section in conjunction with the product registration process, specifically, displaying the total amount on display 18, settlement processing by media such as cash, credit card, electronic money, etc., calculating change based on the deposit in the case of cash settlement and displaying it on display 18, and issuing a receipt from printer 22 on which product information and accounting information (total amount, deposit amount, change amount, point information, etc.) of the settled product are printed.

[0042] The POS terminal 1 also transmits product information and transaction information (collectively referred to as "sales data") for the products for which payment has been processed to the product image storage device 7, which also functions as a store server. In this embodiment, the sales data is transmitted from the POS terminal 1 to the product image storage device 7 when the transaction process is completed, but it may also be transmitted at any timing, such as transmitting all data for one day at once when store T closes.

[0043] In such a system 10, a customer removes an item from a basket placed on the basket mounting table 4 and performs a product registration operation by passing (scanning) the symbol S attached to the item in front of the imaging device 3 (first imaging unit 37 and second imaging unit 38) to capture an image. The customer then places the item for which the product registration operation has been performed into a shopping bag attached to the attachment unit 43. The POS terminal 1 performs product registration processing for the item for which the product registration operation has been performed, based on the product identification information received from the imaging device 3. After completing the product registration operation for all items, the customer then performs a checkout operation by pressing the checkout button. When the checkout operation has been performed, the POS terminal 1 performs a payment process for the items for which the product registration operation has been performed.

[0044] The product image storage device 7 is installed, for example, in the back yard of the store T. The product image storage device 7 collects sales data received from the POS terminal 1. The product image storage device 7 manages the sales of products of the store T based on the collected sales data.

[0045] Next, the hardware configuration of the imaging device 3 will be described. Fig. 3 is a block diagram showing the hardware configuration of the imaging device 3. As shown in Fig. 3, the imaging device 3 includes a CPU 31, which is an example of a processor, a ROM 32, a RAM 33, a memory unit 34, and the like. The CPU 31 is the main controller of the imaging device 3. The ROM 32 stores various programs. The RAM 33 loads programs and various data. The memory unit 34 stores various programs. The CPU 31, ROM 32, RAM 33, and memory unit 34 are connected to one another via a bus 35. The CPU 31 executes control processing of the imaging device 3, which will be described later, by operating in accordance with a control program stored in the ROM 32 or the memory unit 34 and loaded in the RAM 33. Note that the processor may be a processor other than the CPU 31.

[0046] The RAM 33 has an image storage unit 331, a product identification information unit 332, and a barcode information unit 333. The image storage unit 331 stores the image captured by the first imaging unit 37 and the image captured by the second imaging unit 38. The product identification information unit 332 stores the product identification information input from the edge device 5. The barcode information unit 333 stores barcode information obtained by decoding (recognizing the code) the symbol S included in the image captured by the first imaging unit 37.

[0047] The memory unit 34 is configured by a non-volatile memory such as an HDD or flash memory that retains stored information even when the power is turned off, and has a control program unit 341. The control program unit 341 stores a control program for driving the imaging device 3.

[0048] The CPU 31 is also connected to a first imaging unit 37 and a second imaging unit 38 via a bus 35 and a controller 36 .

[0049] The CPU 31 is also connected to a communication unit 39 via a bus 35. The communication unit 39 is communicably connected to the edge device 5 and the product image storage device 7 via a communication line L. The communication unit 39 is also communicably connected to the POS terminal 1 via a USB cable U.

[0050] Next, the hardware configuration of the commodity image storage device 7 will be described. FIG. 4 is a block diagram showing the hardware configuration of the commodity image storage device 7. As shown in FIG. 4, the commodity image storage device 7 includes a CPU 71, which is an example of a processor, a ROM 72, a RAM 73, a memory unit 74, etc. The CPU 71 is the main controller of the commodity image storage device 7. The ROM 72 stores various programs. The RAM 73 expands programs and various data. The memory unit 74 stores various programs. The CPU 71, ROM 72, RAM 73, and memory unit 74 are connected to one another via a bus 75. The CPU 71 executes the control processing of the commodity image storage device 7, which will be described later, by operating in accordance with the control program stored in the ROM 72 or the memory unit 74 and expanded in the RAM 73. Note that the processor may be a processor other than the CPU 71.

[0051] The RAM 73 has a sales information section 731 and a buffer 732. The sales information section 731 stores sales data received from the POS terminal 1 and manages sales of the store T. The buffer 732 stores captured images and barcode information input from the imaging device 3.

[0052] The memory unit 74 is configured with a non-volatile memory (in this embodiment, an HDD) such as a hard disk drive (HDD) or flash memory that retains stored information even when the power is turned off, and has a control program unit 741, an object detection model 742, a feature extraction model 743, a group information unit 744 (storage unit), and a machine learning model 745. The control program unit 741 stores a control program for driving the product image storage device 7.

[0053] The object detection model 742 is a dictionary for identifying the area (range) of an object (product) that appears in a captured image. In other words, the object detection model 742 is a dictionary for identifying the area in which an object (product) appears in a captured image based on the input captured image. In other words, the object detection model 742 refers to a series of product images in which an object has been detected from the captured images input from the imaging device 3 as a scanned image K (described later in FIG. 5). The object detection model 742 is a dictionary for extracting an image of a product (hereinafter referred to as an "object image") that is surrounded by a identified area in the scanned image K.

[0054] The feature extraction model 743 is a dictionary for calculating feature quantities that are effective in identifying the product shown in the identified region (object image). That is, the feature extraction model 743 outputs a feature vector that indicates feature quantities that indicate the characteristics of the product whose region is identified, based on the input object image.

[0055] The group information section 744 stores product images of products identified by the product identification information by dividing them into one or more groups for each product identification information. The group information section 744 will be described later with reference to FIG.

[0056] The CPU 71 is also connected to a display unit 77 and an operation unit 78 via a bus 75 and a controller 76. The display unit 77 is, for example, a liquid crystal display, and displays information to an operator who operates the commodity image storage device 7. The operation unit 78 is, for example, a keyboard, and is operated by the operator of the commodity image storage device 7.

[0057] The CPU 71 is also connected to a communication unit 79 (interface (I / F)) via a bus 75. The communication unit 79 is connected to be able to communicate with the POS terminal 1, the imaging device 3, and the edge device 5 via a communication line L. The communication unit 79 may be composed of three physical interfaces: an image I / F that inputs the captured image output from the imaging device 3, a barcode information I / F that inputs the barcode information output from the imaging device 3, and a time information I / F that inputs the time information output from the imaging device 3, or may be a single interface that inputs the captured image, barcode information, and time information output from the imaging device 3.

[0058] Next, the scanned image will be described. The captured image of a product scanned (scanning operation) by a customer is sent from the imaging device 3 to the product image storage device 7. The images sent from the imaging device 3 include a mixture of images showing the product (product images) and images not showing the product. The product image storage device 7 extracts scanned images showing the object by detecting the object from the input captured image. The object detection may be performed using the object detection model 742, or may be performed simply using an infrared proximity sensor or utilizing technology such as an SSD (Single Shot Multibox Detector).

[0059] 5 is an explanatory diagram illustrating the extraction of scan image K from the image capture data input from the imaging device 3. As shown in FIG. 5, the CPU 71 of the product image storage device 7 extracts scan image K (Frame 0002 to Frame 0008), which is a series of product images in which object B (product) is detected, from a series of images (Frame 0001 to Frame 0009) obtained when a customer scans one product. A symbol S (barcode) appears in scan image K, and the imaging device 3 decodes (recognizes the code) this symbol S to generate barcode information and output it to the product image storage device 7. In the following description, images showing products, including scan image K, will be collectively referred to as product images.

[0060] Next, a description will be given of the process by which the CPU 71 groups product images (scanned images) based on the captured images and barcode information input from the imaging device 3. Fig. 6 is an explanatory diagram showing the process of grouping scanned images K based on the captured images.

[0061] As shown in Fig. 6, first, the CPU 71 extracts the scan image K shown in Fig. 5 from the captured image input from the imaging device 3. At that time, the CPU 71 uses the object detection model 742 to perform object B detection, which identifies the area of ​​the product that appears in the extracted scan image K (step A). ​​Next, the CPU 71 uses the feature extraction model 743 to calculate the feature amounts of the detected product and generate a feature vector (step B). Meanwhile, the CPU 71 searches the group information section 744 based on the barcode information input at the same time, and calculates the feature amounts of the product images included in each group for all groups associated with the barcode information, and generates a feature vector for each group.

[0062] Next, the CPU 71 calculates the similarity between the feature vector generated based on the input captured image and the feature vector for each group generated by searching the group information section 744 (step C). The similarity is calculated, for example, by using a convolutional neural network (CNN) used for product recognition to generate feature vectors for the input scanned image K and the scanned images K stored by group, and then calculating the cosine similarity between the generated feature vectors. Alternatively, local features such as SIFT may be calculated for each scanned image K, and the identity of the packages may be determined by matching the calculated features to calculate the similarity.

[0063] Next, the CPU 71 groups the scanned images K according to their similarities (step D). If there is a group with a high similarity to the generated feature vector, the CPU 71 adds and stores the scanned image K to that group. If there is no group with a high similarity to the generated feature vector, the CPU 71 creates a new group associated with the barcode information in the group information section 744 and stores the scanned image K in the created group. If, as a result of searching the group information section 744, the input barcode information is not stored in the group information section 744, the CPU 71 creates a new group associated with the input barcode information in the group information section 744 and stores the scanned image K in the created group. In this way, the input scanned images K are grouped.

[0064] Next, we will explain the group information section 744. Fig. 7 is a memory map showing the configuration of the group information section 744. Note that the example in Fig. 7 shows group information a of barcode information 1 and group information b of barcode information 2 indicating two types of products, but in reality there is as much group information related to barcode information as there are products sold at store T.

[0065] 7, group information section 744 stores group information a of a product identified by barcode information 1 and group information b of a product identified by barcode information 2. As described above, group information section 744 also stores group information other than group information a and group information b, but this will not be described here.

[0066] Group information a and group information b each have a barcode information section 7441, a group number section 7442, a time information section 7443, and an image information section 7444. The barcode information section 7441 stores barcode information that identifies a product. In this embodiment, barcode information 1 is stored in group information a, and barcode information 2 is stored in group information b. The group number section 7442 stores a group number indicating the group that belongs to the barcode information stored in the barcode information section 7441. In the example of FIG. 7, three groups (group 1, group 2, group 3) related to barcode information 1 belong to group information a. Four groups (group 1, group 2, group 3, group 4) related to barcode information 2 belong to group information b.

[0067] The time information section 7443 stores, for each group, time information (an example of performance information) indicating the time (or date, etc.) when the product image was most recently stored in the group, in correspondence with each group number stored in the group number section 7442.

[0068] The image information unit 7444 stores product images according to their mutual similarities, in association with each group number stored in the group number unit 7442. Specifically, the unit calculates the similarity between the feature amount (feature vector) of scan image K extracted based on the image input from the imaging device 3 and the feature amount (feature vector) calculated from the product images stored in each group, and if there is a group with high similarity (for example, a group with a calculated similarity of 95% or more), it additionally stores the scan image K in that group. If there is no group with high similarity, it creates a new group related to the barcode information, generates and stores a new group number in the group number unit 7442, stores the current time in the time information unit 7443 as time information for the created group, and stores the scan image K in the image information unit 7444.

[0069] When flag information is used as performance information, the flag information is stored in the time information section 7443. In this case, when a scanned image K is stored in the image information section 7444 after the previous training session, the flag information is stored in the time information section 7443 in association with the stored group.

[0070] Next, the control of the imaging device 3 will be described. Fig. 8 is a flowchart showing the flow of control processing of the imaging device 3. As shown in Fig. 8, the CPU 31 of the imaging device 3 determines whether an imaging start signal has been received (S11). The imaging start signal is received from the POS terminal 1, for example, when a customer operates a button to start operation on the POS terminal 1. The CPU 31 waits until the imaging start signal is received (No in S11), and if it is determined that the imaging start signal has been received (Yes in S11), the CPU 31 activates the first imaging unit 37 and the second imaging unit 38 to start imaging (S12).

[0071] Next, the CPU 31 determines whether an imaging end signal has been received (S13). The imaging end signal is received from the POS terminal 1, for example, when a customer has performed a payment process at the POS terminal 1. If it is determined that an imaging end signal has been received (Yes in S13), the CPU 31 ends imaging by the first imaging unit 37 and the second imaging unit 38 (S16). Then, the CPU 31 returns to S11.

[0072] If it is determined that the imaging end signal has not been received (No in S13), the CPU 31 stores the captured images whose imaging has started in S13 in the image storage unit 331 (S14). Then, the CPU 31 outputs the stored captured images to the edge device 5 and the product image storage device 7 (S15). Note that in S14, the CPU 31 stores all images captured from the start to the end of imaging.

[0073] Next, the CPU 31 determines whether the symbol S has been detected from the captured image stored in the image storage unit 331 (S21). If it determines that the symbol S has been detected (Yes in S21), the CPU 31 performs code recognition to decode the detected symbol S (S22). The CPU 31 then stores the barcode information generated by decoding and the time at which the barcode information was generated as time information in the barcode information unit 333 (S23). The CPU 31 then outputs the barcode information stored in the barcode information unit 333 (i.e., the barcode information for the captured image stored in S14) and the time information to the commodity image storage device 7 (S24). In this way, if the stored captured image contains the symbol S, the CPU 31 outputs the captured image, the decoded barcode information, and the time information to the commodity image storage device 7 consecutively at or nearly at the same time. Note that the CPU 31 may also transmit the barcode information stored in the barcode information unit 333 to the POS terminal 1. Then, the CPU 31 returns to S13.

[0074] Furthermore, if it is determined in S21 that the symbol S has not been detected (No in S21), the CPU 31 determines whether commodity identification information has been input from the edge device 5 (S25). If it is determined that commodity identification information has not been input (No in S25), the CPU 31 returns to S13. If it is determined that commodity identification information has been input from the edge device 5 (Yes in S25), the CPU 31 stores the commodity identification information in the commodity identification information unit 332 (S26). Then, the CPU 31 transmits the commodity identification information stored in the commodity identification information unit 332 to the POS terminal 1 (S27). Then, the CPU 31 returns to S13. In this way, the imaging device 3 outputs the captured images captured by the first imaging unit 37 and the second imaging unit 38, the barcode information obtained by decoding the symbol S, and the time information to the commodity image storage device 7.

[0075] Next, we will explain the functional configuration of the product image storage device 7. Fig. 9 is a functional block diagram showing the functional configuration of the product image storage device 7. The CPU 71 of the product image storage device 7 functions as a calculation unit 711, a determination unit 712, a memory control unit 713, a performance storage unit 714, a selection unit 715, a 3D image generation unit 716, and a training unit 717 by following the control programs stored in the ROM 72 and the control program unit 741 of the memory unit 74.

[0076] Based on the product identification information included in the barcode information input from the communication unit 79 (interface), the calculation unit 711 calculates, for each group, the similarity between the scanned image K input together with the product identification information and the product images associated with the product identification information stored by group in the group information unit 744. Specifically, based on the product identification information included in the barcode information input from the communication unit 79, the calculation unit 711 generates a feature vector representing the feature amounts of the scanned image K input together with the product identification information and the product images associated with the product identification information stored by group in the group information unit 744, and calculates the similarity related to the generated feature vector for each group.

[0077] The determination unit 712 determines a group in which to store the input scanned image K according to the calculated similarity. Specifically, if there is a group with high similarity, the determination unit 712 determines that group (an existing group) as the group in which to store the input scanned image K (sets a group number for that group). If there is no group with high similarity (if the similarity with any group is low), the determination unit 712 determines a new group related to the product identification information (creates a new group related to the product identification information and sets a group number for that group). If the product identification information input by the calculation unit 711 is not in the group information unit 744, the determination unit 712 determines a new group created based on the input product identification information (creates a new group related to the new product identification information and sets the group number for that group to "1").

[0078] The storage control unit 713 stores the product images input into the group determined according to the calculated similarity. In an embodiment, the storage control unit 713 stores the scanned image K input into the group determined by the determination unit 712. Specifically, when the determination unit 712 determines an existing group with a high similarity, the storage control unit 713 additionally stores the scanned image in the existing group. Furthermore, when the determination unit 712 determines a new group related to the same barcode information, the storage control unit 713 stores the scanned image in the new group. Furthermore, when the determination unit 712 determines a new group related to new barcode information, the storage control unit 713 stores the scanned image in the new group.

[0079] The result storage unit 714 stores the result information for the group in which the scanned image K is stored. Specifically, the result storage unit 714 stores the time information input from the imaging device 3 for the group in which the scanned image K is stored.

[0080] The selection unit 715 selects a group for which performance information is stored. Specifically, the selection unit 715 selects a group for which time information indicating a period within a predetermined period going back from the current time, which is a predetermined time, is stored (i.e., a group of products that have been frequently purchased recently). The predetermined period is, for example, the range of the date and time of the last training session going back from the current time. The predetermined period can be set by the store T according to the circumstances of the store T.

[0081] The 3D image generating unit 716 generates, for each group, a 3D image of the product based on the product images (accumulated scanned images K) stored in each group, using a technique such as photogrammetry.

[0082] The training unit 717 trains the machine learning model 745 using the 3D image generated by the 3D image generation unit 716. Specifically, the training unit 717 uniformly trains the machine learning model 745 using various faces of the 3D image generated by the 3D image generation unit 716.

[0083] Next, the control of the commodity image storage device 7 will be described. FIG. 10 is a flowchart showing the flow of control processing when the commodity image storage device 7 groups and stores the scanned images K input from the imaging device 3. As shown in FIG. 10, the CPU 71 of the commodity image storage device 7 determines whether a captured image output from the imaging device 3 has been input (acquired) (S31). The CPU 71 waits until the captured image is input (No in S31), and if it determines that the captured image has been input (acquired) (Yes in S31), the CPU 71 performs object detection processing (detection of a commodity) based on the acquired captured image (S32). The CPU 71 then determines whether an object (commodity) has been detected from the input captured image (S33). If it determines that a commodity has been detected (Yes in S33), the CPU 71 extracts a scanned image K from the object image and stores the extracted scanned image K in the buffer 732 (S34).

[0084] Next, the CPU 71 determines whether barcode information and time information related to the input captured image have been acquired from the imaging device 3 (S35). If it has determined that the barcode information and time information related to the input captured image have not been acquired (No in S35), the CPU 71 returns to S31. If it has determined that the barcode information and time information related to the input captured image have been acquired (Yes in S35), the CPU 71 stores the input barcode information and time information in the buffer 732 (S36). Through the processes of S34 and S36, the captured image is associated with the buffer information and time information related to the symbol S that appeared in the captured image. Then, the CPU 71 returns to S31.

[0085] Furthermore, if it is determined that no product has been detected (No in S33; for example, no product is detected when the customer has finished scanning one product), the CPU 71 determines whether barcode information and time information are stored in the buffer 732 (S41). If it is determined that barcode information and time information are not stored in the buffer 732 (No in S41), the CPU 71 clears the buffer 732 (S42). That is, the CPU 71 clears the captured images stored in the buffer 732. Then, the CPU 71 returns to S31.

[0086] If it is determined that barcode information and time information are stored in buffer 732 (Yes in S41), CPU 71 searches group information section 744 based on the barcode information (S43). Then, CPU 71 determines whether the same barcode information is stored in group information section 744 (S44).

[0087] If it is determined that the same barcode information is stored (Yes in S44), the calculation unit 711 calculates the similarity based on the feature vector calculated based on the scanned image K stored in the buffer 732 and the feature vector for each group calculated based on the product image stored in the image information unit 7444 of each group associated with the same barcode information in the group information unit 744 (S45).

[0088] Based on the similarity calculated by the calculation unit 711, the CPU 71 determines whether there is a group having a feature vector highly similar to the feature vector calculated based on the scanned image K stored in the buffer 732 (S46). If it is determined that there is a group having a feature vector with high similarity (Yes in S46), the determination unit 712 determines the group number stored in the group number section 7442 corresponding to that group as the group of scanned image K stored in the buffer 732 (S47). The storage control unit 713 then adds and stores the scanned image K in the image information section 7444 associated with the determined group number (S48). The performance storage unit 714 also overwrites and stores the time information stored in the buffer 732 in the time information section 7443 associated with the determined group number (S48). Through the processing of the performance storage unit 714 in S48, the latest time information when the scanned image K was stored in the group is stored, and the time information is updated. The CPU 71 then ends the processing and returns to S31.

[0089] On the other hand, if it is determined that there is no group having a feature vector with high similarity (No in S46), the determination unit 712 generates a new group number belonging to the barcode information and determines that group number as the group of scanned image K stored in buffer 732 (S51). Then, the storage control unit 713 stores the determined new group number in group number section 7442 (S52). The storage control unit 713 also stores the scanned image K in the group created in image information section 7444 and associated with the new group number (S52). The performance record storage unit 714 also stores the time information stored in buffer 732 in time information section 7443 of the group associated with the determined new group number (S52). The processing of performance record storage unit 714 in S52 stores the latest time information at which scanned image K was stored in the new group. Then, the CPU 71 ends the processing and returns to S31.

[0090] Furthermore, if it is determined in S44 that the same barcode information is not stored (No in S44), the determination unit 712 generates a new group number (group number "1") belonging to the barcode information stored in the buffer 732 and determines this group number as the group of the scanned image K stored in the buffer 732 (S53). The storage control unit 713 then generates group information (e.g., group information c) related to the new barcode information and stores the new group number related to this group information in the group number section 7442 (S54). The storage control unit 713 also stores the scanned image K in the image information section 7444 associated with the new group number (S54). The performance record storage unit 714 also stores the time information stored in the buffer 732 in the time information section 7443 associated with the determined new group number "1" (S54). By the processing of the performance record storage unit 714 in S54, the latest time information when the scanned image K was stored in the group with group number "1" is stored. Then, the CPU 71 ends the process and returns to S31.

[0091] In this embodiment, the product image storage device 7 stores product images of products identified by barcode information in one or more groups, each group being a group of images with high similarity. The input scanned image K is then added to and stored in a group with high similarity, and if the scanned image K is not highly similar to any of the groups, a new group is created and a new group is created to store the scanned image K.

[0092] According to such an embodiment, when the packaging of a product is changed, it is possible to store the product images of the same product that have been changed in a separate group from the product images before the change.

[0093] Next, we will explain the control of the commodity image storage device 7 when the start of training is operated when training the machine learning model 745. FIG. 11 is a flowchart showing the flow of control processing for training the machine learning model 745 by the commodity image storage device 7. As shown in FIG. 11, the selection unit 715 selects, in the group information related to each barcode information stored in the group information unit 744, a group for which the time information stored in the time information unit 7443 is within a predetermined period prior to the current time, as a group to be used for training the machine learning model 745 (S61). Therefore, groups of new products or products whose packaging has recently been changed within the predetermined period are selected. On the other hand, groups of discontinued products or products with old packaging (products with packaging that is no longer on sale) are not selected.

[0094] Next, the 3D image generation unit 716 generates a 3D image for the selected group based on the product images stored in the image information unit 7444 (S62). The training unit 717 then uses the generated 3D image to perform training on the product identified by the barcode information (S63). The training in S63 is performed by slightly changing the orientation (angle) of the generated 3D image each time, thereby uniformly training the product on various sides. Similar training is also performed for different groups. Similar training is also performed for products with different barcode information.

[0095] To end the training, for example, an operation to end the training is performed. The CPU 71 determines whether the operation to end the training has been performed by the operator (S64). If it is determined that the operation to end the training has not been performed (No in S64), the process returns to S63 because further training continues, and the training unit 717 changes the orientation of the 3D image and performs further training (S63). The training unit 717 also performs training for different groups. In addition, similar training is performed for products with different barcode information.

[0096] On the other hand, if it is determined that an operation to end training has been performed (Yes in S64), the CPU 71 ends the process.

[0097] Next, training of the machine learning model 745 using techniques such as Structure from Motion will be described. FIG. 12 is a flowchart showing the flow of control processing related to training of the machine learning model using a Structure from Motion-based image selection method. As shown in FIG. 12, the selection unit 715 selects, in the group information related to each barcode information stored in the group information unit 744, a group whose time information stored in the time information unit 7443 is within a predetermined period prior to the current time, as a group to be used for training the machine learning model 745 (S71). Therefore, groups of new products or products whose packaging has recently been changed within the predetermined period are selected. On the other hand, groups of discontinued products or products with old packaging (products with packaging that is no longer on sale) are not selected.

[0098] Next, the 3D image generation unit 716 estimates the position and orientation of the product and camera for the selected group based on the product images stored in the image information unit 7444 (S72). The 3D image generation unit 716 then selects product images so that the estimated positions and orientations are uniform (S73). The training unit 717 then uses the selected product images to perform training on the product identified by the barcode information (S74). The training unit 717 performs similar training for different groups. The training unit 717 also performs similar training for products with different barcode information.

[0099] To end the training, for example, an operation to end the training is performed. The CPU 71 determines whether the operation to end the training has been performed by the operator (S75). If it is determined that the operation to end the training has not been performed (No in S75), the process returns to S64 because further training continues, and the training unit 717 performs further training (S74). The training unit 717 also performs training for different groups. In addition, similar training is performed for products with different barcode information.

[0100] On the other hand, if it is determined that an operation to end training has been performed (Yes in S75), the CPU 71 ends the process.

[0101] In such an embodiment, when training the machine learning model 745, the product image storage device 7 can select groups in which time information has been stored within a specified period (for example, groups in which packaging has been changed), and can train without selecting groups in which time information has not been stored within the specified period (for example, groups with old packaging outside the specified period), thereby improving the accuracy of training.

[0102] Although an embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be embodied in various other forms, and various omissions, substitutions, modifications, and combinations can be made without departing from the spirit of the invention. This embodiment and its modifications are included within the scope and spirit of the invention, and are also included in the inventions described in the claims and their equivalents.

[0103] For example, in the embodiment, the product image storage device 7 performs grouping (classification) of product images and training of the machine learning model 745. However, this is not limited to this, and the product image storage device 7 may also be a device that performs grouping of product images. In this case, a training device that trains the machine learning model may be separately provided, and the training device may perform training of the machine learning model using information on the groups created by the product image storage device 7. In this case, the 3D image generation unit 716 and the training unit 717 in FIG. 9 are functional configurations of the training device, and the control in FIG. 11 is performed by the training device. In this case, the time information may be stored in the product image storage device 7 or the training device.

[0104] In the embodiment, the machine learning model is trained using the group information section 744 generated by the commodity image storage device 7. However, the present invention is not limited to this, and the commodity image storage device 7 may use the group information section 744 generated by the grouping for purposes other than training the machine learning model.

[0105] In the embodiment, the training unit 717 generated 3D images to train the machine learning model. However, the present invention is not limited to this. The training unit 717 may use the product images accumulated in each group to train the machine learning model without generating 3D images.

[0106] In the embodiment, the edge device 5 has been described as a standalone device. However, this is not limiting, and the edge device 5 may be incorporated into, for example, the product image storage device 7. In other words, the product image storage device 7 may have the functions of the edge device 5.

[0107] In addition, in the embodiment, the current time is used as an example of a predetermined time, but the predetermined time may be a time other than the current time as long as it is not too far from the current time, such as the closing time of a store yesterday.

[0108] The program executed by the product image storage device 7 of the embodiment is provided as a file in an installable or executable format recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk).

[0109] The program executed by the commodity image storage device 7 of the embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program executed by the commodity image storage device 7 of the embodiment may be provided or distributed via a network such as the Internet.

[0110] Furthermore, the program executed by the commodity image storage device 7 of the embodiment may be configured to be provided by being pre-installed in the ROM 72 or the like. [Explanation of symbols]

[0111] 1 POS terminal 3. Imaging device 5. Edge Devices 7 Product image storage device 10 Systems 31 CPU 34 Memory section 37 First imaging unit 38 Second imaging unit 39 Communications Department 71 CPU 74 Memory section 79 Communications Department 131 Product Information Department 331 Image storage unit 332 Product Specific Information Department 333 Barcode Information Department 711 Calculation Unit 712 Decision Section 713 Memory control unit 714 Performance memory section 715 Selection Section 716 3D image generation unit 717 Training Department 732 buffers 742 object detection model 743 Feature Extraction Model 744 Group Information Department 745 machine learning models 7441 Barcode Information Department 7442 Group Number Part 7443 Time information department 7444 Image Information Department [Prior art documents] [Patent documents]

[0112] [Patent Document 1] Japanese Patent Publication No. 2020-47051

Claims

1. An interface, a storage unit that stores group information, which classifies the product images related to the same product identification information into one or more groups based on the product images of the products captured by the imaging unit and the product identification information that identifies the products of the product images, input from the interface, in association with the product identification information; a processor; and The processor: Based on the product identification information input from the interface, a similarity is calculated for each group between the product image input together with the product identification information and the product image associated with the product identification information stored for each group in the storage unit; storing the input product images in a group determined according to the calculated similarity; A product image storage device characterized by:

2. The processor: If the similarity is equal to or greater than a predetermined value, the product image input from the interface is added to the group and stored; If the similarity is less than a predetermined value, the product images input from the interface are stored as a new group related to the product identification information. The product image storage device according to claim 1 .

3. The processor: If the product identification information input from the interface is not stored in the storage unit, the product images are stored in the storage unit as a new group related to the product identification information input from the interface. The product image storage device according to claim 1 .

4. The storage unit stores, for each group, performance information indicating that the product image has been stored within a predetermined period of time, The processor: The performance information is newly stored for the group in which the product image is stored. The product image storage device according to claim 1 .

5. The performance information is time information indicating the time when the product image was stored within a predetermined period of time prior to a predetermined time, The processor: overwriting and storing time information relating to the time when the product image was stored for the group in which the product image was stored; 5. The product image storage device according to claim 4.

6. The processor: Selecting a group within the predetermined period based on the performance information, and training a machine learning model based on the product images stored in the selected group.

6. The product image storage device according to claim 4 or 5.

7. The processor: For each of the selected groups, the position and orientation of the product relative to the camera at the time of photographing is estimated based on the product images stored in the group, the product images are selected according to the estimation results, and a machine learning model is trained using the selected product images. The product image storage device according to claim 6 .

8. The processor: For each of the selected groups, generate a 3D image of the product based on the product images stored in the group; Train a machine learning model using the generated 3D images; The product image storage device according to claim 6 .

9. a computer as a commodity image storage device having an interface, a storage unit that stores group information in which commodity images of commodities captured by an imaging unit and commodity identification information that identifies the commodity of the commodity image input from the interface are classified into one or more groups based on the commodity images of the commodities captured by the imaging unit and the commodity identification information, in association with the commodity identification information; and a processor; Based on the product identification information input from the interface, a similarity is calculated for each group between the product image input together with the product identification information and the product image associated with the product identification information stored for each group in the storage unit; storing the input product images in a group determined according to the calculated similarity; A program to make it work like this.

Images