Imaging device and program

Abstract

To provide an imaging device and a program that can easily learn a dictionary.SOLUTION: An image capturing unit according to the embodiment includes: an image capturing unit; decoding means for generating decoding information by decoding a symbol attached to an article included in an image captured by the image capturing unit; associating means for associating the decoded information with an image captured by the image capturing unit; output means for outputting the associated information associated by the associating means; transmission means for transmitting to an article sales data processor either article identification information contained in the decoded information or the article identification information identified by general object recognition performed on the image captured by the image capturing unit.SELECTED DRAWING: Figure 12

Description

Technical Field

[0001] Embodiments of the present invention relate to an imaging device and a program.

Background Art

[0002] Recently, in stores such as shopping centers, supermarkets, department stores, etc., there is a POS system that performs product registration processing and settlement processing of products sold in the store. Some of such POS systems determine products by package recognition of images of products captured by an imaging device.

[0003] By the way, in order to correctly determine a product from a captured image, it is necessary to improve the accuracy of a dictionary used to estimate the product from the image. In order to improve the accuracy of the dictionary, it is necessary to collect many images related to the product and let the dictionary learn them. For example, since new products are sold or the design of a package or the like may be changed depending on the season even for the same product, it is necessary to learn the dictionary daily.

[0004] However, in order to learn the dictionary, it is necessary for a person to capture an image of a product, which is a burden on the operator.

Summary of the Invention

Problems to be Solved by the Invention

[0005] The problem to be solved by the present invention is to provide an imaging device and a program that can easily learn a dictionary.

Means for Solving the Problems

[0006] The imaging device according to the embodiment includes an imaging unit, A storage means for storing multiple images captured by the imaging unit, the The means of memory remembered decoding means for generating decoded information obtained by decoding a symbol attached to a product included in the image, and the Among multiple images stored by the memory device, the object image contains the product enclosed in a specified area.The system includes association means for associating the decoded information with the product identification information, output means for outputting the association information associated by the association means, and transmission means for transmitting either the product identification information contained in the decoded information or the product identification information identified by general object recognition performed on the image captured by the imaging unit to a product sales data processing device. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 shows a system according to an embodiment. [Figure 2] Figure 2 is an explanatory diagram illustrating the principle of package recognition. [Figure 3] Figure 3 shows an example of how an object detection model extracts product images. [Figure 4] Figure 4 is a perspective view showing the external appearance of the POS terminal and imaging device. [Figure 5] Figure 5 is a timing chart showing the processing flow in the system. [Figure 6] Figure 6 is a block diagram showing the hardware configuration of a POS terminal. [Figure 7] Figure 7 is a block diagram showing the hardware configuration of the imaging device. [Figure 8] Figure 8 is a block diagram showing the hardware configuration of the edge device. [Figure 9] Figure 9 is a block diagram showing the hardware configuration of the store server. [Figure 10] Figure 10 is a flowchart showing the flow of control processing for a POS terminal. [Figure 11] Figure 11 is a functional block diagram showing the functional configuration of the imaging device. [Figure 12] Figure 12 is a flowchart showing the control process flow of the imaging device. [Figure 13] Figure 13 is a flowchart showing the control process flow for edge devices. [Figure 14] Figure 14 is a flowchart showing the flow of control processing for the store server. [Modes for carrying out the invention]

[0008] Embodiments of the present invention will be described below with reference to the drawings. In one embodiment, an imaging device that captures the customer's scanning of products in a system where the customer performs registration and accounting operations will be described as an example. However, the present invention is not limited to the first embodiment described below.

[0009] Figure 1 shows a system according to an embodiment. In Figure 1, system 10 is installed in a store T such as a supermarket, mass retailer, convenience store, or specialty store. System 10 is a system installed within store T and comprises one or more POS (Point of Sales) terminals 1 (product sales data processing devices), an imaging device 3 connected to each POS terminal 1, an edge device 5, and a store server 7. Each POS terminal 1, edge device 5, and store server 7 are connected to each other by a communication line L (e.g., LAN (Local Area Network)). An imaging device 3 is connected to each POS terminal 1, for example, by a USB (Universal Serial Bus) cable U. Each imaging device 3 is also connected to the edge device 5 and the store server 7 by a communication line L.

[0010] System 10 handles product registration, payment processing, and sales management for products sold at store T. System 10 also uses package recognition technology to perform product registration.

[0011] Package recognition is a technology that identifies a product by combining a technology that identifies the product by capturing and analyzing symbols such as barcodes and two-dimensional codes attached to the product (hereinafter referred to as "code recognition") with a known general object recognition technology that recognizes the product based on the shape and appearance of the captured product.

[0012] Regarding general object recognition, for example, it is described in "Keiji Yanai, 'Current Status and Future of General Object Recognition', Transactions of the Information Processing Society of Japan, Vol. 48, No. SIG16 [searched on August 10, 2010], Internet <URL: http: / / mm.cs.uec.ac.jp / IPSJ-TCVIM-Yanai.pdf>".

[0013] In code recognition, when an image of a symbol attached to a product is captured, decoded information obtained by decoding and analyzing the symbol is acquired. The decoded information includes product identification information for identifying the product to which the symbol is attached. The product identification information is, for example, a product code indicating the product, but it does not have to be a product code as long as it is information for identifying the product.

[0014] Next, general object recognition will be described. FIG. 2 is a diagram for explaining general object recognition. As shown in FIG. 2, general object recognition uses three types of dictionaries, an object detection model 542, a feature extraction model 543, and a verification dictionary 544, to identify the captured product.

[0015] The object detection model 542 is a dictionary for identifying the area (range) of an object (product) shown in the captured image. That is, the object detection model 542 is a dictionary for identifying the area in the input captured image where an object (product) is shown based on the captured image. In other words, the object detection model 542 is a dictionary for extracting an image of a product (hereinafter referred to as an "object image") surrounded by a specified area in the captured image (the image of the customer scanning the product described later).

[0016] The feature extraction model 543 is a dictionary for calculating feature amounts effective for differentiating the products shown in the specified area (object image). That is, the feature extraction model 543 outputs a feature vector indicating the feature amounts showing the features of the product with the area specified based on the input object image.

[0017] The verification dictionary 544 is a dictionary for identifying products based on the output feature vectors. That is, the verification dictionary 544 stores, for a large number of products, the feature quantities indicating the features of each product separately for the product identification information for identifying each product. By comparing the feature quantity indicated by the input feature vector with the stored feature quantity, the similarity indicating how similar the stored large number of products are to the imaged product is calculated. The higher the similarity, the higher the degree of similarity to the imaged product (i.e., the probability of being the imaged product). For example, a product with a similarity of 95% or more has a very high probability of being the same product as the imaged product.

[0018] For general object recognition, using the object detection model 542, the area where the product appears is identified from the image captured by an imaging unit such as a camera. Next, using the feature extraction model 543, the feature quantity of the product, which represents the surface information such as the external shape, color tone, pattern, and unevenness of the product appearing in the area (object image) as parameters (the "feature vector" shown in FIG. 2), is extracted. Next, using the verification dictionary 544, the extracted feature quantity is compared with the feature quantity data of a large number of products stored in the dictionary, and the similarity indicating how similar the stored products are to the product is calculated, and the product is recognized according to the similarity. Based on the similarity calculated by such general object recognition, when there is one product with a similarity equal to or higher than the threshold (for example, 95% or more), the system 10 identifies the recognized product as the imaged product (product A in FIG. 2). When there is no product with a similarity higher than the threshold (for example, all similarities are lower than 95%), instead of identifying one product, a plurality of products with high recognized similarities are set as candidate products.

[0019] For package recognition, using the code recognition technology and the general object recognition technology, the product is identified based on, for example, a technology that can identify the product earlier, and the information of the identified product is transmitted (output) to the POS terminal 1 or the like. However, when the product cannot be identified as one by general object recognition and the code has not been recognized yet, package recognition may wait for the product to be identified by code recognition for a further predetermined time.

[0020] The imaging device 3 comprises a first imaging unit 37 (imaging unit) and a second imaging unit 38 (imaging unit) (see Figure 7), and is a device that captures images of products scanned in front of the imaging device 3. The first imaging unit 37 and the second imaging unit 38 are, for example, cameras in which a large number of CCDs (Charge Coupled Devices: image sensors) are arranged in a secondary plane (planar arrangement).

[0021] The imaging device 3 extracts product images containing the product from the images captured by the first imaging unit 37 and the second imaging unit 38. Figure 3 shows an example of extracting product images from captured images. In Figure 3, images from frames 1 to 9 have been captured. In Figure 3, frame 1 shows nothing, but in reality, it shows the customer's hands and background used for scanning the product, and not the product itself. The same is true for frame 9. In contrast, frames 2 to 8 show the product in addition to the customer's hands and background. Therefore, the imaging device 3 uses known techniques to extract the images from frames 2 to 8, which contain the product, as product images from the images from frames 1 to 9.

[0022] Methods for extracting product images from captured images include, for example, a method of distinguishing between the product and hands or background and extracting images that show the product. Other methods include extracting images of the product before and after an image in which a symbol has been recognized, extracting images of the product by working backward from an image in which a symbol has been recognized, and extracting images of the product after an image in which a symbol has been recognized. Alternatively, the imaging device 3 may be equipped with a dictionary similar to the object detection model 542, and object images in which the product region can be calculated using this dictionary may be extracted as product images.

[0023] Furthermore, the imaging device 3 performs code recognition based on the images captured by the first imaging unit 37 and the second imaging unit 38, and generates decoded information. The imaging device 3 then generates association information by associating the generated decoded information with product images of the products to which the decoded symbol is attached (for example, all product images (a series of still images) from the time the product to which the symbol is first detected until the detection is completed). The imaging device 3 then outputs the association information to the edge device 5.

[0024] The edge device 5 performs general object recognition. Specifically, the edge device 5 detects an object image from the images captured by the first imaging unit 37 and the second imaging unit 38, using the object detection model 542 to identify the area (range) of the product (Figure 2A).

[0025] Next, the edge device 5 uses the feature extraction model 543 to calculate feature quantities that represent the characteristics of the product from the detected object image, and outputs the feature vector of the product (Figure 2B). Then, the edge device 5 uses the matching dictionary 544 to calculate a similarity score that indicates the degree of similarity between the product in question and the stored products, based on the output feature vector. The edge device 5 then recognizes products with a high similarity score as the products that were imaged. Finally, the edge device 5 outputs the recognition result to the imaging device 3.

[0026] The edge device 5 also inputs and stores the association information output by the imaging device 3. The edge device 5 then transmits the stored association information to the store server 7 via the communication line L. The edge device 5 then receives and stores the matching dictionary 544 learned based on the transmitted association information from the store server 7. The edge device 5 improves the accuracy of the stored matching dictionary 544 by overwriting the information in the matching dictionary received from the store server 7.

[0027] The imaging device 3 determines which product to send product identification information to the POS terminal 1 based on the recognition result input from the edge device 5 and the generated decoded information. For example, if the product identification information input from the edge device 5 indicates a specific single product (i.e., product identification information relating to a single identified product is input from the edge device 5), the imaging device 3 sends the product identification information to the POS terminal 1 if decoded information has not yet been generated. If decoded information has already been generated, the imaging device 3 sends the product identification information included in the decoded information to the POS terminal 1.

[0028] On the other hand, if the product identification information input from the edge device 5 indicates multiple products (i.e., multiple product identification information is input from the edge device 5), the imaging device 3, if decoded information has already been generated, transmits the product identification information indicated by the decoded information to the POS terminal 1. If decoded information has not yet been generated, it waits for a predetermined time for the decoded information to be generated, and when the decoded information is generated, it transmits the product identification information indicated by the decoded information to the POS terminal 1. If decoded information is not generated even after the predetermined time has elapsed, it transmits the multiple product identification information input from the edge device 5 to the POS terminal 1. Based on the product identification information of the multiple candidate products received, the POS terminal 1 displays these candidate products in a selectable format.

[0029] One or more (three in the first embodiment) POS terminals 1 are installed in the checkout area of ​​the store T. POS terminal 1 is a product sales data processing device in which the customer performs product registration and payment operations. When a customer performs a product registration operation (for example, scanning a barcode or other symbol attached to a product and the appearance of the product to be captured by the imaging device 3), POS terminal 1 executes product registration processing for the product being sold based on product identification information received from the imaging device 3. POS terminal 1 also executes payment processing based on the customer's payment operation (the operation of ending product registration and starting payment processing).

[0030] Product registration processing refers to the process of obtaining a product code (product identification information) that identifies products sold at store T, displaying product information such as the product name and price (product name, product price, etc.) on the display unit 18 (see Figures 4 and 6) based on the obtained product code, and storing the product information in the product information unit 131 (see Figure 6). Settlement processing refers to the process of closing the transaction based on the product information stored in the product information unit 131 in conjunction with the product registration processing, specifically displaying the total amount on the display unit 18, processing settlement using media such as cash, credit card, and electronic money, calculating change based on the deposit in the case of cash payment and displaying it on the display unit 18, and printing a receipt from the printer 22 (see Figures 4 and 6) with product information and accounting information (total amount, deposit amount, change amount, point information, etc.) of the settled products.

[0031] The POS terminal 1 also transmits product information and accounting information (collectively referred to as "sales data") of the processed products to the store server 7. In this embodiment, the transmission of sales data from the POS terminal 1 to the store server 7 is performed when the accounting process is completed, but it may also be transmitted at any arbitrary time, for example, by sending a full day's worth of data at the closing time of store T.

[0032] Store server 7 is installed in the back room of store T. Store server 7 collects sales data received from POS terminal 1. Based on the collected sales data, store server 7 manages the sales of products at store T.

[0033] Furthermore, the store server 7 stores the associated information received from the edge device 5 in addition to the matching dictionary 744 (see Figure 9) that it stores. Specifically, the store server 7 uses the object detection model 742 (see Figure 9) to extract the captured image of the product image included in the associated information received from the edge device 5, and uses the feature extraction model 743 (see Figure 9) to extract features based on the captured image. Then, the store server 7 adds the extracted features to the matching dictionary 744 that it stores, based on the product identification information included in the decoded information associated with the product image. In this way, by increasing the number of features stored in the matching dictionary 744 for each piece of product identification information, the matching dictionary 744 is trained, improving the accuracy of product recognition by the matching dictionary 744. The store server 7 then transmits the information of the trained matching dictionary 744 to the edge device 5.

[0034] In this embodiment, the object detection model 742, the feature extraction model 743, and the matching dictionary 744 are provided on the store server 7. However, the object detection model 742, the feature extraction model 743, and the matching dictionary 744 may be provided on a server other than the store server 7 (for example, a headquarters server located at headquarters that manages and coordinates multiple store servers 7).

[0035] Next, the POS terminal 1 and the imaging device 3 will be described. Figure 4 is a perspective view showing the external appearance of the POS terminal 1 and the imaging device 3. As shown in Figure 4, the POS terminal 1 has a main body 42 and a basket stand 4. The main body 42 houses the circuit board and power supply that control the POS terminal 1. The basket stand 4 is attached to one side of the main body 42 and is a stand on which baskets containing goods are placed.

[0036] Furthermore, the top surface of the main body 42 has a flat product placement section 41. The main body 42 is equipped with a pair of support columns 45 extending upward from the rear of the product placement section 41. A printer 22 for issuing receipts is attached to the support column 45 furthest from the basket placement base 4. A card reader 21 for reading information from cards such as credit cards and point cards is attached to the top of the printer 22.

[0037] Furthermore, an imaging device 3 is attached to the support column 45 on the side closer to the basket rest 4. The imaging device 3 is positioned with its reading surface facing the customer, and the first imaging unit 37 located inside the imaging device 3 captures an image of the product as the customer scans the product after taking it out of the basket. In addition, since customers tend to scan the product towards the imaging device 3 with the symbols attached to the product, the first imaging unit 37 often captures the symbols attached to the product at the same time. Hereafter, in this embodiment, the first imaging unit 37 will be assumed to capture the symbols.

[0038] Furthermore, the second imaging unit 38, which is part of the imaging device 3, is mounted above the imaging device 3. The reading surface of the second imaging unit 38 is positioned diagonally downward from the imaging device 3, and it images the scanned product from a different angle than the first imaging unit 37. In other words, the second imaging unit 38 images a different side of the product than the side imaged by the first imaging unit 37.

[0039] Furthermore, a display unit 18 is provided between the pair of support columns 45. The display unit 18 is positioned for the customer operating the POS terminal 1 and is, for example, a display unit made of liquid crystal. The display unit 18 is positioned slightly above the imaging device 3 and the printer 22, and at the same height as the card reader 21 and the second imaging unit 38.

[0040] The display unit 18 displays product information (product name, product price, etc.) for products that have been registered, and payment information (total amount, payment amount, change amount, etc.) for products that have been processed for payment. The display surface of the display unit 18 is equipped with an operation unit 17 consisting of a touch panel.

[0041] Furthermore, between the pair of support columns 45, below the display unit 18, there is a temporary storage platform 44 for temporarily placing products, and a pair of mounting parts 43 for attaching the handles of a shopping bag containing the products that have undergone product registration processing. Products placed in the shopping bag are then placed on the product placement section 41.

[0042] Additionally, a roughly cylindrical pole 6 extending upward is attached to the support column 45 on the side closer to the cage mounting platform 4. A warning light 61 is attached to the pole 6.

[0043] The warning light 61 lights up or flashes when an abnormality occurs at POS terminal 1 (for example, when the receipt paper runs out) or when a customer calls an attendant.

[0044] The edge device 5 is housed inside the main unit 42.

[0045] In this POS terminal 1, the customer takes items from a basket placed on the basket stand 4 and performs a product registration operation by passing (scanning) the symbols attached to the items in front of the imaging device 3 using the first imaging unit 37 and the second imaging unit 38. The customer then places the registered items into a shopping bag attached to the attachment unit 43. Based on the product identification information received from the imaging device 3, the POS terminal 1 performs a product registration process for the registered items. After the customer has performed the product registration operation for all items, they then perform a checkout operation by pressing the checkout button. When a checkout operation is performed, the POS terminal 1 performs a payment process for the registered items.

[0046] Next, the processing flow in system 10 will be explained. Figure 5 is a timing chart showing the processing flow in system 10. As shown in Figure 5, system 10 includes a POS terminal 1, an imaging device 3, an edge device 5, and a store server 7.

[0047] When a customer registers a product at the POS terminal 1, the imaging device 3 uses the first imaging unit 37 and the second imaging unit 38 to capture images of the customer's product registration operation (S1). Next, the imaging device 3 uses the object detection model 542 to extract a product image based on the captured image information (S2). In S2, the imaging device 3 extracts the product image using both the image captured by the first imaging unit 37 and the image captured by the second imaging unit 38. The imaging device 3 then outputs the extracted product image to the edge device 5 (S3).

[0048] The edge device 5, which receives the product image as input, performs general object recognition. Specifically, the edge device 5 uses an object detection model 542 to generate an object image representing the product by identifying the region of the product contained in the product image, uses a feature extraction model 543 to extract feature vectors that represent the characteristics of the product based on the object image, and uses a matching dictionary 544 to calculate the similarity of products to the imaged product and extract products that are similar to the product (high similarity) (S4). In S4, if the calculated similarity is above a threshold, the edge device 5 identifies one product. On the other hand, if the calculated similarity is below the threshold, the edge device 5 identifies multiple candidate products in descending order of similarity. The edge device 5 then outputs the product identification information of the identified product to the imaging device 3 (S5).

[0049] Meanwhile, the imaging device 3, which output the product image in S3, waits for the first imaging unit 37 or the second imaging unit 38 to capture a symbol in response to the customer's product registration operation. When a symbol is captured, the device decodes the symbol and generates decoded information including product identification information (S6). Depending on the timing of the symbol capture, the order of input of product identification information in S5 and generation of decoded information in S6 may be reversed. If the customer does not orient the symbol correctly towards the first imaging unit 37, the symbol capture will be delayed, and if the generation of decoded information is delayed, the input of product identification information from the edge device 5 may come first. Conversely, if the symbol is captured early, the generation of decoded information may come earlier than the input of product identification information from the edge device 5.

[0050] If the imaging device 3 has not generated decoded information at the time the product identification information is input from the edge device 5 (S5), it transmits the product identification information to the POS terminal 1 without waiting for the decoded information to be generated (S7). Conversely, if the decoded information has been generated before the product identification information is input from the edge device 5, the imaging device 3 transmits the product identification information contained in the decoded information to the POS terminal 1 without waiting for the product identification information to be input from the edge device 5 or for the product identification information to be input from the edge device 5 (S7).

[0051] POS terminal 1 performs product registration processing based on the received product identification information (S8). Furthermore, when the customer instructs POS terminal 1 to complete product registration, it performs payment processing (S9).

[0052] Furthermore, the imaging device 3 associates the product images extracted in S2 with the decoded information generated in S6 and generates association information (S10). The product images extracted in S2 include images captured by the first imaging unit 37 and images captured by the second imaging unit 38, and therefore include product images of the product on the side with a symbol (for example, images from frames 2 to 8 in Figure 2) captured by the first imaging unit 37, and product images of the product on the side without a symbol (captured by the second imaging unit 38). The imaging device 3 associates the decoded information with these product images and generates association information. The imaging device 3 then outputs the generated association information to the edge device 5 (S11). Note that the processing in S7 and the processing in S10 and S11 may be performed in any order.

[0053] The edge device 5, which has received the association information, selects and filters the input product images (S12). Because customers habitually tend to point the side with the symbol towards the first imaging unit 37 when registering a product, product images near the symbol tend to be collected. Collecting object images of the product taken from as many different angles as possible leads to correct product recognition (improving the accuracy of the dictionary), so the edge device 5 selects and filters product images so that product images near the symbol do not become excessively numerous compared to other images. The edge device 5 then sends the association information with the selected product images to the store server 7 (S13).

[0054] Upon receiving the association information, the store server 7 extracts an object image using the object detection model 742, calculates the product features contained in the object image using the feature extraction model 743 (see Figure 9), and additionally stores these features in the matching dictionary 744 (see Figure 9) in association with the same product identification information contained in the received decoded information (S14). The process in S14 allows the matching dictionary 744 to be easily trained on a daily basis. As a result, the accuracy of the matching dictionary 744 is improved.

[0055] Furthermore, if the product imaged in S1 is a new product, the product identification information included in the decoded information may not yet be stored in the matching dictionary 744. In this case, new related information is created in the matching dictionary 744, associating the product identification information and its features to identify the new product. This eliminates the need to separately associate the product identification information with the features of the new product. This process can be easily performed by performing package recognition (which involves both code recognition and general object recognition) (this process cannot be performed by simply performing general object recognition on the product).

[0056] Next, the store server 7 transmits the information of the trained matching dictionary 744 to the edge device 5 (S15). The edge device 5 overwrites the previous matching dictionary 544 with the information of the received matching dictionary 744 as the matching dictionary 544 (S16). In this way, the matching dictionary 544 can be easily trained on a daily basis. As a result, the accuracy of the matching dictionary 544 is improved.

[0057] Next, the hardware configuration of POS terminal 1 will be described. Figure 6 is a block diagram showing the hardware configuration of POS terminal 1. As shown in Figure 6, POS terminal 1 is equipped with a CPU (Central Processing Unit) 11, which is an example of a processor, ROM (Read Only Memory) 12, RAM (Random Access Memory) 13, memory unit 14, etc. The CPU 11 is the main control unit of POS terminal 1. ROM 12 stores various programs. RAM 13 loads programs and various data. Memory unit 14 stores various programs. The CPU 11, ROM 12, RAM 13, and memory unit 14 are connected to each other via a bus 15. The CPU 11, ROM 12, and RAM 13 constitute the control unit 100. That is, the control unit 100 executes the control processing of POS terminal 1, which will be described later, by operating according to the control programs stored in ROM 12 and memory unit 14 and loaded into RAM 13.

[0058] RAM13 has a product information unit 131. The product information unit 131 stores product information (product code that identifies the product, product name, price, etc.) of the product that has been processed for product registration.

[0059] The memory unit 14 is composed of non-volatile memory such as an HDD (Hard Disk Drive) or flash memory that retains stored information even when the power is turned off, and includes a control program unit 141 that stores a control program and a product master 142. The product master 142 stores product information for each product, associating it with product identification information that identifies the product.

[0060] Furthermore, the control unit 100 is connected to the operation unit 17, display unit 18, card reader 21, and printer 22 via the bus 15 and controller 16.

[0061] The operation unit 17 has a closing key 171. The closing key 171 is a key operated when proceeding from product registration processing to payment processing on the POS terminal 1. The display unit 18 is, for example, an LCD display and displays information to the customer, who is the operator of the POS terminal 1. The display unit 18 displays, for example, product information (product name, product price, etc.) of products that have been registered, and accounting information related to payment processing for each product. The operation unit 17 is a touch panel provided on the display unit 18.

[0062] The card reader 21 reads card information from a card, such as a credit card, debit card, or electronic money card, when processing a payment. The card reader 21 also reads and writes point information to a point card. The printer 22 is, for example, a thermal printer, and issues a receipt containing receipt information including product information and payment information related to the processed payment.

[0063] Furthermore, the control unit 100 is connected to the communication unit 23 via the bus 15. The communication unit 23 is connected to the edge device 5 and the store server 7 via the communication line L. The communication unit 23 is also connected to the imaging device 3 via the cable U.

[0064] Next, the hardware configuration of the imaging device 3 will be described. Figure 7 is a block diagram showing the hardware configuration of the imaging device 3. As shown in Figure 7, the imaging device 3 includes a CPU 31, which is an example of a processor, a ROM 32, a RAM 33, a memory unit 34, etc. The CPU 31 is the main control unit of the imaging device 3. The ROM 32 stores various programs. The RAM 33 loads programs and various data. The memory unit 34 stores various programs. The CPU 31, ROM 32, RAM 33, and memory unit 34 are connected to each other via a bus 35. The CPU 31, ROM 32, and RAM 33 constitute the control unit 300. That is, the control unit 300 executes the control processing of the imaging device 3, which will be described later, by operating according to the control programs stored in the ROM 32 and memory unit 34 and loaded into the RAM 33.

[0065] RAM33 includes an image storage unit 331, a product image unit 332, and a decode information unit 333. The image storage unit 331 stores images captured by the first imaging unit 37 and images captured by the second imaging unit 38. The product image unit 332 stores product images extracted from the images stored in the image storage unit 331. The decode information unit 333 stores decode information obtained by decoding symbols contained in the images captured by the first imaging unit 37.

[0066] The memory unit 34 is composed of non-volatile memory such as an HDD or flash memory that retains stored information even when the power is turned off, and includes a control program unit 341 and an object detection model 542. The control program unit 341 stores a control program for driving the imaging device 3.

[0067] Furthermore, the control unit 300 is connected to the first imaging unit 37 and the second imaging unit 38 via the bus 35 and the controller 36.

[0068] Furthermore, the control unit 300 is connected to the communication unit 39 via the bus 35. The communication unit 39 is connected to the edge device 5 and the store server 7 via the communication line L. The communication unit 39 is also connected to the POS terminal 1 via the cable U.

[0069] Next, the hardware configuration of the edge device 5 will be described. Figure 8 is a block diagram showing the hardware configuration of the edge device 5. As shown in Figure 8, the edge device 5 is equipped with a CPU 51, which is an example of a processor, ROM 52, RAM 53, memory unit 54, etc. The CPU 51 is the main control unit of the edge device 5. ROM 52 stores various programs. RAM 53 loads programs and various data. Memory unit 54 stores various programs. The CPU 51, ROM 52, RAM 53, and memory unit 54 are connected to each other via a bus 55. The CPU 51, ROM 52, and RAM 53 constitute the control unit 500. That is, the control unit 500 executes the control processing of the edge device 5, which will be described later, by operating according to the control programs stored in ROM 52 and memory unit 54 and loaded into RAM 53.

[0070] The RAM 53 includes an object image storage unit 531, a feature quantity unit 532, a product identification information unit 533, and an association information unit 534. The object image storage unit 531 stores object images extracted based on product images received from the imaging device 3. The feature quantity unit 532 stores feature vectors representing the features of the product, calculated by the feature extraction model 543 based on the object images stored in the object image storage unit 531. The product identification information unit 533 compares the feature vector calculated by the feature extraction model 543 with the product features stored in the matching dictionary 544 to calculate the similarity of products that are close to the feature vector, and stores product identification information for products with high similarity. If the calculated similarity is above a threshold, it stores product identification information for the single product with the highest similarity. If the highest calculated similarity is below the threshold, it stores product identification information for multiple candidate products in descending order of similarity. The association information unit 534 stores association information received from the imaging device 3. Furthermore, the association information unit 534 stores the association information generated by the edge device 5, although this will be described in more detail later.

[0071] The memory unit 54 is composed of non-volatile memory such as an HDD or flash memory that retains stored information even when the power is turned off, and includes a control program unit 541, an object detection model 542, a feature extraction model 543, and a matching dictionary 544. The control program unit 541 stores a control program for driving the edge device 5.

[0072] The control unit 500 is also connected to the bus 55 and the controller 56. The control unit 500 is also connected to the communication unit 357 via the bus 55. The communication unit 57 is connected to the POS terminal 1, the imaging device 3, and the store server 7 via the communication line L, enabling communication between them.

[0073] Next, the hardware configuration of the store server 7 will be described. Figure 9 is a block diagram showing the hardware configuration of the store server 7. As shown in Figure 9, the store server 7 is equipped with a CPU 71, which is an example of a processor, ROM 72, RAM 73, memory unit 74, etc. The CPU 71 is the main control unit of the store server 7. ROM 72 stores various programs. RAM 73 loads programs and various data. Memory unit 74 stores various programs. The CPU 71, ROM 72, RAM 73, and memory unit 74 are connected to each other via a bus 75. The CPU 71, ROM 72, and RAM 73 constitute the control unit 700. That is, the control unit 700 executes the control processing of the store server 7, which will be described later, by operating according to the control programs stored in ROM 72 and memory unit 74 and loaded into RAM 73.

[0074] RAM73 has a sales information unit 731 and an association information unit 732. The sales information unit 731 stores sales data received from POS terminal 1 and manages the sales of store T. The association information unit 732 stores association information received from edge device 5.

[0075] The memory unit 74 is composed of non-volatile memory such as an HDD or flash memory that retains stored information even when the power is turned off, and includes a control program unit 741, an object detection model 742, a feature extraction model 743, and a matching dictionary 744. The control program unit 741 stores a control program for driving the store server 7. The object detection model 742 is a dictionary with the same contents as the object detection model 542, the feature extraction model 743 is a dictionary with the same contents as the feature extraction model 543, and the matching dictionary 744 is a dictionary with the same contents as the matching dictionary 544.

[0076] Furthermore, the control unit 700 is connected to the display unit 77 and the operation unit 78 via the bus 55 and the controller 56. The display unit 77 is, for example, a liquid crystal display and displays information to the operator who operates the store server 7. The operation unit 78 is, for example, a keyboard and is operated by the operator of the store server 7.

[0077] Furthermore, the control unit 700 is connected to the communication unit 79 via the bus 75. The communication unit 79 is connected to the POS terminal 1, the imaging device 3, and the edge device 5 via the communication line L, enabling communication between them.

[0078] Next, we will explain the control of POS terminal 1. Figure 10 is a flowchart showing the flow of control processing for POS terminal 1. As shown in Figure 10, the control unit 100 of POS terminal 1 determines whether the customer has initiated a transaction (S21). For example, if the customer presses the transaction initiation button (e.g., the start button), the control unit 100 determines that the transaction initiation has been initiated. After waiting until the transaction initiation is initiated (No in S21), if the control unit 100 determines that the customer has initiated a transaction (Yes in S21), it sends a signal to the imaging device 3 to start imaging by the first imaging unit 37 and the second imaging unit 38 (S22).

[0079] Next, the control unit 100 determines whether it has received product identification information from the imaging device 3 (S23). If it determines that product identification information has been received (Yes in S23), the control unit 100 determines whether there is only one product identification piece that has been received (S24). If it determines that there is only one product identification piece that has been received (Yes in S24), the control unit 100 uses that product identification piece to read the product information for the product identified by that product identification piece from the product master 142 and executes the product registration process (S25).

[0080] Furthermore, if the control unit 100 determines that there are multiple product identification pieces (No. in S24), it reads information on multiple products identified by the received multiple product identification pieces (for example, product name and product image) from the product master 142 and displays them on the display unit 18 as candidate products (S26). The customer operates the operation unit 17 to select one product from the displayed candidate products. The control unit 100 determines whether one product has been selected (S27). The control unit 100 waits until a product is selected (No. in S27), and if it determines that a product has been selected (Yes in S27), the control unit 100 reads the product information from the product master 142 for the selected product and executes the product registration process (S25).

[0081] If it is determined that product identification information has not been received (No. in S23) and if product registration processing is performed in S25, the control unit 100 then determines whether the locking key 171 has been operated (S28). If it is determined that the locking key 171 has not been operated (No. in S28), the control unit 100 returns to S23. If it is determined that the locking key 171 has been operated (Yes in S28), the control unit 100 performs settlement processing based on the product information stored in the product information unit 131 (S29). Next, the control unit 100 transmits the sales data related to the settled product to the store server 7 (S30). Then the control unit 100 transmits an imaging completion signal to the imaging device 3 (S31). Then the control unit 100 terminates processing.

[0082] Next, we will explain the functional configuration of the imaging device 3. Figure 11 is a functional block diagram showing the functional configuration of the imaging device 3. The control unit 300 of the imaging device 3 functions as an extraction means 301, a decoding means 302, an association means 303, an output means 304, and a transmission means 305, according to the control program stored in the ROM 32 and the control program unit 341 of the memory unit 34.

[0083] The extraction means 301 extracts product images containing products from images captured by the imaging units (first imaging unit 37 and second imaging unit 38). Specifically, the extraction means 301 extracts product images containing products from images captured by the first imaging unit 37 and second imaging unit 38 that are stored in the image storage unit 331.

[0084] The decoding means 302 generates decoded information by decoding the symbols attached to the product contained in the image captured by the imaging unit. Specifically, the decoding means 302 generates decoded information by decoding the images of the symbols contained in the product image stored in the product image unit 332. Alternatively, the decoding means 302 may generate decoded information by decoding the images of the symbols contained in the image stored in the image storage unit 331.

[0085] The association means 303 associates the image captured by the imaging unit with the decoded information. In this embodiment, the association means 303 associates the product image stored in the product image unit 332 extracted by the extraction means 301 with the decoded information stored in the decoded information unit 333 generated by the decoded means 302. That is, the association means 303 generates association information that associates the product image stored in the product image unit 332 extracted by the extraction means 301 with the decoded information stored in the decoded information unit 333 generated by the decoded means 302.

[0086] The association means 303 may associate the decoded information with all product images stored in the product image unit 332, but the association means 303 may, for example, associate a predetermined number of product images before and after the image containing the symbol decoded by the decoding means 302. Alternatively, the association means 303 may, for example, associate a predetermined number of product images before the image containing the symbol decoded by the decoding means 302. Customers may change the angle of the product in various ways in an attempt to have the imaging unit (especially the first imaging unit 37) read the symbol attached to the product. Therefore, by associating a predetermined number of product images before the image in which the symbol was read, it is possible to associate the product with product images taken from various angles. Alternatively, the association means 303 may, for example, associate a predetermined number of product images after the image containing the symbol decoded by the decoding means 302.

[0087] The output means 304 outputs the association information associated by the association means 303 to the edge device 5. Alternatively, the output means 304 may also output the associated information to the store server 7.

[0088] The transmission means 305 transmits to the POS terminal 1 either product identification information contained in the decoded information, or product identification information identified by general object recognition performed on images captured by the first imaging unit 37 and the second imaging unit 38. Specifically, the transmission means 305 transmits to the POS terminal 1 either product identification information contained in the decoded information generated by the decoded means 302, or product identification information identified and input by general object recognition performed on product images extracted by the extraction means 301 by the edge device 5.

[0089] Next, the control of the imaging device 3 will be explained. Figure 12 is a flowchart showing the flow of the control process for the imaging device 3. As shown in Figure 12, the control unit 300 of the imaging device 3 determines whether it has received an imaging start signal from the POS terminal 1 (S41). It waits until it receives an imaging start signal (No in S41), and if it determines that it has received an imaging start signal from the POS terminal 1 (Yes in S41), the control unit 300 activates the first imaging unit 37 and the second imaging unit 38 and starts imaging (S42).

[0090] Next, the control unit 300 determines whether it has received an image capture completion signal from the POS terminal 1 (S43). If it determines that it has not received an image capture completion signal (No. in S43), the control unit 300 stores the images captured by the first imaging unit 37 and the second imaging unit 38 in the image storage unit 331 (S44). The extraction means 301 then extracts product images containing products from the images stored in the image storage unit 331 (S45). The control unit 300 then stores the extracted product images in the product image unit 332. Finally, the control unit 300 transmits the product images stored in the product image unit 332 to the edge device 5 (S46).

[0091] Next, the control unit 300 determines whether it has detected a symbol from the product image stored in the product image unit 332 (S51). If it determines that a symbol has been detected (Yes in S51), the decoding means 302 decodes the detected symbol and generates decoded information (S52). The control unit 300 then stores the generated decoded information in the decoded information unit 333 (S52).

[0092] Next, the association means 303 generates association information by associating the decoded information stored in the decoded information unit 333 with the product image stored in the product image unit 332 (S53). Then the output means 304 outputs the generated association information to the edge device 5 (S54). Then the control unit 300 returns to S43.

[0093] Furthermore, if in S51 it is determined that no symbol has been detected (No. in S51), the control unit 300 determines whether product identification information has been input from the edge device 5 (S61). If it is determined that no product identification information has been input from the edge device 5 (No. in S61), the control unit 300 returns to S51. If it is determined that product identification information has been input from the edge device 5 (Yes in S61), the control unit 300 then determines whether there is only one product identification piece of information input (S62). If it is determined that there is only one product identification piece of information (Yes in S62), it determines in S52 whether the symbol has already been decoded (S63). If it is determined that it has already been decoded (Yes in S63), the transmission means 305 transmits the product identification information contained in the decoded information to the POS terminal 1 (S64). Then the control unit 300 returns to S43. If it is determined that it has not been decoded (No. in S63), the transmission means 305 transmits the product identification information received in S62 to the POS terminal 1 (S65). Then the control unit 300 returns to S43.

[0094] Furthermore, if in S62 the control unit 300 determines that there are multiple product identification pieces of information entered (No. in S62), it determines whether the symbols have already been decoded in S52 (S66). If it determines that they have already been decoded (Yes in S66), the transmission means 305 transmits the product identification pieces of information contained in the decoded information to the POS terminal 1 (S67). The control unit 300 then returns to S43. If it determines that they have not been decoded (No. in S66), the transmission means 305 transmits the multiple types of product identification pieces of information (multiple product identification pieces of information relating to candidate products) received in S62 to the POS terminal 1 (S68). The control unit 300 then returns to S43.

[0095] In the example in Figure 12, if the answer to S63 is Yes, the product identification information contained in the decoded information is sent to the POS terminal 1 at this point. However, the product identification information contained in the decoded information may be sent to the POS terminal 1 after any of the processes in S52 to S54. In this case, if the answer to S63 is Yes, the control unit 500 returns to S43. The same applies if the answer to S66 is Yes.

[0096] Furthermore, if in S43 the control unit 300 determines that it has received an imaging completion signal from the POS terminal 1 (Yes in S43), the control unit 300 terminates imaging by the first imaging unit 37 and the second imaging unit 38 (S69). The control unit 300 then terminates processing.

[0097] Next, the control of the edge device 5 will be described. Figure 13 is a flowchart showing the flow of the control process for the edge device 5. As shown in Figure 13, the control unit 500 of the edge device 5 determines whether it has received a product image from the imaging device 3 (S71). If it determines that it has received a product image from the imaging device 3 (Yes in S71), the control unit 500 uses the object detection model 542 to identify the product area in the received product image (S72). Next, the control unit 500 stores the object image surrounded by the identified area in the object image storage unit 531 (S73).

[0098] Next, the control unit 500 uses the feature extraction model 543 to calculate the feature vector of the object image stored in the object image storage unit 531 (S74). The control unit 500 then stores the calculated feature vector in the feature quantity unit 532 (S75). Next, the control unit 500 uses the matching dictionary 544 to compare the feature vector stored in the feature quantity unit 532 with the feature quantities stored in the matching dictionary 544 for each product identification information, and extracts the product identification information of products with high similarity (S76). The product identification information extracted in S76 is, for example, the single product identification information with the highest similarity if there are products with a similarity of 95% or more, and if all products have a similarity of less than 95%, it is the product identification information of multiple (for example, 5) candidate products in order of their similarity. The control unit 500 then stores the extracted product identification information in the product identification information unit 533. The control unit 500 then transmits the product identification information stored in the product identification information unit 533 to the imaging device 3 (S77). Then the control unit 500 returns to S71.

[0099] The control unit 500 may also generate association information by associating the product identification information stored in the product identification information unit 533 with the object image stored in the object image storage unit 531 in S73, and transmit this information to the store server 7. In this way, the store server 7 can collect more images and train the matching dictionary 744.

[0100] Furthermore, if it is determined in S71 that the product image has not been received from the imaging device 3 (No in S71), the control unit 500 determines whether it has received association information from the imaging device 3 (S81). If it is determined that it has received association information from the imaging device 3 (Yes in S81), the control unit 500 stores the received association information in the association information unit 534 (S82). The control unit 500 then transmits the association information stored in the association information unit 534 to the store server 7 (S83). The control unit 500 then returns to S71.

[0101] Furthermore, if it is determined that the reception of association information is not from the imaging device 3 (No in S81), the control unit 500 determines whether it has received information for the matching dictionary 744 from the store server 7 (S84). The information for the matching dictionary 744 received in S84 is the matching dictionary that has been trained by the store server 7 by additionally storing association information. If it is determined that information for the matching dictionary 744 has been received from the store server 7 (Yes in S84), the control unit 500 overwrites the matching dictionary 544 with the received information for the matching dictionary 744 and stores it (S88). In this way, the matching dictionary 544 is trained, and the accuracy of the matching dictionary 544 when calculating similarity is improved. Then the control unit 500 returns to S71.

[0102] Next, the control of the store server 7 will be explained. Figure 14 is a flowchart showing the flow of the control process for the store server 7. As shown in Figure 14, the control unit 700 of the store server 7 determines whether it has received sales data from the POS terminal 1 (S91). If it determines that it has received sales data from the POS terminal 1 (Yes in S91), the control unit 700 adds the received sales data to the sales information unit 731 and stores it (S92), and manages the sales of store T. Then the control unit 700 returns to S91.

[0103] Furthermore, if it is determined that the data received is not from POS terminal 1 (No. in S91), the control unit 700 determines whether it has received association information from edge device 5 (S93). If it is determined that it has not received association information (No. in S93), the control unit 700 returns to S91. If it is determined that it has received association information from edge device 5 (Yes in S93), the control unit 700 stores the received association information in the association information unit 732 (S94).

[0104] Next, the control unit 700 updates the matching dictionary 744 based on the association information stored in the association information unit 732 (S95). That is, the control unit 700 performs general object recognition on the product image (which may also be an object image) associated with the product identification information as association information. Specifically, the control unit 700 uses the object detection model 742 to identify the region of the product image and uses the feature extraction model 743 to extract feature quantities that represent the features of the object image of the product whose region has been identified. The control unit 700 then associates the extracted feature quantities with the same product identification information stored in the matching dictionary 744 that is associated with the product image, and additionally stores them. In this way, by adding feature quantities associated with the product identification information to the matching dictionary 744 and training the matching dictionary 744, the accuracy of the matching dictionary 744 is improved.

[0105] The control unit 700 then transmits the learned information from the matching dictionary 744 to the edge device 5 (S96). The control unit 700 then returns to S91. The edge device 5, having received the information from the matching dictionary 744 from the store server 7, updates the matching dictionary 544 by overwriting it with the received information from the matching dictionary 744. In other words, the matching dictionary 544 is learned in the same way as the matching dictionary 744.

[0106] As described above, the imaging device 3 of the embodiment comprises a first imaging unit 37 and a second imaging unit 38, a decoding means 302 that generates decoded information by decoding symbols attached to products included in the image captured by the first imaging unit 37, an association means 303 that associates the images captured by the first imaging unit 37 and the second imaging unit 38 with the decoded information, an output means 304 that outputs the associated association information, and a transmission means 305 that transmits to the POS terminal 1 either product identification information included in the decoded information or product identification information identified by general object recognition performed on the images captured by the first imaging unit 37 and the second imaging unit 38.

[0107] In this embodiment, the imaging device 3 outputs the decoded symbol information based on the image and symbol captured by the customer's product registration operation, thus eliminating the need to perform a separate operation to associate the decoded information with the image in order to train the matching dictionary 544, making it easy to train the matching dictionary 544.

[0108] Although embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents.

[0109] For example, in this embodiment, the edge device 5 was described as a standalone device. However, the invention is not limited to this, and the edge device 5 may be incorporated into, for example, the imaging device 3, the POS terminal 1, or the store server 7. In other words, the imaging device 3, the POS terminal 1, or the store server 7 may have the functions of the edge device 5.

[0110] Furthermore, in this embodiment, the POS terminal 1 was described as a device operated by the customer themselves. However, the POS terminal 1 is not limited to this, and may be a device operated by a store employee of store T. In this case, the imaging device 3 associates the product image with the decoded information based on the product registration operation performed by the store employee.

[0111] In this embodiment, the product image extracted by the extraction means 301 from the images captured by the first imaging unit 37 and the second imaging unit 38 was associated with the decoded information. However, the invention is not limited to this, and for example, the images captured by the first imaging unit 37 and the second imaging unit 38 themselves may be associated with the decoded information.

[0112] Furthermore, in this embodiment, two imaging units are provided: a first imaging unit 37 and a second imaging unit 38. However, the system is not limited to this, and either the first imaging unit 37 or the second imaging unit 38 may be provided as the imaging unit. Also, the imaging unit may have one or more individual imaging units. The more imaging units are arranged in different positions, the more comprehensively the product can be imaged.

[0113] In addition, the association means 303 described in the embodiment as part of the configuration of the imaging device 3 may also be provided by an edge device 5 or a store server 7 (the edge device 5 or store server 7 are collectively referred to as the "information processing device"). In the other embodiment, the imaging device 3 transmits the image captured by the imaging unit and the decoded information decoded by the decoding means to the information processing device. The information processing device receives the image and decoded information from the imaging device 3 and executes the association means to associate the image and the decoded information. If the information processing device is an edge device 5, it transmits the associated information to the store server 7, and the store server 7 trains the matching dictionary 744 based on the associated information received from the edge device 5. The store server 7 then transmits the trained matching dictionary 744 to the edge device 5. If the information processing device is a store server 7, it trains the matching dictionary 744 based on the associated information and then transmits the trained matching dictionary 744 to the edge device 5.

[0114] The program executed by the imaging device 3 of this embodiment is provided as an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk).

[0115] Furthermore, the program executed by the imaging device 3 of the embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Alternatively, the program executed by the imaging device 3 of the embodiment may be provided or distributed via a network such as the Internet.

[0116] Furthermore, the program executed by the imaging device 3 of the embodiment may be pre-installed and provided in ROM or the like. [Explanation of symbols]

[0117] 1 POS terminal 3. Imaging device 5 Edge devices 7 Store Servers 10 Systems 11 CPU 31 CPU 37 First Imaging Unit 38. Second Imaging Unit 51 CPU 71 CPU 100 Control Unit 131 Product Information Department 142 Product Master 300 Control Unit 301 Extraction means 302 Decoding means 303 Related means 304 Output means 305 Transmission means 331 Image storage unit 332 Product Image Section 333 Decode Information Department 500 Control Unit 531 Object Image Storage Unit 532 Feature section 533 Product Specific Information Department 534 Related Information Department 542 Object Detection Models 543 Feature Extraction Models 544 Dictionary for comparison 700 Control Unit 731 Sales Information Department 732 Related Information Department 742 Object Detection Models 743 Feature Extraction Models 744 Dictionary for comparison [Prior art documents] [Patent Documents]

[0118] [Patent Document 1] Japanese Patent Publication No. 2014-21915

Claims

1. Imaging unit, A storage means for storing multiple images captured by the imaging unit, A decoding means that generates decoded information by decoding the symbols attached to the products contained in the image stored by the storage means, The aforementioned storage means associates the decoded information with an object image containing a product enclosed in a specified region from among a plurality of images stored by the storage means, An output means that outputs association information associated by the association means, A transmission means for transmitting either product identification information contained in the decoded information or product identification information identified by general object recognition performed on an image captured by the imaging unit to a product sales data processing device, An imaging device equipped with [a specific feature].

2. The system further comprises an extraction means for extracting the object image from the image captured by the imaging unit, The association means associates the object image with the decoded information. The imaging apparatus according to claim 1.

3. The extraction means extracts an image within a predetermined range that includes the image of the symbol. The imaging apparatus according to claim 2.

4. The extraction means extracts object images of a predetermined number of frames before and after the image of the symbol, or object images of a predetermined number of frames before the image containing the symbol, or object images of a predetermined number of frames after the image containing the symbol. The imaging apparatus according to claim 2.

5. The imaging unit further comprises a first imaging unit and a second imaging unit installed at a different position and angle from the first imaging unit. The extraction means further extracts the object image from the image captured by the first imaging unit and the image captured by the second imaging unit. The association means associates the object image extracted from the image captured by the first imaging unit and the second imaging unit with the decoded information. The imaging apparatus according to any one of claims 2 to 4.

6. A computer as an imaging device equipped with an imaging unit, A storage means for storing multiple images captured by the imaging unit, A decoding means that generates decoded information by decoding the symbols attached to the products contained in the image stored by the storage means, The aforementioned storage means associates the decoded information with an object image containing a product enclosed in a specified region from among a plurality of images stored by the storage means, An output means that outputs association information associated by the association means, A transmission means for transmitting either product identification information contained in the decoded information or product identification information identified by general object recognition performed on an image captured by the imaging unit to a product sales data processing device, A program to make it work.

Images