Imaging device and program
The imaging device automates product dictionary learning in POS systems by decoding symbols and associating images, enhancing accuracy and reducing manual effort in product identification.
Patent Information
- Application Number
- JP2023076450
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-08
- Publication Date
- 2024-11-20
- Estimated Expiration
- 2043-05-08
AI Technical Summary
Existing POS systems face challenges in accurately determining products from captured images due to the need for manual image collection to update product dictionaries, which is burdensome for operators.
An imaging device and program that includes an imaging unit, decoding unit, associating means, output means, and transmitting means to automatically generate and associate product information, enabling easy dictionary learning through symbol decoding and image analysis.
Facilitates efficient and accurate product identification by automating the dictionary learning process, reducing operator burden and improving recognition accuracy through symbol decoding and image analysis.
Smart Images

Figure 2024161642000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an imaging device and a program. [Background technology]
[0002] Recently, POS systems that perform product registration and payment processing for products sold at stores such as shopping centers, mass retailers, department stores, etc. Some of these POS systems identify products by recognizing the package of the product in an image captured by an imaging device.
[0003] Incidentally, in order to correctly identify a product from a captured image, it is necessary to improve the accuracy of the dictionary used to estimate the product from the image. In order to improve the accuracy of the dictionary, it is necessary to collect many images related to the product and have the dictionary learn them. For example, new products may be released, and the design of the package or the like of the same product may change depending on the season, so it is necessary to learn the dictionary on a daily basis.
[0004] However, in order to learn the dictionary, a worker must manually take pictures of the products, which places a burden on the worker. Summary of the Invention [Problem to be solved by the invention]
[0005] An object of the present invention is to provide an imaging device and a program that allow a dictionary to be easily learned. [Means for solving the problem]
[0006] The imaging device of the embodiment includes an imaging unit, a decoding means for generating decoded information by decoding a symbol attached to a product included in an image captured by the imaging unit, an association means for associating the image captured by the imaging unit with the decoded information, an output means for outputting the association information associated by the association means, and a transmission means for transmitting either product identification information included in the decoded information or product identification information identified by general object recognition performed on the image captured by the imaging unit to a product sales data processing device. [Brief description of the drawings]
[0007] [Figure 1] FIG. 1 is a diagram showing a system according to an embodiment. [Diagram 2] FIG. 2 is an explanatory diagram showing the principle of package recognition. [Diagram 3] FIG. 3 is a diagram showing an example in which the object detection model extracts a product image. [Figure 4] FIG. 4 is a perspective view showing the appearance of the POS terminal and the imaging device. [Diagram 5] FIG. 5 is a timing chart showing the flow of processing in the system. [Figure 6] FIG. 6 is a block diagram showing the hardware configuration of the POS terminal. [Figure 7] FIG. 7 is a block diagram showing the hardware configuration of the imaging apparatus. [Figure 8] FIG. 8 is a block diagram illustrating a hardware configuration of the edge device. [Figure 9] FIG. 9 is a block diagram showing a hardware configuration of the store server. [Figure 10] FIG. 10 is a flowchart showing the flow of the control process of the POS terminal. [Figure 11] FIG. 11 is a functional block diagram showing the functional configuration of the imaging device. [Figure 12] FIG. 12 is a flowchart showing the flow of control processing of the imaging device. [Figure 13]FIG. 13 is a flowchart showing the flow of the control process of the edge device. [Figure 14] FIG. 14 is a flowchart showing the flow of control processing of the store server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the embodiment, an imaging device that captures an image of a customer scanning a product in a system in which the customer performs registration and accounting operations will be described as an example. Note that the present invention is not limited to the first embodiment described below.
[0009] FIG. 1 is a diagram showing a system according to an embodiment. In FIG. 1, a system 10 is installed in a store T such as a supermarket, a mass retailer, a convenience store, or a specialty store. The system 10 is installed in the store T and includes one or more POS (Point of Sales) terminals 1 (product sales data processing devices), an imaging device 3, an edge device 5, and a store server 7, which are connected to each POS terminal 1, for example. Each POS terminal 1, the edge device 5, and the store server 7 are connected to a communication line L (for example, a LAN (Local Area Network)) that allows mutual communication. Furthermore, each imaging device 3 is connected to each POS terminal 1, and is connected to the POS terminal 1, for example, by a USB (Universal Serial Bus) cable U. Furthermore, each imaging device 3 is connected to the edge device 5 and the store server 7 by the communication line L.
[0010] The system 10 performs product registration processing, payment processing, and sales management for products sold in the store T. The system 10 also performs product registration processing using package recognition technology.
[0011] Package recognition is a technology that identifies products by combining a technology to identify products by imaging and analyzing symbols such as barcodes and two-dimensional codes attached to the product (hereinafter referred to as "code recognition") with a well-known general object recognition technology that recognizes products based on the shape and appearance of an image of the product.
[0012] General object recognition is described, for example, in "Yanai Keiji, 'The Current State and Future of General Object Recognition,' Transactions of the Information Processing Society of Japan, Vol. 48, No. SIG16 [searched August 10, 2010], Internet <URL: http: / / mm.cs.uec.ac.jp / IPSJ-TCVIM-Yanai.pdf>".
[0013] In code recognition, when an image of a symbol attached to a product is captured, the symbol is decoded and analyzed to obtain decoded information. The decoded information includes product identification information that identifies the product to which the symbol is attached. The product identification information is, for example, a product code that indicates the product, but it does not have to be a product code as long as it is information that identifies the product.
[0014] Next, the generic object recognition will be described. Fig. 2 is a diagram for explaining the generic object recognition. As shown in Fig. 2, the generic object recognition identifies an imaged product using three types of dictionaries, namely, an object detection model 542, a feature extraction model 543, and a matching dictionary 544.
[0015] The object detection model 542 is a dictionary for identifying an area (range) of an object (product) captured in a captured image. That is, the object detection model 542 is a dictionary for identifying an area in which an object (product) is captured from a captured image based on the input captured image. In other words, the object detection model 542 is a dictionary for extracting an image of a product (hereinafter referred to as an "object image") surrounded by an identified area from a captured image (an image in which a customer scans the product, as described later).
[0016] The feature extraction model 543 is a dictionary for calculating feature amounts effective for identifying a product shown in a specified area (object image). That is, the feature extraction model 543 outputs a feature vector indicating feature amounts indicating the features of a product whose area is specified, based on the input object image.
[0017] The matching dictionary 544 is a dictionary for identifying a product based on the output feature vector. In other words, the matching dictionary 544 stores a large number of products in association with feature amounts indicating the characteristics of the products according to product identification information that identifies each product, and calculates a similarity indicating how similar the large number of stored products are to the captured product by comparing the feature amounts indicated by the input feature vector with the stored feature amounts. The higher the similarity of a product, the higher the degree to which it resembles the captured product (i.e., the probability that it is the captured product). For example, a product with a similarity of 95% or more is highly likely to be the same product as the captured product.
[0018] In the generic object recognition, an object detection model 542 is used to identify an area in which a product is captured from an image captured by an imaging unit such as a camera, and then a feature extraction model 543 is used to extract the feature of the product, which represents the surface information of the product captured in the area (object image) such as the external shape, color, pattern, and unevenness, as parameters ("feature vector" shown in FIG. 2), and then a matching dictionary 544 is used to match the extracted feature with feature data of a large number of products stored in the dictionary, to calculate a similarity indicating how similar the stored product is to the product, and to recognize the product according to the similarity. Based on the similarity calculated by such generic object recognition, if there is one product whose similarity is equal to or greater than a threshold (e.g., 95% or greater), the system 10 identifies the recognized product as the captured product (product A in FIG. 2) and, if there is no product whose similarity is higher than the threshold (e.g., all similarities are lower than 95%), a single product is not identified and multiple recognized products with high similarity are selected as candidate products.
[0019] The package recognition uses code recognition technology and general object recognition technology to identify a product based on the technology that can identify the product faster, for example, and transmits (outputs) information about the identified product to the POS terminal 1, etc. However, if a product cannot be identified by general object recognition and code recognition has not yet been performed, the package recognition may further wait for a predetermined time until the product is identified by code recognition.
[0020] The imaging device 3 includes a first imaging section 37 (imaging section) and a second imaging section 38 (imaging section) (see FIG. 7), and is a device that captures an image of a commodity scanned in front of the imaging device 3. The first imaging section 37 and the second imaging section 38 are, for example, cameras in which a large number of CCDs (Charge Coupled Devices: imaging elements) are arranged in a two-dimensional form (planar shape).
[0021] The imaging device 3 extracts product images showing products from images captured by the first imaging unit 37 and the second imaging unit 38. FIG. 3 shows an example of extracting product images from captured images. In FIG. 3, images of frames 1 to 9 are captured. In FIG. 3, frame 1 shows nothing, but in reality, the customer's hand for scanning the product and the background are shown, and the product is not shown. The same is true for frame 9. In contrast, frames 2 to 8 show the product in addition to the customer's hand and the background. Therefore, the imaging device 3 uses a known technique to extract images of frames 2 to 8 showing the product from images of frames 1 to 9 as product images.
[0022] As a method for extracting a product image from a captured image, for example, there is a method for extracting an image in which a product appears by distinguishing between the product and a hand or background, but there are also other methods, such as a method for extracting images in which a product appears before and after an image in which a symbol is recognized as the center, a method for extracting images in which a product appears by going back from an image in which a symbol is recognized, a method for extracting images in which a product appears after an image in which a symbol is recognized, etc. Also, a dictionary similar to the object detection model 542 may be provided in the imaging device 3, and an object image in which the area of the product can be calculated using the dictionary may be extracted as the product image.
[0023] Furthermore, the imaging device 3 executes code recognition based on the images captured by the first imaging unit 37 and the second imaging unit 38, and generates decoded information. The imaging device 3 then generates association information that associates the generated decoded information with a product image of a product to which a symbol obtained by decoded the decoded information is attached (for example, all product images (several consecutive still images) from the start of detection of the product to which the symbol is attached to the end of detection). The imaging device 3 then outputs the association information to the edge device 5.
[0024] The edge device 5 executes general object recognition. That is, the edge device 5 detects an object image in which the area (range) of the product is specified from the images captured by the first imaging unit 37 and the second imaging unit 38 using the object detection model 542 (A in FIG. 2).
[0025] Next, the edge device 5 uses the feature extraction model 543 to calculate a feature amount indicating the features of the product from the detected object image, and outputs a feature vector of the product (B in FIG. 2). Then, the edge device 5 uses the matching dictionary 544 to calculate a similarity indicating the degree of similarity between the stored product and the product in question, based on the output feature vector. Then, the edge device 5 recognizes a product with a high similarity as the product that has been imaged. Then, the edge device 5 outputs the recognition result to the imaging device 3.
[0026] The edge device 5 also receives and accumulates the association information output by the imaging device 3. The edge device 5 then transmits the accumulated association information to the store server 7 via the communication line L. The edge device 5 then receives and stores a matching dictionary 544 learned based on the transmitted association information from the store server 7. The edge device 5 improves the accuracy of the stored matching dictionary 544 by overwriting and storing information on the matching dictionary received from the store server 7.
[0027] The imaging device 3 determines a product for which product identification information is to be transmitted to the POS terminal 1, based on the recognition result input from the edge device 5 and the generated decoded information. For example, when the product identification information input from the edge device 5 is information indicating one specific product (i.e., when the product identification information relating to one identified product is input from the edge device 5), if the decoded information has not yet been generated, the imaging device 3 transmits the product identification information to the POS terminal 1. If the decoded information has already been generated, the imaging device 3 transmits the product identification information included in the decoded information to the POS terminal 1.
[0028] On the other hand, when the product identification information input from the edge device 5 is information indicating multiple products (i.e., when multiple product identification information is input from the edge device 5), if decoded information has already been generated, the imaging device 3 transmits the product identification information indicated by the decoded information to the POS terminal 1. If the decoded information has not yet been generated, the imaging device 3 waits for the decoded information to be generated for a predetermined time, and when the decoded information is generated, transmits the product identification information indicated by the decoded information to the POS terminal 1. If the decoded information is not generated even after the predetermined time has elapsed, the imaging device 3 transmits the multiple product identification information input from the edge device 5 to the POS terminal 1. The POS terminal 1 displays the candidate products selectable based on the received product identification information of the multiple candidate products.
[0029] One or more POS terminals 1 (three in the first embodiment) are installed in the checkout area in store T. The POS terminal 1 is a product sales data processing device that allows customers to perform product registration and accounting operations themselves. When a customer performs a product registration operation (for example, an operation of scanning a symbol such as a barcode attached to a product and the appearance of the product and having the imaging device 3 capture an image of the product), the POS terminal 1 executes product registration processing related to the product to be sold based on the product identification information received from the imaging device 3. The POS terminal 1 also executes payment processing based on the customer's accounting operation (an operation of ending the product registration operation and starting the accounting process).
[0030] The product registration process refers to a process of acquiring a product code (product identification information) that identifies a product sold at the store T, displaying product information (product name, product price, etc.) such as the product name and price of the product on the display 18 (see Figs. 4 and 6) based on the acquired product code, and storing the product information in the product information unit 131 (see Fig. 6). The settlement process refers to a closing process related to the transaction based on the product information stored in the product information unit 131 in conjunction with the product registration process, specifically, displaying the total amount on the display 18, settlement processing by media such as cash, credit card, electronic money, etc., calculating the change based on the deposit in the case of cash settlement and displaying it on the display 18, issuing a receipt from the printer 22 (see Figs. 4 and 6) on which product information and accounting information (total amount, deposit amount, change amount, point information, etc.) of the settled product are printed.
[0031] The POS terminal 1 also transmits product information and transaction information (collectively referred to as "sales data") for the products for which payment has been processed to the store server 7. In this embodiment, the sales data is transmitted from the POS terminal 1 to the store server 7 when the transaction process is completed, but it may be transmitted at any timing, such as transmitting all data for one day at once when store T closes.
[0032] The store server 7 is installed in the back yard of the store T. The store server 7 collects sales data received from the POS terminal 1. The store server 7 manages the sales of products of the store T based on the collected sales data.
[0033] The store server 7 additionally stores the association information received from the edge device 5 in the matching dictionary 744 (see FIG. 9) stored therein. Specifically, the store server 7 extracts a captured image from the product image included in the association information received from the edge device 5 using the object detection model 742 (see FIG. 9), and extracts features based on the captured image using the feature extraction model 743 (see FIG. 9). The store server 7 then adds and stores the extracted features in the matching dictionary 744 stored therein, based on the product identification information included in the decoded information associated with the product image. In this way, the feature amounts stored for each product identification information in the matching dictionary 744 are increased, thereby training the matching dictionary 744 and improving the product recognition accuracy by the matching dictionary 744. The store server 7 then transmits the trained information of the matching dictionary 744 to the edge device 5.
[0034] In the embodiment, the object detection model 742, the feature extraction model 743, and the matching dictionary 744 are provided in the store server 7, but the object detection model 742, the feature extraction model 743, and the matching dictionary 744 may be provided in a server other than the store server 7 (for example, a headquarters server installed in a headquarters that manages and oversees multiple store servers 7).
[0035] Next, the POS terminal 1 and the imaging device 3 will be described. Fig. 4 is a perspective view showing the appearance of the POS terminal 1 and the imaging device 3. As shown in Fig. 4, the POS terminal 1 has a main body 42 and a basket placement stand 4. The main body 42 incorporates a circuit board and a power source for controlling the POS terminal 1. The basket placement stand 4 is attached to one side of the main body 42, and is a stand for placing a basket containing products.
[0036] The top surface of the main body 42 has a flat product placement section 41. The main body 42 has a pair of support posts 45 extending upward at the rear of the product placement section 41. A printer 22 that issues receipts is attached to the support posts 45 on the side farther from the basket placement platform 4. A card reader 21 that reads information from cards such as credit cards and point cards is attached to the top of the printer 22.
[0037] Additionally, an imaging device 3 is attached to the support pillar 45 on the side closer to the basket placement platform 4. The imaging device 3 is installed with its reading surface facing the customer, and a first imaging section 37 provided inside the imaging device 3 captures an image of a commodity by scanning the commodity that the customer has removed from the basket. Additionally, since customers tend to aim the symbol attached to the commodity at the imaging device 3 to scan the commodity, the first imaging section 37 often simultaneously captures an image of the symbol attached to the commodity. In the following embodiments, it is assumed that the first imaging section 37 captures an image of the symbol.
[0038] Further, the second imaging section 38, which is a part of the imaging device 3, is attached above the imaging device 3. The second imaging section 38 is installed such that its reading surface faces diagonally downward from the imaging device 3, and images the scanned commodity from a different angle than the first imaging section 37. In other words, the second imaging section 38 images a surface of the commodity different from the surface imaged by the first imaging section 37.
[0039] A display 18 is provided between the pair of support columns 45. The display 18 is a display made of, for example, liquid crystal, and is installed facing a customer who operates the POS terminal 1. The display 18 is located slightly above the imaging device 3 and the printer 22, and at the same height as the card reader 21 and the second imaging unit 38.
[0040] The display 18 displays product information (product name, product price, etc.) of the registered product and payment information (total amount, payment amount, change amount, etc.) of the payment processed. An operation unit 17 consisting of a touch panel is provided on the display surface of the display 18.
[0041] In addition, between the pair of support columns 45 and below the display 18, there are provided a temporary placement table 44 for temporarily placing products, and a pair of attachment parts 43 for attaching handles of a shopping bag for containing products that have undergone product registration processing. The products placed in the shopping bag are placed on the product placement part 41.
[0042] Furthermore, a substantially cylindrical pole 6 extending upward is attached to the support 45 on the side closer to the car placement platform 4. A patrol lamp 61 is attached to the pole 6.
[0043] The patrol lamp 61 lights up or flashes when an abnormality occurs in the POS terminal 1 (for example, when the receipt paper runs out) or when a customer calls an attendant.
[0044] The edge device 5 is stored inside the main body 42.
[0045] In such a POS terminal 1, a customer performs a product registration operation by removing a product from a basket placed on the basket placement table 4 and capturing an image of the symbol affixed to the product by passing (scanning) in front of the imaging device 3 with the first imaging unit 37 and the second imaging unit 38. The customer then places the product for which the product registration operation has been performed into a shopping bag attached to the mounting unit 43. The POS terminal 1 executes product registration processing for the product for which the product registration operation has been performed based on the product identification information received from the imaging device 3. The customer who has performed the product registration operation for all products then performs a checkout operation by operating the checkout button. When the checkout operation has been performed, the POS terminal 1 executes payment processing for the products for which the product registration operation has been performed.
[0046] Next, a description will be given of the flow of processing in the system 10. Fig. 5 is a timing chart showing the flow of processing in the system 10. As shown in Fig. 5, the system 10 includes a POS terminal 1, an imaging device 3, an edge device 5, and a store server 7.
[0047] When a customer performs a product registration operation on the POS terminal 1, the imaging device 3 captures an image of the product registration operation by the customer using the first imaging unit 37 and the second imaging unit 38 (S1). Next, the imaging device 3 uses the object detection model 542 to extract a product image based on captured image information (S2). In S2, the imaging device 3 extracts a product image using both the image captured by the first imaging unit 37 and the image captured by the second imaging unit 38. Then, the imaging device 3 outputs the extracted product image to the edge device 5 (S3).
[0048] The edge device 5 that has received the product image performs general object recognition. That is, the edge device 5 uses the object detection model 542 to generate an object image showing the product by specifying the area of the product included in the product image, uses the feature extraction model 543 to extract a feature vector showing the features of the product based on the object image, and uses the matching dictionary 544 to calculate the similarity of the product to the captured product, and extracts products similar to the product (having high similarity) (S4). In S4, the edge device 5 specifies one product if the calculated similarity is equal to or greater than a threshold. On the other hand, if the calculated similarity is less than the threshold, the edge device 5 specifies multiple candidate products in descending order of similarity. Then, the edge device 5 outputs product specification information of the specified product to the imaging device 3 (S5).
[0049] Meanwhile, the imaging device 3 that output the product image in S3 waits for the first imaging unit 37 or the second imaging unit 38 to image the symbol in response to the customer's product registration operation. When the symbol is imaged, the symbol is decoded to generate decoded information including the product identification information (S6). Note that the order of input of the product identification information by S5 and generation of the decoded information by S6 may change depending on the timing at which the symbol is imaged. If the customer does not correctly face the symbol toward the first imaging unit 37, the image of the symbol will be captured late, and if the generation of the decoded information is delayed, the input of the product identification information from the edge device 5 may come first. Conversely, if the image of the symbol is captured early, the generation of the decoded information may come earlier than the input of the product identification information from the edge device 5.
[0050] If the decoded information has not been generated at the timing when the product identification information is input from the edge device 5 (S5), the imaging device 3 transmits the product identification information to the POS terminal 1 without waiting for the generation of the decoded information (S7). Conversely, if the decoded information has been generated before the product identification information is input from the edge device 5, the imaging device 3 transmits the product identification information included in the decoded information to the POS terminal 1 without waiting for the timing when the product identification information is input from the edge device 5 or for the product identification information to be input from the edge device 5 (S7).
[0051] The POS terminal 1 executes the product registration process based on the received product identification information (S8). Furthermore, when the customer instructs to end the product registration, the POS terminal 1 executes the payment process (S9).
[0052] The imaging device 3 also generates associated information by associating the product images extracted in S2 with the decoded information generated in S6 (S10). The product images extracted in S2 include images captured by the first imaging unit 37 and images captured by the second imaging unit 38, and therefore include product images of products with symbols (for example, images of frames 2 to 8 in FIG. 2) captured by the first imaging unit 37 and product images of products with no symbols captured by the second imaging unit 38. The imaging device 3 generates associated information by associating the decoded information with these product images. The imaging device 3 then outputs the generated associated information to the edge device 5 (S11). Note that the process of S7 and the processes of S10 and S11 may be performed in any order.
[0053] The edge device 5 that has received the association information selects and discards the input product images (S12). Since customers tend to habitually perform product registration operations with the side with the symbol facing the first imaging unit 37, product images near the symbol tend to be collected. Since collecting object images of products captured from different angles as much as possible leads to correct product recognition (improving the accuracy of the dictionary), the edge device 5 selects and discards the product images so that the number of product images near the symbol is not excessively large compared to other images. Then, the edge device 5 transmits the association information containing the selected product images to the store server 7 (S13).
[0054] The store server 7 that has received the association information extracts an object image using the object detection model 742, calculates a feature amount of a product included in the object image using the feature extraction model 743 (see FIG. 9), and additionally stores the feature amount in association with the same product identification information as the product identification information included in the received decoded information stored in the matching dictionary 744 (see FIG. 9) (S14). The process of S14 allows the matching dictionary 744 to be easily trained on a daily basis. This improves the accuracy of the matching dictionary 744.
[0055] If the product imaged in S1 is a new product, the product identification information included in the decoded information may not yet be stored in the matching dictionary 744. In this case, new associated information that associates the product identification information that identifies the new product with the feature amount is created in the matching dictionary 744. In this way, it is not necessary to perform a separate task of associating the product identification information with the feature amount of the new product. This process can be easily performed by performing package recognition (performing both code recognition and general object recognition) (this process cannot be performed by simply performing general object recognition on the product).
[0056] Next, the store server 7 transmits information about the learned matching dictionary 744 to the edge device 5 (S15). The edge device 5 overwrites the previous matching dictionary 544 with the received information about the matching dictionary 744 as the matching dictionary 544 (S16). In this way, the matching dictionary 544 can be easily learned on a daily basis. This improves the accuracy of the matching dictionary 544.
[0057] Next, the hardware configuration of the POS terminal 1 will be described. FIG. 6 is a block diagram showing the hardware configuration of the POS terminal 1. As shown in FIG. 6, the POS terminal 1 includes a CPU (Central Processing Unit) 11, which is an example of a processor, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a memory unit 14, and the like. The CPU 11 is the main controller of the POS terminal 1. The ROM 12 stores various programs. The RAM 13 expands programs and various data. The memory unit 14 stores various programs. The CPU 11, the ROM 12, the RAM 13, and the memory unit 14 are connected to one another via a bus 15. The CPU 11, the ROM 12, and the RAM 13 constitute a control unit 100. That is, the control unit 100 executes a control process of the POS terminal 1, which will be described later, by the CPU 11 operating in accordance with a control program stored in the ROM 12 or the memory unit 14 and expanded in the RAM 13.
[0058] The RAM 13 includes a product information section 131. The product information section 131 stores product information (product code that identifies the product, product name, price, etc.) of a product that has been registered.
[0059] The memory unit 14 is composed of a non-volatile memory such as a hard disk drive (HDD) or a flash memory that retains stored information even when the power is turned off, and has a control program unit 141 that stores a control program, and a product master 142. The product master 142 stores product information of each product in association with product identification information that identifies the product.
[0060] In addition, the control unit 100 is connected to an operation unit 17 , a display unit 18 , a card reader 21 , and a printer 22 via a bus 15 and a controller 16 .
[0061] The operation unit 17 has a closing key 171. The closing key 171 is a key that is operated when advancing the process from product registration processing to payment processing in the POS terminal 1. The display 18 is, for example, a liquid crystal display, and displays information to a customer who is the operator operating the POS terminal 1. The display 18 displays, for example, product information (product name, product price, etc.) of products that have been registered, and accounting information related to the payment processing for each product. The operation unit 17 is a touch panel provided on the display 18.
[0062] The card reader 21 reads card information from a card, for example, a credit card, a debit card, or an electronic money card when performing payment processing. The card reader 21 also reads and writes point information from a point card. The printer 22 is, for example, a thermal printer, and issues a receipt on which receipt information including product information and payment information relating to the product for which payment processing has been performed is printed.
[0063] The control unit 100 is also connected to a communication unit 23 via a bus 15. The communication unit 23 is communicatively connected to the edge device 5 and the store server 7 via a communication line L. The communication unit 23 is also communicatively connected to the imaging device 3 via a cable U.
[0064] Next, the hardware configuration of the imaging device 3 will be described. FIG. 7 is a block diagram showing the hardware configuration of the imaging device 3. As shown in FIG. 7, the imaging device 3 includes a CPU 31, which is an example of a processor, a ROM 32, a RAM 33, a memory unit 34, and the like. The CPU 31 is the main controller of the imaging device 3. The ROM 32 stores various programs. The RAM 33 expands programs and various data. The memory unit 34 stores various programs. The CPU 31, the ROM 32, the RAM 33, and the memory unit 34 are connected to each other via a bus 35. The CPU 31, the ROM 32, and the RAM 33 constitute a control unit 300. That is, the control unit 300 executes a control process of the imaging device 3, which will be described later, by the CPU 31 operating according to a control program stored in the ROM 32 or the memory unit 34 and expanded in the RAM 33.
[0065] The RAM 33 has an image storage unit 331, a product image unit 332, and a decoded information unit 333. The image storage unit 331 stores an image captured by the first imaging unit 37 and an image captured by the second imaging unit 38. The product image unit 332 stores a product image extracted from the image stored in the image storage unit 331. The decoded information unit 333 stores decoded information obtained by decoding a symbol included in the image captured by the first imaging unit 37.
[0066] The memory unit 34 is configured with a non-volatile memory such as an HDD or a flash memory in which stored information is retained even when the power is turned off, and has a control program unit 341 and an object detection model 542. The control program unit 341 stores a control program for driving the imaging device 3.
[0067] In addition, the control unit 300 is connected to a first imaging unit 37 and a second imaging unit 38 via a bus 35 and a controller 36.
[0068] The control unit 300 is also connected to a communication unit 39 via a bus 35. The communication unit 39 is communicatively connected to the edge device 5 and the store server 7 via a communication line L. The communication unit 39 is also communicatively connected to the POS terminal 1 via a cable U.
[0069] Next, the hardware configuration of the edge device 5 will be described. FIG. 8 is a block diagram showing the hardware configuration of the edge device 5. As shown in FIG. 8, the edge device 5 includes a CPU 51, which is an example of a processor, a ROM 52, a RAM 53, a memory unit 54, and the like. The CPU 51 is a main controller of the edge device 5. The ROM 52 stores various programs. The RAM 53 expands programs and various data. The memory unit 54 stores various programs. The CPU 51, the ROM 52, the RAM 53, and the memory unit 54 are connected to each other via a bus 55. The CPU 51, the ROM 52, and the RAM 53 constitute a control unit 500. That is, the control unit 500 executes a control process of the edge device 5, which will be described later, by the CPU 51 operating according to a control program stored in the ROM 52 or the memory unit 54 and expanded in the RAM 53.
[0070] The RAM 53 has an object image storage unit 531, a feature amount unit 532, a product identification information unit 533, and an association information unit 534. The object image storage unit 531 stores an object image extracted based on a product image received from the imaging device 3. The feature amount unit 532 stores a feature vector indicating the feature amount of the product calculated by the feature extraction model 543 based on the object image stored in the object image storage unit 531. The product identification information unit 533 compares the feature vector calculated by the feature extraction model 543 with the feature amount of the product stored in the comparison dictionary 544, calculates the similarity of a product close to the feature vector, and stores the product identification information of a product with a high similarity. If the calculated similarity is equal to or greater than a threshold, the product identification information of one product with the highest similarity is stored. If the calculated highest similarity is less than a threshold, the product identification information of multiple types of candidate products is stored in descending order of similarity. The association information unit 534 stores the association information received from the imaging device 3. The association information unit 534 also stores association information generated by the edge device 5, the details of which will be described later.
[0071] The memory unit 54 is composed of a non-volatile memory such as an HDD or a flash memory in which stored information is retained even when the power is turned off, and has a control program unit 541, an object detection model 542, a feature extraction model 543, and a matching dictionary 544. The control program unit 541 stores a control program for driving the edge device 5.
[0072] The control unit 500 is also connected to the bus 55 and the controller 56. The control unit 500 is also connected to the communication unit 357 via the bus 55. The communication unit 57 is connected to the POS terminal 1, the imaging device 3, and the store server 7 via the communication line L so as to be able to communicate with them.
[0073] Next, the hardware configuration of the store server 7 will be described. FIG. 9 is a block diagram showing the hardware configuration of the store server 7. As shown in FIG. 9, the store server 7 includes a CPU 71, which is an example of a processor, a ROM 72, a RAM 73, a memory unit 74, and the like. The CPU 71 is the main controller of the store server 7. The ROM 72 stores various programs. The RAM 73 expands programs and various data. The memory unit 74 stores various programs. The CPU 71, the ROM 72, the RAM 73, and the memory unit 74 are connected to one another via a bus 75. The CPU 71, the ROM 72, and the RAM 73 constitute a control unit 700. That is, the control unit 700 executes a control process of the store server 7, which will be described later, by the CPU 71 operating in accordance with a control program stored in the ROM 72 or the memory unit 74 and expanded in the RAM 73.
[0074] The RAM 73 has a sales information section 731 and an association information section 732. The sales information section 731 stores sales data received from the POS terminal 1 and manages sales of the store T. The association information section 732 stores association information received from the edge device 5.
[0075] The memory unit 74 is composed of a non-volatile memory such as an HDD or flash memory in which stored information is retained even when the power is turned off, and has a control program unit 741, an object detection model 742, a feature extraction model 743, and a matching dictionary 744. The control program unit 741 stores a control program for driving the store server 7. The object detection model 742 is a dictionary with the same content as the object detection model 542, the feature extraction model 743 is a dictionary with the same content as the feature extraction model 543, and the matching dictionary 744 is a dictionary with the same content as the matching dictionary 544.
[0076] In addition, the control unit 700 is connected to a display unit 77 and an operation unit 78 via the bus 55 and the controller 56. The display unit 77 is, for example, a liquid crystal display, and displays information to an operator who operates the store server 7. The operation unit 78 is, for example, a keyboard, and is operated by the operator of the store server 7.
[0077] The control unit 700 is also connected to a communication unit 79 via a bus 75. The communication unit 79 is connected to the POS terminal 1, the imaging device 3, and the edge device 5 via a communication line L so as to be able to communicate with each other.
[0078] From here, the control of the POS terminal 1 will be described. FIG. 10 is a flowchart showing the flow of the control process of the POS terminal 1. As shown in FIG. 10, the control unit 100 of the POS terminal 1 judges whether or not a customer has operated to start a transaction (S21). For example, when a customer operates a button to start a transaction (e.g., a start button), the control unit 100 judges that an operation to start a transaction has been performed. The control unit 100 waits until an operation to start a transaction is performed (No in S21), and when it is determined that an operation to start a transaction has been performed by the customer (Yes in S21), the control unit 100 transmits a signal to the imaging device 3 to start imaging by the first imaging unit 37 and the second imaging unit 38 (S22).
[0079] Next, the control unit 100 judges whether product identification information for identifying a product has been received from the imaging device 3 (S23). If it is judged that product identification information has been received (Yes in S23), the control unit 100 judges whether the received product identification information is one (S24). If it is judged that the received product identification information is one (Yes in S24), the control unit 100 uses the product identification information to read information on the product identified by the product identification information from the product master 142 and executes product registration processing (S25).
[0080] Furthermore, if it is determined that a plurality of pieces of product identification information have been received (No in S24), the control unit 100 reads out information on a plurality of products identified by the plurality of pieces of received product identification information (e.g., product names and product images) from the product master 142, and displays them as candidate products on the display 18 (S26). The customer operates the operation unit 17 to select one product from the displayed candidate products. The control unit 100 determines whether any one product has been selected (S27). The control unit 100 waits until a product is selected (No in S27), and if it is determined that a product has been selected (Yes in S27), the control unit 100 reads out product information for the selected product from the product master 142 and executes product registration processing (S25).
[0081] If it is determined that the product specification information has not been received (No in S23) and if the product registration process has been executed in S25, the control unit 100 then determines whether the closing key 171 has been operated (S28). If it is determined that the closing key 171 has not been operated (No in S28), the control unit 100 returns to S23, and if it is determined that the closing key 171 has been operated (Yes in S28), the control unit 100 executes the payment process based on the product information stored in the product information unit 131 (S29). Next, the control unit 100 transmits sales data related to the product for which the payment process has been performed to the store server 7 (S30). Then, the control unit 100 transmits an imaging end signal to the imaging device 3 (S31). Then, the control unit 100 terminates the process.
[0082] From here, we will explain the functional configuration of the imaging device 3. Fig. 11 is a functional block diagram showing the functional configuration of the imaging device 3. A control unit 300 of the imaging device 3 functions as an extraction means 301, a decoding means 302, an association means 303, an output means 304, and a transmission means 305 by following a control program stored in the ROM 32 or a control program unit 341 of the memory unit 34.
[0083] The extraction means 301 extracts a product image showing a product from the images captured by the imaging units (the first imaging unit 37 and the second imaging unit 38). Specifically, the extraction means 301 extracts a product image showing a product from the images captured by the first imaging unit 37 and the second imaging unit 38 stored in the image storage unit 331.
[0084] The decoding means 302 generates decoded information by decoding a symbol attached to a product included in an image captured by the imaging unit. Specifically, the decoding means 302 generates decoded information by decoding an image of a symbol included in a product image stored in the product image unit 332. Note that the decoding means 302 may generate decoded information by decoding an image of a symbol included in an image stored in the image storage unit 331.
[0085] The associating means 303 associates the image captured by the imaging unit with the decoded information. In the embodiment, the associating means 303 associates the product image stored in the product image section 332 extracted by the extraction means 301 with the decoded information stored in the decoded information section 333 generated by the decoded means 302. That is, the associating means 303 generates association information that associates the product image stored in the product image section 332 extracted by the extraction means 301 with the decoded information stored in the decoded information section 333 generated by the decoded means 302.
[0086] The associating means 303 may associate the decoded information with all the commodity images stored in the commodity image section 332, but the associating means 303 may, for example, associate the commodity images of a predetermined number of frames before and after the image including the symbol decoded by the decoding means 302. The associating means 303 may, for example, associate the commodity images of a predetermined number of frames before the image including the symbol decoded by the decoding means 302. The customer may perform an operation to change the angle of the commodity in various ways in order to make the imaging unit (particularly the first imaging unit 37) read the symbol attached to the commodity. Therefore, by associating the commodity images of a predetermined number of frames before the image in which the symbol is read, there is a possibility that the commodity images in which the commodity is imaged from various angles may be associated. The associating means 303 may, for example, associate the commodity images of a predetermined number of frames after the image including the symbol decoded by the decoding means 302.
[0087] The output unit 304 outputs the association information associated by the associating unit 303 to the edge device 5. The output unit 304 may output the associated information to the store server 7.
[0088] The transmission means 305 transmits to the POS terminal 1 either the product identification information included in the decoded information or the product identification information identified by the general object recognition performed on the images captured by the first imaging unit 37 and the second imaging unit 38. Specifically, the transmission means 305 transmits to the POS terminal 1 either the product identification information included in the decoded information generated by the decoded means 302 or the product identification information identified and input by the general object recognition performed on the edge device 5 on the product image extracted by the extraction means 301.
[0089] Next, the control of the imaging device 3 will be described. Fig. 12 is a flowchart showing the flow of the control process of the imaging device 3. As shown in Fig. 12, the control unit 300 of the imaging device 3 judges whether an imaging start signal has been received from the POS terminal 1 (S41). The control unit 300 waits until the imaging start signal is received (No in S41), and if it is determined that the imaging start signal has been received from the POS terminal 1 (Yes in S41), the control unit 300 starts the first imaging unit 37 and the second imaging unit 38 to start imaging (S42).
[0090] Next, the control unit 300 judges whether an imaging end signal has been received from the POS terminal 1 (S43). If it is judged that an imaging end signal has not been received (No in S43), the control unit 300 stores the images captured by the first imaging unit 37 and the second imaging unit 38 in the image storage unit 331 (S44). Then, the extraction means 301 extracts a product image showing a product from the images stored in the image storage unit 331 (S45). Then, the control unit 300 stores the extracted product image in the product image unit 332. Then, the control unit 300 transmits the product image stored in the product image unit 332 to the edge device 5 (S46).
[0091] Next, the control unit 300 judges whether a symbol has been detected from the product image stored in the product image section 332 (S51). If it is judged that a symbol has been detected (Yes in S51), the decoding means 302 decodes the detected symbol to generate decoded information (S52). Then, the control unit 300 stores the generated decoded information in the decoded information section 333 (S52).
[0092] Next, the associating means 303 generates association information that associates the decoded information stored in the decoded information unit 333 with the product image stored in the product image unit 332 (S53). Then, the output means 304 outputs the generated association information to the edge device 5 (S54). Then, the control unit 300 returns to S43.
[0093] Also, in S51, if it is determined that the symbol is not detected (No in S51), the control unit 300 determines whether the product identification information is input from the edge device 5 (S61). If it is determined that the product identification information is not input from the edge device 5 (No in S61), the control unit 300 returns to S51. If it is determined that the product identification information is input from the edge device 5 (Yes in S61), the control unit 300 next determines whether the input product identification information is one (S62). If it is determined that the product identification information is one (Yes in S62), the control unit 300 determines whether the symbol has already been decoded in S52 (S63). If it is determined that the symbol has already been decoded (Yes in S63), the transmission means 305 transmits the product identification information included in the decoded information to the POS terminal 1 (S64). Then, the control unit 300 returns to S43. If it is determined that the product identification information is not decoded (No in S63), the transmission means 305 transmits the product identification information received in S62 to the POS terminal 1 (S65). Then, the control unit 300 returns to S43.
[0094] Furthermore, if it is determined in S62 that the input product identification information is a plurality of pieces (No in S62), the control unit 300 determines whether the symbol has already been decoded in S52 (S66). If it is determined that the symbol has already been decoded (Yes in S66), the transmission means 305 transmits the product identification information contained in the decoded information to the POS terminal 1 (S67). Then, the control unit 300 returns to S43. If it is determined that the decoded information has not been decoded (No in S66), the transmission means 305 transmits the plurality of types of product identification information (plural pieces of product identification information related to candidate products) received in S62 to the POS terminal 1 (S68). Then, the control unit 300 returns to S43.
[0095] 12, if the answer is Yes in S63, the product identification information included in the decoded information is transmitted to the POS terminal 1 at this point, but the product identification information included in the decoded information may be transmitted to the POS terminal 1 after any of the processes in S52 to S54. In this case, if the answer is Yes in S63, the control unit 500 returns to S43. The same applies if the answer is Yes in S66.
[0096] Furthermore, in S43, when it is determined that an imaging end signal has been received from the POS terminal 1 (Yes in S43), the control unit 300 ends imaging by the first imaging unit 37 and the second imaging unit 38 (S69). Then, the control unit 300 ends the process.
[0097] Next, the control of the edge device 5 will be described. FIG. 13 is a flowchart showing the flow of the control process of the edge device 5. As shown in FIG. 13, the control unit 500 of the edge device 5 judges whether a product image has been received from the imaging device 3 (S71). If it is judged that a product image has been received from the imaging device 3 (Yes in S71), the control unit 500 uses the object detection model 542 to identify a product area in the received product image (S72). Next, the control unit 500 stores an object image surrounded by the identified area in the object image storage unit 531 (S73).
[0098] Next, the control unit 500 calculates a feature vector of the object image stored in the object image storage unit 531 using the feature extraction model 543 (S74). Then, the control unit 500 stores the calculated feature vector in the feature amount unit 532 (S75). Next, the control unit 500 uses the comparison dictionary 544 to compare the feature vector stored in the feature amount unit 532 with the feature amount stored for each item of product identification information stored in the comparison dictionary 544, and extracts product identification information of items with high similarity (S76). The product identification information extracted in S76 is, for example, the highest product identification information when there is a product with a similarity of 95% or more, and when all items have a similarity of less than 95%, the product identification information is the product identification information of multiple (e.g., five) candidate items in descending order of similarity from among the products. Then, the control unit 500 stores the extracted product identification information in the product identification information unit 533. Then, the control unit 500 transmits the product identification information stored in the product identification information unit 533 to the imaging device 3 (S77). Then, the control unit 500 returns to S71.
[0099] The control unit 500 may generate association information that associates the product identification information stored in the product identification information unit 533 with the object image stored in the object image storage unit 531 in S73, and transmit the association information to the store server 7. In this way, the store server 7 can collect more images and train the matching dictionary 744.
[0100] Furthermore, in S71, if it is determined that the product image has not been received from the imaging device 3 (No in S71), it is determined whether association information has been received from the imaging device 3 (S81). If it is determined that association information has been received from the imaging device 3 (Yes in S81), the control unit 500 stores the received association information in the association information unit 534 (S82). Then, the control unit 500 transmits the association information stored in the association information unit 534 to the store server 7 (S83). Then, the control unit 500 returns to S71.
[0101] Also, when it is determined that the association information is not received from the imaging device 3 (No in S81), the control unit 500 determines whether or not information on the matching dictionary 744 is received from the store server 7 (S84). The information on the matching dictionary 744 received in S84 is a matching dictionary that is additionally stored and learned in the store server 7 by the association information. When it is determined that the information on the matching dictionary 744 is received from the store server 7 (Yes in S84), the control unit 500 overwrites the received information on the matching dictionary 744 in the matching dictionary 544 and stores it (S88). In this way, the matching dictionary 544 is trained, and the accuracy of the matching dictionary 544 when calculating the similarity is improved. Then, the control unit 500 returns to S71.
[0102] Next, the control of the store server 7 will be described. FIG. 14 is a flowchart showing the flow of the control process of the store server 7. As shown in FIG. 14, the control unit 700 of the store server 7 judges whether sales data has been received from the POS terminal 1 (S91). If it is judged that sales data has been received from the POS terminal 1 (Yes in S91), the control unit 700 adds and stores the received sales data in the sales information unit 731 (S92) and manages the sales of the store T. Then, the control unit 700 returns to S91.
[0103] Furthermore, if it is determined that the sales data has not been received from the POS terminal 1 (No in S91), the control unit 700 determines whether association information has been received from the edge device 5 (S93). If it is determined that association information has not been received (No in S93), the control unit 700 returns to S91, and if it is determined that association information has been received from the edge device 5 (Yes in S93), the control unit 700 stores the received association information in the association information unit 732 (S94).
[0104] Next, the control unit 700 updates the matching dictionary 744 based on the associated information stored in the associated information unit 732 (S95). That is, the control unit 700 performs general object recognition on the product image (which may be an object image) associated with the product identification information as the associated information. Specifically, the control unit 700 uses the object detection model 742 to specify the area of the product image, and uses the feature extraction model 743 to extract a feature amount indicating the feature of the object image of the product whose area has been specified. Then, the control unit 700 additionally stores the extracted feature amount in association with the same product identification information as the product identification information associated with the product image stored in the matching dictionary 744. In this way, the feature amount associated with the product identification information is added to the matching dictionary 744, and the matching dictionary 744 is made to learn the feature amount, thereby improving the accuracy of the matching dictionary 744.
[0105] Then, the control unit 700 transmits the learned information of the matching dictionary 744 to the edge device 5 (S96). Then, the control unit 700 returns to S91. The edge device 5 that has received the information of the matching dictionary 744 from the store server 7 updates the matching dictionary 544 by overwriting the matching dictionary 544 with the received information of the matching dictionary 744. That is, the matching dictionary 544 is learned in the same way as the matching dictionary 744.
[0106] As described above, the imaging device 3 of the embodiment comprises a first imaging unit 37 and a second imaging unit 38, a decoding means 302 that generates decoded information by decoding a symbol attached to a product included in the image captured by the first imaging unit 37, an association means 303 that associates the image captured by the first imaging unit 37 and the second imaging unit 38 with the decoded information, an output means 304 that outputs the associated association information, and a transmission means 305 that transmits to the POS terminal 1 either product identification information included in the decoded information or product identification information identified by general object recognition performed on the images captured by the first imaging unit 37 and the second imaging unit 38.
[0107] In such an embodiment, the imaging device 3 outputs the symbol decode information and image in association with each other based on the image and symbol captured by the customer's product registration operation, so there is no need to perform a separate task of associating the decode information with the image in order to train the matching dictionary 544, and the matching dictionary 544 can be easily trained.
[0108] Although the embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.
[0109] For example, in the embodiment, the edge device 5 has been described as a standalone device. However, the present invention is not limited to this, and the edge device 5 may be incorporated in, for example, the imaging device 3, the POS terminal 1, or the store server 7. In other words, the imaging device 3, the POS terminal 1, or the store server 7 may have the functions of the edge device 5.
[0110] In the embodiment, the POS terminal 1 is described as a device operated by the customer himself / herself. However, the present invention is not limited to this, and the POS terminal 1 may be a device operated by a clerk of the store T. In this case, the imaging device 3 associates the product image with the decoded information based on the product registration operation by the clerk.
[0111] In the embodiment, the product image extracted by the extraction means 301 from the images captured by the first imaging unit 37 and the second imaging unit 38 is associated with the decoded information. However, this is not limiting, and for example, the images themselves captured by the first imaging unit 37 and the second imaging unit 38 may be associated with the decoded information.
[0112] In the embodiment, two imaging units, the first imaging unit 37 and the second imaging unit 38, are provided. However, the present invention is not limited to this, and either the first imaging unit 37 or the second imaging unit 38 may be provided as an imaging unit. The imaging unit may have one or more individual imaging units. The more imaging units are arranged in different positions, the more multi-angled images of the product can be captured.
[0113] In addition, there is another embodiment in which the associating means 303 described as a configuration of the imaging device 3 in the embodiment is included in the edge device 5 or the store server 7 (the edge device 5 and the store server 7 are collectively referred to as the "information processing device"). In the case of another embodiment, the imaging device 3 transmits an image captured by the imaging unit and decoded information decoded by the decoding means to the information processing device. The information processing device receives the image and decoded information from the imaging device 3, and executes an associating means for associating the image and the decoded information. Then, when the information processing device is the edge device 5, it transmits the associated association information to the store server 7, and the store server 7 trains the matching dictionary 744 based on the association information received from the edge device 5. Then, the store server 7 transmits the trained matching dictionary 744 to the edge device 5. Also, when the information processing device is the store server 7, it trains the matching dictionary 744 based on the associated association information. Then, it transmits the trained matching dictionary 744 to the edge device 5.
[0114] The program executed by the imaging device 3 of the embodiment is provided as a file in an installable or executable format recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk).
[0115] The program executed by the imaging device 3 of the embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program executed by the imaging device 3 of the embodiment may be provided or distributed via a network such as the Internet.
[0116] Furthermore, the programs executed by the imaging device 3 of the embodiment may be provided by being pre-installed in a ROM or the like. [Explanation of symbols]
[0117] 1 POS terminal 3. Imaging device 5. Edge Devices 7 Store Server 10. System 11 CPU 31 CPU 37 First Imaging Unit 38 Second Imaging Unit 51 CPU 71 CPU 100 Control section 131 Product Information Department 142 Product Master 300 Control section 301 Extraction means 302 Decoding Method 303 Association means 304 Output Method 305 Transmission Method 331 Image storage unit 332 Product Image Section 333 Decode Information Section 500 Control section 531 Object Image Storage Unit 532 Feature Quantity Section 533 Product Specific Information Department 534 Related Information Section 542 Object Detection Model 543 Feature Extraction Model 544 Matching Dictionary 700 Control section 731 Sales Information Department 732 Related Information Section 742 Object Detection Model 743 Feature Extraction Model 744 Matching Dictionary [Prior art documents] [Patent documents]
[0118] [Patent Document 1] JP 2014-21915 A
Claims
1. An imaging unit; a decoding means for generating decoded information by decoding a symbol attached to a product included in an image captured by the imaging unit; an association unit for associating the image captured by the imaging unit with the decoded information; an output means for outputting the associated information associated by the associating means; a transmission means for transmitting to a product sales data processing device either product identification information included in the decoded information or product identification information identified by a generic object recognition performed on an image captured by the imaging unit; An imaging device comprising:
2. The apparatus further includes an extraction unit for extracting a product image in which the product is shown from the image captured by the imaging unit, The associating means associates the product image with the decoded information. The imaging device according to claim 1 .
3. The extraction means extracts an image of a predetermined range including an image of the symbol. The imaging device according to claim 2 .
4. The extraction means extracts an image of a predetermined range including the image of the symbol and before and after the image of the symbol. The imaging device according to claim 2 .
5. The imaging unit further includes a first imaging unit and a second imaging unit installed at a different position and angle from the first imaging unit, The extraction means further extracts a product image in which a product is shown from the image captured by the first imaging unit and the image captured by the second imaging unit, The associating means associates the product images captured by the first imaging unit and the second imaging unit with the decoded information.
5. The imaging device according to claim 2.
6. A computer as an imaging device having an imaging unit, a decoding means for generating decoded information by decoding a symbol attached to a product included in an image captured by the imaging unit; an association unit for associating the image captured by the imaging unit with the decoded information; an output means for outputting the associated information associated by the associating means; a transmission means for transmitting to a product sales data processing device either product identification information included in the decoded information or product identification information identified by a generic object recognition performed on an image captured by the imaging unit; A program to make it function as such.
Citation Information
Patent Citations
Recognition system, information processor and program
JP2017215791A
Commodity recognition device and recognition dictionary adding program
JP2014021915A
