Commodity sales data processing device, checkout device, and program

Abstract

To provide a self-type commodity sales data processing device, a checkout device, and a program that can reduce a burden related to cancellation processing on a shop clerk or the like.SOLUTION: A commodity sales data processing device that executes commodity registration processing of a commodity through operation of a customer himself / herself, includes: an interface to which first commodity specifying information for specifying, in an image of a commodity captured by an image capturing unit, the commodity whose code has been recognized based on a code symbol included in the image and second commodity specifying information for specifying, in the image, the commodity whose package has been recognized using a general object recognition technology can be input; and cancellation means for enabling, for a commodity related to the second commodity specifying information input to the interface, cancellation of the commodity registration processing by the customer after commodity registration processing of the commodity, and disabling, for a commodity related to the first commodity specifying information input to the interface, cancellation of the commodity registration processing performed by the customer after commodity registration processing of the commodity.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] An embodiment of the present invention relates to a product sales data processing device, a checkout device, and a program. [Background technology]

[0002] Recently, self-service POS terminals have been available in stores such as shopping centers, mass retailers, and department stores, where customers themselves perform product registration and payment processing operations for products sold at the store. Some of these POS terminals capture images of products purchased by customers and identify the products by recognizing the packages based on the captured images.

[0003] However, since such POS terminals are operated by customers, they are equipped with a function to prevent fraud by customers. For example, a customer may register an item to be purchased, and then cancel the item without actually returning it, thereby pretending to have cancelled the purchase of the item. In order to prevent such fraud, current self-service POS terminals are designed so that only store clerks such as checkers or managers (hereinafter collectively referred to as "clerks, etc.") can cancel items. Therefore, when a customer wants to cancel an item that has already been registered, they have to call a clerk, etc. each time. This has created a problem of excessive burden on clerks, etc. involved in the cancellation process. Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a self-service product sales data processing device, checkout device, and program that can reduce the burden on store clerks and others involved in cancellation processing. [Means for solving the problem]

[0005] The product sales data processing device of the embodiment is a product sales data processing device that executes product registration processing for a product through operation by a customer himself, and is equipped with an interface into which first product identification information that identifies the product by code recognition based on a code symbol contained in an image of a product captured by an imaging unit and second product identification information that identifies the product by package recognition using general object recognition technology for the image, and a cancellation means that allows the customer to cancel the product registration processing for a product related to the second product identification information inputted into the interface after the product registration processing for the product, and prevents the customer from canceling the product registration processing for a product related to the first product identification information inputted into the interface after the product registration processing for the product. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram showing a system according to an embodiment. [Diagram 2] FIG. 2 is an explanatory diagram showing the principle of package recognition. [Diagram 3] FIG. 3 is a diagram showing an example in which the object detection model extracts a product image. [Figure 4] FIG. 4 is a perspective view showing the appearance of the checkout device. [Diagram 5] FIG. 5 is a sequence diagram showing the flow of processing in the system. [Figure 6] FIG. 6 is a block diagram showing the hardware configuration of the POS terminal. [Figure 7] FIG. 7 is a block diagram showing the hardware configuration of the imaging apparatus. [Figure 8] FIG. 8 is a block diagram illustrating a hardware configuration of the edge device. [Figure 9] FIG. 9 is a block diagram showing a hardware configuration of the store server. [Figure 10] FIG. 10 is a functional block diagram showing the functional configuration of the POS terminal. [Figure 11]FIG. 11 is a flowchart showing the flow of the control process of the POS terminal. [Figure 12] FIG. 12 is a diagram showing an example of a display on the POS terminal. [Figure 13] FIG. 13 is a diagram showing another example of a display on the POS terminal. [Figure 14] FIG. 14 is a functional block diagram showing the functional configuration of the imaging device. [Figure 15] FIG. 15 is a flowchart showing the flow of control processing of the imaging device. [Figure 16] FIG. 16 is a functional block diagram showing the functional configuration of the edge device. [Figure 17] FIG. 17 is a flowchart showing the flow of the control process of the edge device. [Figure 18] FIG. 18 is a flowchart showing the flow of control processing of the store server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the embodiment, a self-service POS terminal (hereinafter referred to as a "POS terminal") where a customer performs registration and accounting operations will be described as an example of a product sales data processing device. Note that the present invention is not limited to the embodiment described below.

[0008] Fig. 1 is a diagram showing a system according to an embodiment. In Fig. 1, a system 10 is installed in a store T such as a supermarket, a mass retailer, a convenience store, or a specialty store. The system 10 is installed in the store T and includes one or more POS (Point of Sales) terminals 1 (product sales data processing devices), an imaging device 3 (imaging unit) connected to each POS terminal 1, an edge device 5, and a store server 7. Each POS terminal 1, the edge device 5, and the store server 7 are connected to a communication line L (for example, a dedicated line such as a LAN (Local Area Network), a public line, the Internet, or other network) that allows mutual communication.

[0009] The edge device 5 is, for example, a computer provided in the store server 7. Alternatively, the edge device 5 is a computer installed on a cloud. Alternatively, the edge device 5 is a computer provided in each POS terminal 1. In the embodiment, one edge device 5 is provided in the store.

[0010] The imaging devices 3 are connected to the POS terminals 1 one by one, for example, via a USB (Universal Serial Bus) cable U. Each imaging device 3 is also connected to an edge device 5 and a store server 7 via a communication line L.

[0011] Also in the system 10, the imaging device 3 captures an image of a product. If the captured image includes a code symbol such as a barcode or a two-dimensional code, the imaging device 3 performs code recognition based on the code symbol. The edge device 5 performs package recognition based on the image captured by the imaging device 3. The POS terminal 1 executes product registration processing based on product identification information of the product identified by the code recognition or package recognition. The POS terminal 1, the imaging device 3, and the edge device 5 constitute a checkout device 9 (see FIG. 5).

[0012] Package recognition is a technology that identifies products using a known general object recognition technology that recognizes products based on the shape and appearance of an image of the product. As a result of package recognition, product identification information that identifies the product (for example, a product code, but it may be something other than a product code) is obtained.

[0013] Regarding general object recognition, see, for example, "Keiji Yanai, 'The Current State and Future of General Object Recognition,' Transactions of the Information Processing Society of Japan, Vol. 48, No. SIG16 [Retrieved August 10, 2010], Internet<URL:http: / / mm.cs.uec.ac.jp / IPSJ-TCVIM-Yanai.pdf> " is stated in the article.

[0014] In code recognition, when an image of a code symbol attached to a product is captured, the code symbol is decoded and analyzed to obtain decoded information, which includes product identification information that identifies the product to which the code symbol is attached.

[0015] Next, the process of performing package recognition will be described. Fig. 2 is a diagram for explaining package recognition. As shown in Fig. 2, package recognition uses three types of dictionaries, namely, an object detection model 542, a feature extraction model 543, and a matching dictionary 544, to identify an image of a product.

[0016] The object detection model 542 is a dictionary for identifying an area (range) of an object (product) captured in a captured image. That is, the object detection model 542 is a dictionary for identifying an area in which an object (product) is captured from a captured image based on the input captured image. In other words, the object detection model 542 is a dictionary for detecting an image of a product (hereinafter referred to as an "object image") surrounded by an identified area from a captured image (an image in which a customer scans the product, which will be described later).

[0017] The feature extraction model 543 is a dictionary for calculating feature amounts effective for identifying a product shown in a specified area (object image). That is, the feature extraction model 543 outputs a feature vector indicating feature amounts indicating the features of a product whose area is specified, based on the input object image.

[0018] The matching dictionary 544 is a dictionary for identifying a product based on the output feature vector. In other words, the matching dictionary 544 stores information on feature amounts indicating the characteristics of a large number of products in association with product identification information that identifies each product, and calculates a similarity indicating how similar the large number of stored products are to the captured product by matching the feature amounts indicated by the input feature vector with the stored feature amounts. The higher the similarity of a product, the higher the degree of similarity of the product to the captured product (i.e., the higher the probability that it is the captured product). For example, a product with a similarity of 95% or more is highly likely to be the same product as the captured product.

[0019] In package recognition, an object detection model 542 is used to identify an area in which a product is captured from an image captured by an imaging unit such as a camera (object image A is detected in FIG. 2), and then a feature extraction model 543 is used to extract a feature of the product, which represents surface information such as the external shape, color, pattern, and unevenness of the product captured in the area (object image) as parameters (feature vectors) (feature B is extracted in FIG. 2), and then a matching dictionary 544 is used to match the extracted feature with feature data of multiple products stored in the dictionary ("match C" in FIG. 2), to calculate a similarity indicating how similar the stored product is to the product, and recognize the product according to the similarity. Based on the similarity calculated by such package recognition, if there is one product whose similarity is equal to or greater than a threshold (e.g., 95% or greater), the system 10 identifies the recognized product as the captured product (product A in FIG. 2) and, if there is no product whose similarity is higher than the threshold (e.g., all similarities are lower than 95%), a single product is not identified and multiple recognized products with high similarity are set as candidate products.

[0020] Returning to the explanation of Fig. 1, the imaging device 3 includes a first imaging section 37 (imaging section) and a second imaging section 38 (imaging section) (see Figs. 4 and 7), and is a device that captures an image of a commodity scanned in front of the imaging device 3. The first imaging section 37 and the second imaging section 38 are, for example, cameras in which a large number of CCDs (Charge Coupled Devices: imaging elements) are arranged two-dimensionally (planarly).

[0021] The imaging device 3 extracts product images showing products from images captured by the first imaging unit 37 and the second imaging unit 38. FIG. 3 shows an example of extracting product images from captured images. In FIG. 3, the imaging device 3 captures images of frames 1 to 9. In FIG. 3, frame 1 shows nothing, but in reality, the customer's hand for scanning the product and the background are shown, and the product is not shown. The same is true for frame 9. In contrast, frames 2 to 8 show the product in addition to the customer's hand and the background. Therefore, the imaging device 3 uses a known technique to extract images of frames 2 to 8 showing the product as product images from the captured images of frames 1 to 9.

[0022] As a method for extracting a product image from a captured image, for example, there is a method for extracting an image in which a product appears by distinguishing the product from a hand or background, but there are also other methods, such as a method for extracting images in which a product appears before and after an image in which a code symbol is recognized as the center, a method for extracting images in which a product appears by going back from an image in which a code symbol is recognized, a method for extracting images in which a product appears after an image in which a code symbol is recognized, etc. Also, a dictionary similar to the object detection model 542 may be provided in the imaging device 3, and an object image in which the area of ​​the product can be calculated using the dictionary may be extracted as the product image.

[0023] Furthermore, the imaging device 3 extracts a product image based on the images captured by the first imaging unit 37 and the second imaging unit 38, and transmits the extracted product image to the edge device 5. The imaging device 3 also executes code recognition based on the code symbol (code symbol S in FIG. 3) appearing in the extracted product image, and generates decoded information. The imaging device 3 then outputs product identification information included in the decoded information to the POS terminal 1. The imaging device 3 also transmits the generated decoded information to the edge device 5. Note that when performing code recognition, the imaging device 3 may use images captured by the first imaging unit 37 and the second imaging unit 38, instead of the product image.

[0024] The edge device 5 executes package recognition. That is, the edge device 5 detects an object image in which the area (range) of the product is specified using the object detection model 542 based on the product image extracted from the images captured by the first imaging unit 37 and the second imaging unit 38 (A in FIG. 2).

[0025] Next, the edge device 5 uses the feature extraction model 543 to calculate feature amounts indicating the features of the product from the detected object image, and outputs a feature vector of the product (B in FIG. 2). Then, the edge device 5 uses the matching dictionary 544 to match the stored product features based on the output feature vector (C in FIG. 2), and calculates a similarity indicating the degree of similarity between the product and the product. The edge device 5 then recognizes a product with a high similarity as the product that was imaged. The edge device 5 then outputs product identification information of the recognized product to the imaging device 3.

[0026] Furthermore, the edge device 5 creates association information that associates the product image output by the imaging device 3 with the decoded information. Then, the edge device 5 transmits the created association information to the store server 7 via the communication line L. Then, the edge device 5 receives information of the matching dictionary 744 learned based on the transmitted association information from the store server 7, overwrites the information in the matching dictionary 544, and stores it, thereby updating the stored matching dictionary 544 and improving the accuracy of identifying products using the matching dictionary 544.

[0027] The imaging device 3 determines a product for which product identification information is to be transmitted to the POS terminal 1, based on the product identification information input from the edge device 5 and the generated decoded information. For example, when the product identification information input from the edge device 5 is information indicating one specific product (i.e., when product identification information related to one identified product is input from the edge device 5), the imaging device 3 transmits the product identification information to the POS terminal 1 if the decoded information has not yet been generated. In addition, when the decoded information is generated before the product identification information is input from the edge device 5, the imaging device 3 transmits the product identification information included in the decoded information to the POS terminal 1.

[0028] One or more POS terminals 1 (three in the first embodiment) are installed in the checkout area in store T. The POS terminal 1 is a product sales data processing device that allows customers to perform product registration and accounting operations themselves. When a customer performs a product registration operation (for example, scanning a code symbol such as a barcode attached to a product or the appearance of the product and having the imaging device 3 capture an image of the product), the POS terminal 1 executes product registration processing related to the product to be sold based on the product identification information received from the imaging device 3. The POS terminal 1 also executes payment processing based on the customer's accounting operation (operation to end the product registration operation and start the accounting process).

[0029] The product registration process refers to a process of acquiring a product code (which may be the same as the product identification information) that identifies a product sold at store T, displaying product information (product name, product price, etc.) such as the product name and price of the product on display 18 (see Figs. 4 and 6) based on the acquired product code, and storing the product information in product information unit 131 (see Fig. 6). The payment process refers to a process of displaying the total amount on display 18 based on the product information stored in product information unit 131 in conjunction with the product registration process, processing payment by media such as cash, credit card, electronic money, etc., calculating change based on the deposit in the case of cash payment and displaying it on display 18, issuing a receipt from printer 22 (see Figs. 4 and 6) on which product information and accounting information (total amount, deposit amount, change amount, point information, etc.) of the product that has been paid for are printed.

[0030] Furthermore, the POS terminal 1 executes the cancellation process by performing a cancellation operation for the product for which the product registration process has been performed. The cancellation process is a process for canceling a product registration process that has been performed once. Specifically, if the POS terminal 1 executes the product registration process based on the product identification information of the product for which the customer has performed the product registration process based on package recognition, when the customer performs the cancellation operation, the POS terminal 1 executes the cancellation process for the product for which the cancellation operation has been performed. If the POS terminal 1 executes the product registration process based on the product identification information of the product for which the customer has performed the product registration process based on code recognition, when the customer performs the cancellation operation, the POS terminal 1 executes the cancellation process for the product for which the cancellation operation has been performed. If the POS terminal 1 executes the product registration process based on the product identification information of the product for which the customer has performed the product registration process based on code recognition, when the customer performs the cancellation operation, the POS terminal 1 executes the cancellation process for the product for which the cancellation operation has been performed. In other words, if the POS terminal 1 executes the product registration process based on the product identification information of the product for which the code recognition process has been performed, the POS terminal 1 does not execute the cancellation process for the product for which the cancellation operation has been performed based only on the cancellation operation performed by the customer.

[0031] The reliability of a product identified by package recognition (information recognized by package recognition) is generally lower than that of a product identified by code recognition (information recognized by code recognition). Therefore, the possibility of product misrecognition by package recognition is higher than that of code recognition. In addition, since customers often hold the side of a product with a code symbol over the imaging device 3, the number of products identified by code recognition tends to be higher than the number of products identified by package recognition. For this reason, the POS terminal 1 of the embodiment executes a cancellation process for the product when the customer himself performs a cancellation operation for information products recognized by package recognition, which has a lower reliability than code recognition, since the possibility of misrecognition is higher than that of code recognition, and for products identified by code recognition, when the customer performs a cancellation operation, a store clerk or the like is called and the cancellation process for the product is executed based on the cancellation operation of the store clerk or the like. Such a POS terminal 1 executes a cancellation process based on the cancellation operation performed by the customer only for package-recognized products that have a low reliability (high number of misrecognitions) but are processed less, so that it is possible to reduce the burden on store clerks and the like while preventing fraudulent acts by customers to a certain level.

[0032] The POS terminal 1 also transmits product information and transaction information (collectively referred to as "sales data") for the products for which payment has been processed to the store server 7. In this embodiment, the sales data is transmitted from the POS terminal 1 to the store server 7 when the transaction process is completed, but it may be transmitted at any timing, such as transmitting all data for one day at once when store T closes.

[0033] The store server 7 is installed in the back yard of the store T. The store server 7 collects sales data received from the POS terminal 1. The store server 7 manages the sales of products of the store T based on the collected sales data.

[0034] The store server 7 additionally stores the association information received from the edge device 5 in the matching dictionary 744 (see FIG. 9) stored therein. Specifically, the store server 7 uses the object detection model 742 (see FIG. 9) to extract an object image from the product image included in the association information received from the edge device 5, and uses the feature extraction model 743 (see FIG. 9) to extract a feature based on the object image. The store server 7 then adds the extracted feature to the matching dictionary 744 stored therein, based on the product identification information included in the decoded information associated with the product image, and stores the feature. In this way, the feature stored for each product identification information in the matching dictionary 744 is increased, thereby training the matching dictionary 744 and improving the product recognition accuracy by the matching dictionary 744. The store server 7 then transmits the trained information of the matching dictionary 744 to the edge device 5.

[0035] In the embodiment, the object detection model 742, the feature extraction model 743, and the matching dictionary 744 are provided in the store server 7, but the object detection model 742, the feature extraction model 743, and the matching dictionary 744 may be provided in a server other than the store server 7 (for example, a headquarters server installed in a headquarters that manages and oversees multiple store servers 7).

[0036] Next, the checkout device 9 will be described. The checkout device 9 is composed of a POS terminal 1, an imaging device 3, and an edge device 5. Fig. 4 is a perspective view showing the appearance of the checkout device 9. As shown in Fig. 4, the checkout device 9 has a main body 42 and a basket placement stand 4. The main body 42 incorporates a circuit board and a power source that control the POS terminal 1. The basket placement stand 4 is attached to one side of the main body 42, and is a stand on which a basket containing products is placed.

[0037] The top surface of the main body 42 has a flat product placement section 41. The main body 42 has a pair of support posts 45 extending upward at the rear of the product placement section 41. A printer 22 that issues receipts is attached to the support posts 45 on the side farther from the basket placement platform 4. A card reader 21 that reads information from cards such as credit cards and point cards is attached to the top of the printer 22.

[0038] Additionally, an imaging device 3 is attached to the support post 45 on the side closer to the basket placement platform 4. The imaging device 3 is installed with its reading surface facing the customer, and a first imaging section 37 provided inside the imaging device 3 captures an image of a product by scanning the product that the customer has removed from the basket. Additionally, since customers tend to aim the code symbol attached to the product at the imaging device 3 to scan the product, the first imaging section 37 often simultaneously captures an image of the code symbol attached to the product. In the following embodiments, it is assumed that the first imaging section 37 captures an image of the code symbol.

[0039] Further, the second imaging section 38, which is a part of the imaging device 3, is attached above the imaging device 3. The second imaging section 38 is installed such that its reading surface faces diagonally downward from the imaging device 3, and images the scanned commodity from a different angle than the first imaging section 37. In other words, the second imaging section 38 images a surface of the commodity different from the surface imaged by the first imaging section 37.

[0040] A display 18 is provided between the pair of support columns 45. The display 18 is a display made of, for example, liquid crystal, and is installed facing a customer who operates the POS terminal 1. The display 18 is located slightly above the imaging device 3 and the printer 22, and at the same height as the card reader 21 and the second imaging unit 38.

[0041] The display 18 displays product information (product name, product price, etc.) of the registered product and payment information (total amount, payment amount, change amount, etc.) of the payment processed. An operation unit 17 consisting of a touch panel is provided on the display surface of the display 18.

[0042] In addition, between the pair of support columns 45 and below the display 18, there are provided a temporary placement table 44 for temporarily placing products, and a pair of attachment parts 43 for attaching handles of a shopping bag for containing products that have undergone product registration processing. The products placed in the shopping bag are placed on the product placement part 41.

[0043] Furthermore, a substantially cylindrical pole 6 extending upward is attached to the support 45 on the side closer to the car placement platform 4. A patrol lamp 61 is attached to the pole 6.

[0044] The patrol lamp 61 lights up or flashes when an abnormality occurs in the POS terminal 1 (for example, when the receipt paper runs out) or when a customer calls a store attendant.

[0045] The edge device 5 constituting the checkout apparatus 9 is provided alongside the store server 7 or installed on the cloud.

[0046] In such a checkout device 9, a customer performs a product registration operation by removing an item from a basket placed on the basket placement table 4 and passing (scanning) it in front of the imaging device 3 to capture an image. The customer then places the item for which the product registration operation has been performed into a shopping bag attached to the mounting unit 43. The imaging device 3 captures an image of the scanned item using the first imaging unit 37 and the second imaging unit 38. The POS terminal 1 performs product registration processing for the item for which the product registration operation has been performed, based on the product identification information received from the imaging device 3. The customer who has performed the product registration operation for all items then performs a checkout operation by operating the checkout button. When the checkout operation has been performed, the POS terminal 1 performs a payment process for the items for which the product registration operation has been performed.

[0047] Next, a description will be given of the flow of processing in the system 10. Fig. 5 is a sequence diagram showing the flow of processing in the system 10. As shown in Fig. 5, the system 10 includes a POS terminal 1, an imaging device 3, an edge device 5, and a store server 7.

[0048] When the start key 171 (see FIG. 6) is operated by a customer on the POS terminal 1 (T1), the POS terminal 1 transmits an imaging start signal to the imaging device 3 to start imaging (T2). The imaging device 3 that has received the imaging start signal captures an image of the product registration operation performed by the customer using the first imaging unit 37 and the second imaging unit 38 (T3). Next, the imaging device 3 transmits a product image extracted based on the captured image information to the edge device 5 (T4).

[0049] The edge device 5 that has received the product image performs general object recognition (T5). That is, the edge device 5 uses the object detection model 542 to generate an object image showing the product by identifying the area of ​​the product included in the product image, uses the feature extraction model 543 to extract a feature vector showing the features of the product based on the object image, and uses the matching dictionary 544 to calculate the similarity of the product to the captured product, and extracts products that are similar (highly similar) to the product. At T5, the edge device 5 identifies one product if the calculated similarity is equal to or greater than a threshold. Then, the edge device 5 outputs product identification information of the identified product to the imaging device 3 (T7).

[0050] On the other hand, if the captured image contains a code symbol, the imaging device 3 that captured the image at T3 decodes the code symbol and generates decoded information including the product identification information (T6). Note that the order of input of the product identification information to the imaging device 3 at T7 and generation of the decoded information at T6 changes depending on the timing at which the code symbol is captured. For example, if the customer does not correctly face the code symbol toward the first imaging unit 37, the capture of the code symbol will be delayed, and the generation of the decoded information will be delayed. In this case, the input of the product identification information from the edge device 5 may be early. Conversely, if the image of the code symbol is captured early, the generation of the decoded information may be earlier than the input of the product identification information from the edge device 5.

[0051] If the decoded information has not been generated at the timing (T7) when the product identification information is input from the edge device 5, the imaging device 3 transmits the product identification information to the POS terminal 1 without waiting for the generation of the decoded information (T8). In this case, the imaging device 3 transmits the product identification information to the POS terminal 1 to which additional information (added by the edge device) indicating that the product identification information is the product identification information input from the edge device 5 (i.e., the package-recognized product) has been added. For example, when the product identification information input from the edge device 5 is "4911111", for example, the product identification information is transmitted to the POS terminal 1 as product identification information indicated as "P4911111" with additional information (code) of "P" added to the beginning of the product identification information.

[0052] Conversely, if the decoded information has been generated before the product identification information is input from the edge device 5, the imaging device 3 transmits the product identification information included in the decoded information to the POS terminal 1 (T8) without waiting for the product identification information to be input from the edge device 5. In this case, the imaging device 3 adds additional information indicating that the product identification information is the product identification information included in the decoded information generated by decoding the code symbol to the product identification information and transmits the product identification information to the POS terminal 1. For example, if the product identification information included in the decoded information is "4911111", for example, additional information (code) of "F" is added to the beginning of the product identification information and transmitted to the POS terminal 1 as product identification information indicated as "F4911111".

[0053] The POS terminal 1 executes a product registration process for the product based on the product identification information received in T8 (T9). The POS terminal 1 also displays product information (product name, product price, etc.) related to the product for which the product registration process has been performed (T10). When displaying this information, the POS terminal 1 distinguishes between whether the product for which the product registration process has been performed is a package-recognized product or a code-recognized product so that the user can see the information.

[0054] When a customer performs a cancel operation on a displayed package-recognized product, the POS terminal 1 executes a cancel process for that product (T11). Also, when a customer performs a call operation on a displayed code-recognized product, the POS terminal 1 executes a call process for a store clerk or the like, and executes a cancel process for the product that has been canceled by the store clerk or the like (T11).

[0055] When the closing key 172 (see FIG. 6) is operated (T12), the POS terminal 1 executes the payment process for the product that has been registered (T13). Then, the POS terminal 1 outputs an imaging end signal to the imaging device 3 to end imaging (T14). The imaging device 3 that has received the imaging end signal ends imaging by the first imaging unit 37 and the second imaging unit 38 (T15).

[0056] The POS terminal 1 also transmits the decoded information generated in T6 to the edge device 5 (T21). The edge device 5, which has received the decoded information, generates association information that associates the product identification information included in the decoded information with the product image received in T4 (T22). The edge device 5 then transmits the generated association information to the store server 7 (T23). The store server 7 updates the matching dictionary 744 by machine learning based on the received association information (T24). The processing of T24 improves the accuracy of the matching dictionary 744 in identifying products. The store server 7 then transmits information about the updated matching dictionary 744 to the edge device 5 (T25). The edge device 5 overwrites and updates the matching dictionary 544 based on the information received in T25 (T26). The processing of T26 allows the accuracy of the matching dictionary 544 for identifying products to be routinely improved.

[0057] Next, the hardware configuration of the POS terminal 1 will be described. FIG. 6 is a block diagram showing the hardware configuration of the POS terminal 1. As shown in FIG. 6, the POS terminal 1 includes a CPU (Central Processing Unit) 11, which is an example of a processor, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a memory unit 14, and the like. The CPU 11 is the main controller of the POS terminal 1. The ROM 12 stores various programs. The RAM 13 expands programs and various data. The memory unit 14 stores various programs. The CPU 11, the ROM 12, the RAM 13, and the memory unit 14 are connected to one another via a bus 15. The CPU 11, the ROM 12, and the RAM 13 constitute a control unit 100. That is, the control unit 100 executes a control process of the POS terminal 1, which will be described later, by the CPU 11 operating in accordance with a control program stored in the ROM 12 or the memory unit 14 and expanded in the RAM 13.

[0058] The RAM 13 includes a product information section 131. The product information section 131 stores product information (product code that identifies the product, product name, price, etc.) of a product that has been registered.

[0059] The memory unit 14 is composed of a non-volatile memory such as a hard disk drive (HDD) or a flash memory that retains stored information even when the power is turned off, and has a control program unit 141 that stores a control program, a product master 142, and a log information unit 143. The product master 142 stores product information of each product in association with product identification information that identifies the product. The log information unit 143 stores all logs of the coded product identification information input to the POS terminal 1 from the imaging device 3. That is, the log information unit 143 newly stores and accumulates the coded product identification information input to the POS terminal 1 from the imaging device 3 every time it is input. Based on the information stored in the log information unit 143, it is possible to analyze the ratio of package-recognized products and code-recognized products among the products related to the product identification information input to the POS terminal 1, the ratio of package recognition and code recognition for each product, and the like.

[0060] In addition, the control unit 100 is connected to an operation unit 17 , a display unit 18 , a card reader 21 , and a printer 22 via a bus 15 and a controller 16 .

[0061] The operation unit 17 includes, for example, a touch panel provided on the display 18, and has a start key 171, a closing key 172, and a cancel button 173. The start key 171 is operated when a customer starts operating the POS terminal 1. The closing key 172 is a key operated when terminating the product registration process in the POS terminal 1 and proceeding to the payment process. The cancel button 173 is operated when canceling a package-recognized product that has been registered by the customer and processed by the POS terminal 1. The cancel button 173 is a button provided on a cancel button display 1813 described later.

[0062] The display 18 is, for example, a liquid crystal display, and displays information to a customer who is an operator operating the POS terminal 1. The display 18 displays, for example, product information (product name, product price, etc.) of products that have been registered, and accounting information related to the payment process, for each product.

[0063] The card reader 21 reads card information from a card, for example, a credit card, a debit card, or an electronic money card when performing payment processing. The card reader 21 also reads and writes point information from a point card. The printer 22 is, for example, a thermal printer, and issues a receipt on which receipt information including product information and payment information relating to the product for which payment processing has been performed is printed.

[0064] The control unit 100 is also connected to a communication unit 23 (interface) via a bus 15. The communication unit 23 is communicatively connected to the edge device 5 and the store server 7 via a communication line L. The communication unit 23 is also communicatively connected to the imaging device 3 via a cable U.

[0065] Next, the hardware configuration of the imaging device 3 will be described. FIG. 7 is a block diagram showing the hardware configuration of the imaging device 3. As shown in FIG. 7, the imaging device 3 includes a CPU 31, which is an example of a processor, a ROM 32, a RAM 33, a memory unit 34, and the like. The CPU 31 is the main controller of the imaging device 3. The ROM 32 stores various programs. The RAM 33 expands programs and various data. The memory unit 34 stores various programs. The CPU 31, the ROM 32, the RAM 33, and the memory unit 34 are connected to each other via a bus 35. The CPU 31, the ROM 32, and the RAM 33 constitute a control unit 300. That is, the control unit 300 executes a control process of the imaging device 3, which will be described later, by the CPU 31 operating according to a control program stored in the ROM 32 or the memory unit 34 and expanded in the RAM 33.

[0066] The RAM 33 has an image storage unit 331, a product image unit 332, a decoded information unit 333, and a product identification information unit 334. The image storage unit 331 stores an image captured by the first imaging unit 37 and an image captured by the second imaging unit 38. The product image unit 332 stores a product image extracted from the image stored in the image storage unit 331. The decoded information unit 333 stores decoded information obtained by decoding a code symbol included in the image captured by the first imaging unit 37. The product identification information unit 334 stores the product identification information input from the edge device 5.

[0067] The memory unit 34 is configured with a non-volatile memory such as an HDD or a flash memory in which stored information is retained even when the power is turned off, and has a control program unit 341. The control program unit 341 stores a control program for driving the imaging device 3.

[0068] In addition, the control unit 300 is connected to a first imaging unit 37 and a second imaging unit 38 via a bus 35 and a controller 36.

[0069] The control unit 300 is also connected to a communication unit 39 via a bus 35. The communication unit 39 is communicatively connected to the edge device 5 and the store server 7 via a communication line L. The communication unit 39 is also communicatively connected to the POS terminal 1 via a cable U.

[0070] Next, the hardware configuration of the edge device 5 will be described. FIG. 8 is a block diagram showing the hardware configuration of the edge device 5. As shown in FIG. 8, the edge device 5 includes a CPU 51, which is an example of a processor, a ROM 52, a RAM 53, a memory unit 54, and the like. The CPU 51 is a main controller of the edge device 5. The ROM 52 stores various programs. The RAM 53 expands programs and various data. The memory unit 54 stores various programs. The CPU 51, the ROM 52, the RAM 53, and the memory unit 54 are connected to each other via a bus 55. The CPU 51, the ROM 52, and the RAM 53 constitute a control unit 500. That is, the control unit 500 executes a control process of the edge device 5, which will be described later, by the CPU 51 operating according to a control program stored in the ROM 52 or the memory unit 54 and expanded in the RAM 53.

[0071] The RAM 53 has an object image storage unit 531, a feature amount unit 532, a product identification information unit 533, and an association information unit 534. The object image storage unit 531 stores an object image extracted based on a product image received from the imaging device 3. The feature amount unit 532 stores a feature vector indicating the feature amount of the product calculated by the feature extraction model 543 based on the object image stored in the object image storage unit 531. The product identification information unit 533 compares the feature vector calculated by the feature extraction model 543 with the feature amount of the product stored in the comparison dictionary 544, calculates the similarity of a product close to the feature vector, and stores product identification information of one product having a high similarity. The association information unit 534 stores association information that associates the product image and decoded information related to the same product input from the imaging device 3.

[0072] The memory unit 54 is composed of a non-volatile memory such as an HDD or a flash memory in which stored information is retained even when the power is turned off, and has a control program unit 541, an object detection model 542, a feature extraction model 543, and a matching dictionary 544. The control program unit 541 stores a control program for driving the edge device 5.

[0073] The control unit 500 is also connected to the bus 55 and the controller 56. The control unit 500 is also connected to a communication unit 57 via the bus 55. The communication unit 57 is connected to the POS terminal 1, the imaging device 3, and the store server 7 via a communication line L so as to be able to communicate with them.

[0074] Next, the hardware configuration of the store server 7 will be described. FIG. 9 is a block diagram showing the hardware configuration of the store server 7. As shown in FIG. 9, the store server 7 includes a CPU 71, which is an example of a processor, a ROM 72, a RAM 73, a memory unit 74, and the like. The CPU 71 is the main controller of the store server 7. The ROM 72 stores various programs. The RAM 73 expands programs and various data. The memory unit 74 stores various programs. The CPU 71, the ROM 72, the RAM 73, and the memory unit 74 are connected to one another via a bus 75. The CPU 71, the ROM 72, and the RAM 73 constitute a control unit 700. That is, the control unit 700 executes a control process of the store server 7, which will be described later, by the CPU 71 operating in accordance with a control program stored in the ROM 72 or the memory unit 74 and expanded in the RAM 73.

[0075] The RAM 73 has a sales information section 731 and an association information section 732. The sales information section 731 stores sales data received from the POS terminal 1 and manages sales of the store T. The association information section 732 stores association information received from the edge device 5.

[0076] The memory unit 74 is composed of a non-volatile memory such as an HDD or flash memory in which stored information is retained even when the power is turned off, and has a control program unit 741, an object detection model 742, a feature extraction model 743, and a matching dictionary 744. The control program unit 741 stores a control program for driving the store server 7. The object detection model 742 is a dictionary with the same content as the object detection model 542, the feature extraction model 743 is a dictionary with the same content as the feature extraction model 543, and the matching dictionary 744 is a dictionary with the same content as the matching dictionary 544.

[0077] In addition, the control unit 700 is connected to a display unit 77 and an operation unit 78 via the bus 55 and the controller 56. The display unit 77 is, for example, a liquid crystal display, and displays information to an operator who operates the store server 7. The operation unit 78 is, for example, a keyboard, and is operated by the operator of the store server 7.

[0078] The control unit 700 is also connected to a communication unit 79 via a bus 75. The communication unit 79 is connected to the POS terminal 1, the imaging device 3, and the edge device 5 via a communication line L so as to be able to communicate with each other.

[0079] From here, we will explain the functional configuration of the POS terminal 1. Fig. 10 is a functional block diagram showing the functional configuration of the POS terminal 1. The control unit 100 of the POS terminal 1 functions as a determination means 101, a display means 102, a cancellation operation determination means 103, a cancellation means 104, and a cancellation display means 105 by the CPU 11 following the programs stored in the ROM 12 and the control programs stored in the control program section 141 of the memory section 14.

[0080] The determination means 101 determines whether the product identification information input from the imaging device 3 is product identification information recognized by package recognition in the edge device 5 (second product identification information) or product identification information recognized by code recognition in the imaging device 3 (first product identification information). The determination means 101 determines whether the product identification information input from the imaging device 3 is information recognized by package recognition in the edge device 5 or information recognized by code recognition in the imaging device 3, based on whether information recognized by package recognition (e.g., "P") or information recognized by code recognition (e.g., "F") is added to the product identification information input from the imaging device 3. If information recognized by package recognition (e.g., "P") is added, the determination means 101 determines that the product identification information is information recognized by package recognition. If information recognized by code recognition (e.g., "F") is added, the determination means 101 determines that the product identification information is information recognized by code recognition.

[0081] The display means 102 displays the product information related to the first product identification information and the product information related to the second product identification information, which have been subjected to the product registration process, on the display 18 in a manner that enables distinction between the information related to the first product identification information and the information related to the second product identification information. Specifically, when the determination means 101 determines that the information is related to the first product identification information, the display means 102 displays the product information related to the product identification information (product name, unit price, sales volume, amount) on the display 18, and when the determination means 101 determines that the information is related to the second product identification information, the display means 102 displays the product information related to the product identification information and a display indicating that the product is a product recognized by package recognition (specifically, a letter "P" 1812 indicating that the product is a product recognized by package recognition and a display 1813 of a cancel button 173 operated to cancel the product (see Figs. 12 and 13 for both)) in the same row as the product information. Note that the display means 102 does not display, on the product information related to the first product identification information, a display indicating that the information is information recognized by code recognition, and a display 1813 of the cancel button 173 to be operated when canceling the product. Therefore, in the embodiment, the information of the product related to the first product identification information and the information of the product related to the second product identification information that have been subjected to the product registration process are displayed in a list format on the display 18.

[0082] Therefore, the customer can recognize that a product that displays a display 1813 indicating that the information has been recognized by package recognition and a cancel button 173 is a package-recognized product, and that a product that does not display a display 1813 indicating that the information has been recognized by package recognition and a cancel button 173 is a code-recognized product.

[0083] The cancel operation determination means 103 determines whether the cancel button 173 has been operated.

[0084] The cancellation means 104 enables a customer to cancel a product registration process for a product related to the second product identification information input to the communication unit 23 after the product registration process of the product, and disables a customer to cancel a product registration process for a product related to the first product identification information input to the communication unit 23 after the product registration process of the product. Specifically, when a customer operates the cancel button 173 for a product related to the second product identification information for which product registration processing has been performed, the cancellation means 104 executes a cancellation process for the product. The cancellation means 104 disables a customer to cancel a product related to the first product identification information for which product registration processing has been performed.

[0085] When a customer wishes to cancel a product related to the first product identification information that has been registered, the customer performs a call operation to call a store clerk or the like at the POS terminal 1. For example, the customer calls a store clerk or the like by operating a button provided on the POS terminal 1 for calling a store clerk or the like. Alternatively, the POS terminal 1 may perform a process to call a store clerk or the like when the customer touches the field of the product related to the first product identification information that the customer wishes to cancel (corresponding to a call operation). When such a call operation is performed, it is desirable for the POS terminal 1 to display a message such as "A store clerk is being called, please wait for a while." The called store clerk or the like performs a cancellation process for the product related to the first product identification information at the POS terminal 1. In this case, for example, the store clerk or the like may read a store clerk code that identifies the store clerk or the like, and the control unit 100 of the POS terminal 1 may perform a cancellation process if the store clerk code is confirmed.

[0086] When a cancellation process is executed for a product related to the first product specifying information, the cancellation display means 105 displays information indicating that the product has been cancelled in association with the display of the product for which the cancellation process has been executed.

[0087] From here, the control of the POS terminal 1 will be described. Fig. 11 is a flowchart showing the flow of the control process of the POS terminal 1. As shown in Fig. 11, the control unit 100 of the POS terminal 1 judges whether the start key 171 indicating the start of a transaction has been operated by a customer (S11). The control unit 100 waits until the operation is performed (No in S11), and if it is determined that the start key 171 has been operated by a customer (Yes in S11), it transmits an imaging start signal to the imaging device 3 to start imaging by the first imaging unit 37 and the second imaging unit 38 (S12).

[0088] Next, the control unit 100 judges whether product identification information for identifying a product has been received from the imaging device 3 (S13). If it is judged that the product identification information has been received (Yes in S13), the control unit 100 executes a product registration process for the product based on the received product identification information (S14).

[0089] Next, the determination means 101 determines whether the product identification information related to the product that has been registered is the product identification information (second product identification information) recognized by package recognition in the edge device 5 or the product identification information (first product identification information) recognized by code recognition in the imaging device 3 (S15). If it is determined that the product identification information is recognized by package recognition (Yes in S15), the display means 102 displays the product information related to the product identified by the product identification information, the character "P" 1812 indicating that the product identification information is information recognized by package recognition, and the cancel button 173 operated when canceling the product, in association with each other on the display 18 (S16). If it is determined that the product identification information is not recognized by package recognition (i.e., the product identification information is recognized by code recognition) (No in S15), the display means 102 displays the product information related to the product identified by the product identification information on the display 18 (S17).

[0090] FIG. 12 shows an example of a list of registered products displayed in S16 and S17. As shown in FIG. 12, in the case of a package-recognized product, the display 18 displays a list 181 of registered products, a product name 182 and a price 183 of a product that has been registered most recently, and a total number and a total price 184 of products that have been registered up to now. In the column of each product in the list 181, the product name 1811 of the product, a display "P" 1812 indicating that the product is identified by package recognition, a display 1813 of a cancel button 173 operated by the customer when canceling the product, a purchase quantity 1814 and a unit price 1815 of the product, and a price 1816 of the product are displayed in one line in a mutually associated manner. In the case of a code-recognized product, the display 18 displays a list 181 of registered products, a product name 182 and a price 183 of a product that has been registered most recently, and a total number and a total price 184 of products that have been registered up to now. In the column of each product in the list 181, the product name 1811, the purchase quantity 1814 and unit price 1815 of the product, and the amount 1816 of the product are displayed in one line in correspondence with each other. In the case of a coded product, the display "P" 1812 indicating that the information is recognized by package recognition and the display 1813 of the cancel button 173 operated by the customer when canceling the product are not displayed.

[0091] 12, the display "P" 1812 and the display 1813 of the cancel button 173 are displayed in association with the product A and the product G, respectively, and therefore it can be seen that the product A and the product G are package-recognized products. In addition, the display "P" 1812 and the display 1813 of the cancel button 173 are not displayed for the products other than the product A and the product G (product B, product C, product D, product E, product F, and product H), and therefore it can be seen that the products are code-recognized products.

[0092] Returning to the description of FIG. 11, when it is determined that the product identification information has not been received from the imaging device 3 (No in S13), or after the processes of S16 and S17 have been executed, the cancellation operation determination means 103 determines whether the cancel button 173 has been operated (S21). When the cancel button 173 has been operated, the cancellation means 104 executes a cancellation process of the product registration process for the product for which the cancel button 173 has been operated (S22). Since the product for which the cancel button 173 has been operated is a package-recognized product, cancellation can be performed in response to a customer's purchase operation. Then, the cancellation display means 105 displays information indicating that cancellation of the product registration process has been performed in association with the product (S23).

[0093] Fig. 13 shows an example of a product list displaying information indicating that cancellation processing has been performed in S23. As shown in Fig. 13, a "CANCELLED" display 1818 is displayed in association with product A, indicating that cancellation processing has been performed to cancel the product registration processing for that product. Also, a strikethrough line 1817 is displayed in the row for product A, indicating that cancellation processing has been performed for that product. These displays indicate that the product registration processing for product A has been canceled.

[0094] Returning to the explanation of Fig. 11, if it is determined that the operation was not an operation of the cancel button 173 (No in S21), the control unit 100 determines whether or not the customer has performed an operation to call a store clerk or the like (S24). If the customer wishes to cancel the product whose code has been recognized, the customer performs an operation to call a store clerk or the like. If it is determined that an operation to call a store clerk or the like has been performed (Yes in S24), the control unit 100 executes a process to call the store clerk or the like who manages the POS terminal 1 (S25).

[0095] If the customer has not called a store clerk or the like (No in S24), or after the process of S23 or S25 has been executed, the control unit 100 judges whether the closing key 172 has been operated (S31). If it is judged that the closing key 172 has not been operated (No in S31), the control unit 100 returns to S13 and judges whether to input product identification information for the next product. If it is judged that the closing key 172 has been operated (Yes in S31), the control unit 100 executes a payment process for the product based on the product information stored in the product information unit 131 (S32). Then, the control unit 100 transmits the product information and payment information for the product for which payment has been processed to the store server 7 (S33). In addition, in S33, the control unit 100 stores log information (product identification information including at least additional information indicating that the code has been recognized and additional information indicating that the package has been recognized) for the product for which payment has been processed in the log information unit 143. Then, the control unit 100 transmits an imaging end signal to the imaging device 3 to end imaging by the first imaging unit 37 and the second imaging unit 38 (S34).

[0096] According to the POS terminal 1 configured as above, for a package-recognized product among the registered products, the cancellation process is executed based on the cancellation operation of the product performed by the customer himself / herself. Therefore, it is possible to reduce the burden on the store clerk, etc., involved in the cancellation process.

[0097] From here, a description will be given of the functional configuration of the imaging device 3. Fig. 14 is a functional block diagram showing the functional configuration of the imaging device 3. The control unit 300 of the imaging device 3 functions as a first generating means 301 and a first output means 302 as a result of the CPU 31 following a program stored in the ROM 32 and a control program stored in a control program unit 341 of the memory unit 34.

[0098] The first generating means 301 decodes a code symbol included in the captured image to generate first product identification information related to a specific product. Specifically, when a code symbol is included in the image captured by the first imaging unit 37 and the second imaging unit 38, the first generating means 301 generates decoded information (including the first product identification information) by decoding the code symbol.

[0099] The first output means 302 outputs the generated first product identification information to the POS terminal 1. Specifically, the first output means 302 outputs to the POS terminal 1 the first product identification information obtained by adding additional information indicating that the code has been recognized to the generated first product identification information.

[0100] Furthermore, the first output unit 302 outputs to the POS terminal 1 second product identification information output from the edge device 5 by a second output unit 502 described later.

[0101] Next, the control of the imaging device 3 will be described. Fig. 15 is a flowchart showing the flow of the control process of the imaging device 3. As shown in Fig. 15, the control unit 300 of the imaging device 3 judges whether an imaging start signal has been received from the POS terminal 1 (S41). The control unit 300 waits until the imaging start signal is received (No in S41), and if it is determined that the imaging start signal has been received from the POS terminal 1 (Yes in S41), the control unit 300 starts the first imaging unit 37 and the second imaging unit 38 to start imaging (S42).

[0102] Next, the control unit 300 judges whether an imaging end signal has been received from the POS terminal 1 (S43). If it is judged that an imaging end signal has not been received (No in S43), the control unit 300 stores the images captured by the first imaging unit 37 and the second imaging unit 38 in the image storage unit 331 (S44). Then, the control unit 300 extracts a product image showing a product from the images stored in the image storage unit 331 (S45). Then, the control unit 300 stores the extracted product image in the product image unit 332. Then, the control unit 300 transmits the product image stored in the product image unit 332 to the edge device 5 (S46).

[0103] Next, the control unit 300 judges whether a code symbol has been detected from the product image stored in the product image section 332 (S51). If it is judged that a code symbol has been detected (Yes in S51), the first generating means 301 decodes the detected code symbol to generate decoded information including the first product identification information (S52). Then, the control unit 300 stores the generated decoded information in the decoded information section 333 (S52).

[0104] Next, the control unit 300 outputs the decoded information stored in the decoded information unit 333 to the edge device 5 (S53). Next, the control unit 300 judges whether the second product identification information received from the edge device 5 has already been output to the POS terminal 1 (S54). When the control unit 300 transmits the second product identification information to the POS terminal 1 in S58 described later, the control unit 300 judges that the second product identification information has already been output to the POS terminal 1. When the control unit 300 judges that the second product identification information received from the edge device 5 has not been output to the POS terminal 1 (No in S54), the first output unit 302 outputs the first product identification information obtained by adding additional information indicating that the code has been recognized to the first product identification information included in the decoded information to the POS terminal 1 (S55). Then, the control unit 300 returns to S43.

[0105] In addition, in S51, if it is determined that the code symbol is not detected from the product image (No in S51), the control unit 300 then determines whether or not the second product identification information has been received from the edge device 5 (S56). If it is determined that the second product identification information has not been received from the edge device 5 (No in S56), the control unit 300 returns to S51. If it is determined that the second product identification information has been received from the edge device 5 (Yes in S56), the control unit 300 then determines whether or not the first product identification information included in the decoded information generated by the first generating means 301 has been output to the POS terminal 1 (S57). If the control unit 300 has transmitted the first product identification information to the POS terminal 1 in S55, it determines that the first product identification information has been output to the POS terminal 1. If it is determined that the first product identification information has not been output to the POS terminal 1 (No in S57), the control unit 300 outputs the second product identification information received from the edge device 5 to the POS terminal 1 (S58). Then, the control unit 300 returns to S43.

[0106] If the second product identification information has already been output to the POS terminal 1 (Yes in S54), or if the first product identification information has been output to the POS terminal 1 (Yes in S57), the control unit 300 returns to S43.

[0107] Furthermore, when it is determined in S43 that an imaging end signal has been received from the POS terminal 1 (Yes in S43), the control unit 300 ends imaging by the first imaging unit 37 and the second imaging unit 38 (S59). Then, the control unit 300 ends the process and returns to S41.

[0108] From here, a functional configuration of the edge device 5 will be described. Fig. 16 is a functional block diagram showing the functional configuration of the edge device 5. The control unit 500 of the edge device 5 functions as a second generating means 501 and a second output means 502 as the CPU 51 follows a program stored in the ROM 52 and a control program stored in a control program unit 541 of the memory unit 54.

[0109] The second generating means 501 generates second product identification information related to a specific product by using a package recognition technique for the captured image. Specifically, the second generating means 501 performs package recognition based on the images captured by the first imaging unit 37 and the second imaging unit 38, and generates second product identification information related to the product.

[0110] The second output means 502 outputs the generated second product identification information to the imaging device 3. Specifically, the second output means 502 outputs to the imaging device 3 the second product identification information generated by the second generation means 501 and to which additional information indicating that the package has been recognized has been added.

[0111] Next, the control of the edge device 5 will be described. FIG. 17 is a flowchart showing the flow of the control process of the edge device 5. As shown in FIG. 17, the control unit 500 of the edge device 5 judges whether a product image has been received from the imaging device 3 (S71). If it is judged that a product image has been received from the imaging device 3 (Yes in S71), the control unit 500 uses the object detection model 542 to identify a product area in the received product image (S72). Next, the control unit 500 stores an object image surrounded by the identified area in the object image storage unit 531 (S73).

[0112] Next, the control unit 500 calculates a feature vector of the object image stored in the object image storage unit 531 using the feature extraction model 543 (S74). Then, the control unit 500 stores the calculated feature vector in the feature amount unit 532 (S75). Next, the second generation means 501 uses the comparison dictionary 544 to compare the feature vector stored in the feature amount unit 532 with the feature amount stored for each item identification information stored in the comparison dictionary 544, and extracts the item identification information of the item with the highest similarity (S76). That is, the second generation means 501 generates the second item identification information. Then, the control unit 500 stores the extracted second item identification information in the item identification information unit 533. Then, the second output means 502 transmits the second item identification information, which is the item identification information stored in the item identification information unit 533 and additional information indicating that the package has been recognized, to the imaging device 3 (S77). Then, the control unit 500 returns to S71.

[0113] Furthermore, in S71, if it is determined that the product image has not been received from the imaging device 3 (No in S71), it is determined whether decoded information has been received from the imaging device 3 (S81). If it is determined that decoded information has been received (Yes in S81), the control unit 500 generates association information that associates the received decoded information with the product image received in Yes in S71 (S82). Then, the control unit 500 stores the generated association information in the association information unit 534 (S83). Then, the control unit 500 transmits the association information stored in the association information unit 534 to the store server 7 (S84). Then, the control unit 500 returns to S71.

[0114] Also, when it is determined that the decoded information is not received from the imaging device 3 (No in S81), the control unit 500 determines whether or not information on the matching dictionary 744 is received from the store server 7 (S85). The information on the matching dictionary 744 received in S85 is information that is additionally stored and learned in the store server 7 as associated information. When it is determined that the information on the matching dictionary 744 is received from the store server 7 (Yes in S85), the control unit 500 overwrites the received information on the matching dictionary 744 in the matching dictionary 544 and stores it (S86). In this way, the matching dictionary 544 is trained, and the accuracy of the matching dictionary 544 when calculating the similarity is improved. Then, the control unit 500 returns to S71. When it is determined that the information on the matching dictionary 744 is not received from the store server 7 (No in S85), the control unit 500 returns to S71.

[0115] Next, the control of the store server 7 will be described. FIG. 18 is a flowchart showing the flow of the control process of the store server 7. As shown in FIG. 18, the control unit 700 of the store server 7 judges whether sales data has been received from the POS terminal 1 (S91). If it is judged that sales data has been received from the POS terminal 1 (Yes in S91), the control unit 700 adds the received sales data to the sales information unit 731 and stores it (S92), thereby managing the sales of the store T. Then, the control unit 700 returns to S91. In addition, in S92, the POS terminal 1 receives and stores the log information stored in the log information unit 143. That is, the store server 7 collects log information from all the POS terminals 1 connected thereto. In this way, the store server 7 can analyze information such as the ratio of code-recognized products and package-recognized products for products sold in the store T, and the ratio of code-recognized products and package-recognized products for each product.

[0116] Furthermore, if it is determined that the sales data has not been received from the POS terminal 1 (No in S91), the control unit 700 determines whether association information has been received from the edge device 5 (S93). If it is determined that association information has not been received (No in S93), the control unit 700 returns to S91, and if it is determined that association information has been received from the edge device 5 (Yes in S93), the control unit 700 stores the received association information in the association information unit 732 (S94).

[0117] Next, the control unit 700 updates the matching dictionary 744 based on the associated information stored in the associated information unit 732 (S95). That is, the control unit 700 performs general object recognition on the product image (which may be an object image) associated with the product identification information as the associated information. Specifically, the control unit 700 uses the object detection model 742 to specify the area of ​​the product image, and uses the feature extraction model 743 to extract a feature amount indicating the feature of the object image of the product whose area has been specified. Then, the control unit 700 additionally stores the extracted feature amount in association with the same product identification information as the product identification information associated with the product image stored in the matching dictionary 744. In this way, the feature amount associated with the product identification information is added to the matching dictionary 744, and the matching dictionary 744 is made to learn the feature amount, thereby improving the accuracy of the matching dictionary 744.

[0118] Then, the control unit 700 transmits the learned information of the matching dictionary 744 to the edge device 5 (S96). Then, the control unit 700 returns to S91. The edge device 5 that has received the information of the matching dictionary 744 from the store server 7 updates the matching dictionary 544 by overwriting the matching dictionary 544 with the received information of the matching dictionary 744. That is, the matching dictionary 544 is learned in the same way as the matching dictionary 744.

[0119] As described above, the POS terminal 1 of the embodiment is a POS terminal 1 that executes product registration processing of a product through operation by a customer himself, and is equipped with a communication unit 23 that can input first product identification information that identifies the product through code recognition based on a code symbol contained in an image of the product captured by the imaging device 3, and second product identification information that identifies the product through package recognition using general object recognition technology for the image, and a cancellation means 104 that allows a customer to cancel the product registration processing of a product related to the second product identification information inputted to the communication unit 23 after the product registration processing of the product, and prevents a customer from canceling the product registration processing of a product related to the first product identification information inputted to the communication unit 23 after the product registration processing of the product.

[0120] In such an embodiment, the POS terminal 1 allows a customer to cancel the product registration process for a product related to the second product identification information (i.e., a product whose package has been recognized) after the product registration process has been completed, thereby reducing the burden on store clerks and others involved in the cancellation process.

[0121] Although the embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

[0122] For example, in the embodiment, the edge device 5 has been described as a separate device. However, the present invention is not limited to this, and the edge device 5 may be incorporated in, for example, the imaging device 3 or the POS terminal 1. In other words, the imaging device 3 or the POS terminal 1 may have the functions of the edge device 5.

[0123] In the embodiment, two imaging units, the first imaging unit 37 and the second imaging unit 38, are provided. However, the present invention is not limited to this, and either the first imaging unit 37 or the second imaging unit 38 may be provided as an imaging unit. The imaging unit may have one or more individual imaging units. The more imaging units are arranged in different positions, the more multi-angled images of the product can be captured.

[0124] In the embodiment, for a product related to the first product identification information, only the product information related to the product identification information is displayed on the display 18. However, without being limited thereto, for a product related to the first product identification information, the product information related to the product identification information may be displayed on the display 18 in association with an indication indicating that the information is information recognized by code recognition.

[0125] In the embodiment, the POS terminal 1 inputs information obtained by adding information indicating that the code has been recognized to the first product identification information that has been code-recognized, and inputs information obtained by adding information indicating that the package has been recognized to the second product identification information that has been package-recognized, from the imaging device 3. However, the present invention is not limited to this. For example, the POS terminal 1 may connect the imaging device 3 and the edge device 5 with different interfaces, the imaging device 3 may not add information indicating that the code has been recognized to the first product identification information, and the edge device 5 may not add information indicating that the package has been recognized to the second product identification information. The POS terminal 1 may recognize the first product identification information as code-recognized information when input from the interface connected to the imaging device 3, and recognize the second product identification information as package-recognized information when input from the interface connected to the edge device 5. In this case, the second product identification information is directly input from the edge device 5 to the POS terminal 1.

[0126] In the embodiment, the additional information indicating that the product is a package-recognized product (the information of "P" in the embodiment) is added to the second product identification information by the edge device 5. However, the present invention is not limited to this, and for example, the additional information indicating that the product is a package-recognized product may be added to the second product identification information received from the edge device 5 by the imaging device 3.

[0127] The program executed by the POS terminal 1 of the embodiment is provided in the form of an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD).

[0128] The program executed by the POS terminal 1 of the embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program executed by the POS terminal 1 of the embodiment may be provided or distributed via a network such as the Internet.

[0129] Furthermore, the programs executed by the POS terminal 1 of the embodiment may be provided by being pre-installed in a ROM or the like. [Explanation of symbols]

[0130] 1 POS terminal 3. Imaging device 5. Edge Devices 7 Store Server 9 Checkout Device 10. System 11 CPU 17 Control section 18 Display 23 Communications Department 31 CPU 51 CPU 71 CPU 100 Control section 101 Judgment means 102 Display means 103 Cancellation operation determination means 104 Means of cancellation 105 Cancellation display means 131 Product Information Department 143 Log Information Section 171 Start key 172 Closing key 173 Cancel button 181 List 300 Control section 301 First generation means 302 First output means 331 Image storage unit 332 Product Image Section 333 Decode Information Section 334 Product Specific Information Department 500 Control section 501 Second generation means 502 second output means 531 Object Image Storage Unit 532 Feature Quantity Section 533 Product Specific Information Department 534 Related Information Section 542 Object Detection Model 543 Feature Extraction Model 544 Matching Dictionary 700 Control section 731 Sales Information Department 732 Related Information Section 1813 Cancel button 1817 Strikethrough [Prior art documents] [Patent documents]

[0131] [Patent Document 1] JP 2020-42461 A

Claims

1. A product sales data processing device that executes product registration processing of products through the operation of a customer himself / herself, an interface into which first product identification information for identifying a product that has been code-recognized based on a code symbol included in an image of a product captured by an imaging unit and second product identification information for identifying a product that has been package-recognized using a generic object recognition technology for the image can be input; a cancellation means for allowing the customer to cancel a product registration process for a product related to the second product identification information input to the interface after the product registration process for the product has been completed, and for disallowing the customer to cancel a product registration process for a product related to the first product identification information input to the interface after the product registration process for the product; A product sales data processing device comprising:

2. a display means for displaying on a display device the information on the product related to the first product identification information and the information on the product related to the second product identification information, which have been subjected to the product registration process, in such a manner as to distinguish between the information being the code recognition information and the information being the package recognition information; Further equipped with the cancellation means enables the customer to cancel the purchase of the product related to the second product identification information displayed by the display means. The product sales data processing device according to claim 1 .

3. the first product identification information is information to which information indicating that the code has been recognized has been added, and the second product identification information is information to which information indicating that the package has been recognized has been added, and the display means displays the code-recognized information and the package-recognized information in a distinguished manner based on the first product identification information to which information indicating that the code has been recognized has been added and the second product identification information to which information indicating that the package has been recognized has been added. The product sales data processing device according to claim 2 .

4. the display means displays a cancel button for canceling the product related to the displayed second product identification information, The cancellation means enables the cancellation of the product related to the designated second product identification information. The product sales data processing device according to claim 3.

5. The display means displays a display indicating that the cancellation has been processed for the product related to the canceled second product identification information. The product sales data processing device according to claim 4.

6. The cancellation means executes a call process for a store clerk or the like when a call operation related to the first product identification information is performed by the customer.

6. A product sales data processing device according to claim 2.

7. a log storage unit configured to store log information relating to the first product identification information to which information indicating that the code has been recognized has been added and the second product identification information to which information indicating that the package has been recognized has been added, after the product registration process has been performed; The product sales data processing device according to claim 3.

8. A checkout device including an imaging unit, an edge device, and a product sales data processing device that executes product registration processing of products through a customer's own operation, The imaging unit includes: A camera for capturing an image of a product; a first recognition means for decoding a code symbol included in a captured image to generate first product identification information relating to a specific product; a first output means for outputting the recognized first commodity identification information to the commodity sales data processing device; Equipped with The edge device includes: A second generating means for generating second product identification information relating to a specific product from the captured image by using a generic object recognition technology; A second output means for outputting the generated second product identification information to the imaging unit; Equipped with The first output means outputs the second product specification information output by the second output means to the product sales data processing device, The product sales data processing device includes: an interface into which the first product identification information and the second product identification information can be input; a cancellation means for allowing the customer to cancel a product registration process for a product related to the second product identification information input to the interface after the product registration process for the product has been completed, and for disallowing the customer to cancel a product registration process for a product related to the first product identification information input to the interface after the product registration process for the product; A check-out device comprising:

9. a computer as a product sales data processing device, comprising an interface through which first product identification information for identifying a product that is code-recognized based on a code symbol included in an image of a product captured by an imaging unit and second product identification information for identifying a product that is package-recognized using a generic object recognition technique for the image, the computer being operable to execute product registration processing of a product by a customer's own operation; a cancellation means for enabling the customer to cancel a product registration process for a product related to the second product identification information inputted to the interface after the product registration process for the product has been completed, and for disallowing the customer to cancel a product registration process for a product related to the first product identification information inputted to the interface after the product registration process for the product; A program to function as a

Images