Product processing system, program

Abstract

To reduce fraud in devices operated by customers. [Solution] The product processing system comprises: an acquisition unit that acquires the product code of the product to be registered; a recognition unit that recognizes the appearance of the product from an image captured in the area where the product is to be registered; a determination unit that determines whether predetermined conditions related to the product information of the product identified by the product code acquired by the acquisition unit and the appearance of the product recognized by the recognition unit are met; and a notification unit that notifies the operator based on the determination result of the determination unit.

Description

Technical Field

[0001] The present invention relates to a merchandise processing system and a program.

Background Art

[0002] Self-checkout in which customers register merchandise is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, fraud is likely to occur in the device operated by the self-checkout customer, which is a problem of the device operated by the customer.

[0005] The present invention has been made in view of such circumstances, and an object thereof is to provide a technique for reducing fraud in a device operated by a customer.

Means for Solving the Problems

[0006] One aspect for solving the above-described problems is a merchandise processing system including: an acquisition unit that acquires a merchandise code of a merchandise to be registered; a recognition unit that recognizes the appearance of a merchandise from a captured image of an area where the merchandise is registered; a determination unit that determines whether a predetermined condition related to merchandise information of the merchandise specified by the merchandise code acquired by the acquisition unit and the appearance of the merchandise recognized by the recognition unit is satisfied; and a notification unit that notifies an operator based on a determination result of the determination unit.

Brief Description of the Drawings

[0007] [Figure 1] This is an example of the appearance of a registration settlement device. [Figure 2] This is an example of the configuration of a registration and settlement device. [Figure 3] This is an example of the structure of a product master, etc. [Figure 4] This is a flowchart illustrating an example of the operation of the registration and settlement device. [Figure 5] This is an example of a display on a registration and payment device. [Figure 6] This flowchart shows another example of the operation of the registration and settlement device. [Figure 7] This flowchart shows another example of the operation of the registration and settlement device. [Figure 8] This flowchart shows another example of the operation of the registration and settlement device. [Modes for carrying out the invention]

[0008] (Sales system) The registration and settlement device 10 in this embodiment is one of the devices that constitute the sales system 1 (not shown). The sales system 1 can be introduced into various types of businesses (supermarkets, convenience stores, drugstores, home centers, consumer electronics stores, etc.). In addition to the registration and settlement device 10, the sales system 1 includes a transaction status management device 50 (not shown), a store controller 60 (not shown), and a headquarters server 70 (not shown). The registration and settlement device 10, the transaction status management device 50, and the store controller 60 are installed inside the store. These are connected so that they can communicate via LAN (which may be wired or wireless). The headquarters server 70 is installed outside the store (for example, at the head office or a data center). The devices installed inside the store (registration and settlement device 10, transaction status management device 50, store controller 60) and the devices installed outside the store (headquarters server 70) can communicate (either directly or via the store controller 60). There is no particular limit to the number of each device that can be installed.

[0009] (Registration and settlement device 10) The registration and payment device 10 is a so-called fully self-service register (fully self-service device) that registers products based on customer operations and settles payments based on customer operations. In other words, the registration and payment device 10 performs a registration process to register products and a settlement process to settle payments for the registered products, but both processes are basically performed based on the customer's own operations.

[0010] (Transaction status management device 50) The transaction status management device 50 is a device that manages (monitors and controls, etc., processing status, operating status, etc.) devices installed in the store. The transaction status management device 50 may also manage the operators of the devices installed in the store (monitors the operations of customers using the devices, etc., and confirms the operations of customers after use, etc.). The transaction status management device 50 comprises at least a CPU, a memory unit, a display unit (which may be configured as a touch panel), an operation unit (keyboard, mouse, etc.), and a communication unit. The transaction status management device 50 is used by store employees.

[0011] (Store Controller 60) The store controller 60 is a server primarily responsible for communicating information between devices installed inside the store and devices installed outside the store. The store controller 60 comprises at least a CPU, a memory unit, and a communication unit.

[0012] (Headquarters Server 70) The headquarters server 70 is installed outside the store (for example, at the head office or a data center). The headquarters server 70 includes at least a CPU, a memory unit, and a communication unit. The headquarters server 70 may also be a cloud server. The headquarters server 70 may be composed of multiple servers.

[0013] In this example, the registration and settlement device 10 (and the transaction status management device 50) communicates with the headquarters server 70 via the store controller 60, but it may also communicate with the headquarters server 70 without going through the store controller 60.

[0014] Also, in this example, a cloud server may be installed outside the store separately from the head office server 70. Hereinafter, for the sake of distinction, the cloud server realized as the head office server 70 is referred to as the cloud server 70, and the cloud server separate from the head office server 70 may be referred to as the cloud server 71.

[0015] Figure 1 is an external view example of the registration and settlement device 10. The registration and settlement device 10 includes a main body part 11, an upstream counter 12, and a downstream counter 13. The upstream counter 12 is a counter provided on the upstream side of the customer flow line relative to the main body part 11. On the upstream counter 12, for example, unregistered products, shopping baskets containing unregistered products, etc. are placed. The downstream counter 13 is a counter provided on the downstream side of the customer flow line relative to the main body part 11. On the downstream counter 13, for example, products after registration, shopping bags (personal bags, paid cash register bags, etc.) containing products after registration, etc. are placed. Note that the upstream counter 12 (similarly for the downstream counter 13) may be a separate body attached to the registration and settlement device 10 (main body part 11).

[0016] A cash settlement part (change machine) 106 is provided in the lower part of the main body part 11. In the upper part of the main body part 11, a printing part 108, a scanner part 104, a display part 103, a non-cash settlement part (settlement terminal) 105, an imaging part (camera) 110, and a support pole part 111 are provided. On the support pole part 111, a light emitting part 112, a monitoring display part 113, and a detection part (sensor part) 120 are provided in order from the top.

[0017] Figure 2 shows a configuration example of the registration settlement device 10. Although some parts have already been mentioned in FIG. 1, the registration settlement device 10 includes a CPU 101, a storage unit 102, a display unit 103, a scanner unit 104, a non-cash settlement unit 105, a cash settlement unit 106, a communication unit 107, a printing unit 108, an audio output unit 109, an imaging unit 110, a light emitting unit 112, a monitoring display unit 113, and a detection unit 120. These can communicate with each other via a bus. Although not shown in the figure, the registration settlement device 10 includes a timing unit (timing function). For example, the registration settlement device 10 can manage the current time, the elapsed time from a certain point, and the remaining time until a certain time by the timing unit. The registration settlement device 10 may include an audio input unit (microphone).

[0018] The CPU 101 is a central processing unit, and controls the operation of the registration settlement device 10 by reading and executing the programs stored in the storage unit 102. For example, the CPU 101 controls the entire transaction from the registration of goods to the settlement. That is, the CPU 101 executes (advances) the registration process and the settlement process.

[0019] The storage unit 102 is a general term for one or more various storage members (for example, those serving as memory or storage. For example, RAM, ROM, hard disk, SSD (Solid State Drive), etc.). The storage unit 102 stores various information.

[0020] The display unit 103 is a touch display, which displays various information (for example, screens in the registration process, screens in the settlement process) to the operator (customer), and also accepts the operations of the operator.

[0021] The scanner unit 104 optically reads various information. For example, the scanner unit 104 scans the code attached to the goods and reads the product code. The scanner unit 104 may image the code (area including the code) attached to the goods by an internal camera (scanner camera) and read the code.

[0022] The non-cash payment unit 105 is a payment mechanism that uses non-cash payment methods (e.g., credit card payment, electronic money payment, etc.). The non-cash payment unit 105 may be a separate payment terminal from the registration and settlement device 10. The payment terminal 105 may include, for example, a reading unit, a communication unit (a communication unit that communicates with the registration and settlement device 10, a communication unit that communicates with an external server such as a credit server that performs credit card payments), an operation unit, and a display unit. The cash payment unit 106 is a payment mechanism that uses cash. The cash payment unit 106 has a slot for inserting banknotes and coins, and a slot for dispensing banknotes and coins. The cash payment unit 106 detects the currency inserted into the slot, identifies the number of bills of each denomination inserted, calculates the amount inserted, calculates the change amount which is the difference between the amount inserted and the purchase amount, and dispenses the change from the dispensing slot. The insertion of banknotes and coins and the removal of change may be detected by sensors.

[0023] The communication unit 107 transmits and receives information with other devices (for example, the transaction status management device 50, the store controller 60, etc.).

[0024] The printing unit 108 prints and issues various media (receipts, coupons). The removal of media may be detected by a sensor. The audio output unit 109 outputs audio (e.g., confirmation sound, warning sound, voice guidance, etc.).

[0025] The imaging unit 110 captures images of the area below (from the upstream counter 12 to the front of the main unit 11 to the downstream counter 13). In other words, the imaging unit 110 can capture actions such as a customer taking items out of a shopping basket, a customer scanning items, and inserting cash into the cash settlement unit 106 (or withdrawing cash from the cash settlement unit 106). The images captured by the imaging unit 110 may be still images or moving images. The images captured by the imaging unit 110 (still images, moving images) may be stored in the storage unit 102.

[0026] The light-emitting unit 112 notifies various information by emitting light. For example, the light-emitting unit 112 may notify that a store employee is being called. For example, the light-emitting unit 112 may also notify that the registration is fraudulent (described later).

[0027] The monitoring display unit 113 is, for example, a liquid crystal display that displays various information to the surroundings (the operator, other customers, and store staff). For example, the monitoring display unit 113 may display the device number when in standby mode (waiting for customers) and display the image captured by the imaging unit 110 (moving or still image) during processing. The device number is indicated on both sides and the back of the monitoring display unit 113.

[0028] The detection unit 120 detects the presence of a person using a human presence sensor (for example, a sensor using infrared light). The detection unit 120 may be equipped with multiple human presence sensors with different detection positions. For example, the detection unit 120 may consist of a first sensor that detects a person in a direction looking down from the front of the upstream counter 12 (i.e., detecting the presence of a person in the front of the upstream counter 12), a second sensor that detects a person in a direction looking down from the front of the main unit 11 (i.e., detecting the presence of a person in the front of the main unit 11), and a third sensor that detects a person in a direction looking down from the front of the downstream counter 13 (i.e., detecting the presence of a person in the front of the downstream counter 13).

[0029] (Image processing) The registration and settlement device 10 (CPU 101) is equipped with an image processing function (image recognition function). For example, the registration and settlement device 10 is equipped with an object recognition function (object determination function) that recognizes objects in the image captured by the imaging unit 110. In addition, the registration and settlement device 10 is equipped with the following functions in relation to the object recognition function.

[0030] The registration and payment device 10 includes a product retrieval operation recognition function that recognizes, based on the captured image, the product retrieval operation performed by the customer when they take a product from a shopping basket placed on the upstream counter 12. The product retrieval operation recognition function may also recognize a new object as a product (moving, expanding object) taken out from the shopping basket placed on the upstream counter 12.

[0031] The registration and payment device 10 includes an item movement recognition function that recognizes, based on the captured image, an item movement action performed by a customer to move an item taken out of the shopping basket to the front of the main unit 11. The item movement recognition function may, for example, recognize whether a new object has appeared on the front (center) of the main unit 11.

[0032] The registration and payment device 10 is equipped with a product appearance recognition function that recognizes the appearance (shape) of the product whose barcode the customer scans (the product from which product identification information is acquired) or the product whose barcode has been scanned (the product from which product identification information has been acquired) based on the captured image. The product appearance recognition function may, for example, recognize the appearance of an object that has newly appeared on the front of the main unit 11.

[0033] Typically, a customer removes an item from their shopping basket, places it in front of the main unit 11, and scans the barcode of the item they have moved. In other words, in this sequence of events, the item removed from the shopping basket (the item newly appearing in front of the main unit 11) and the item whose barcode is scanned are usually the same item.

[0034] (Monitoring for fraudulent registrations) The registration and settlement device 10 monitors for fraudulent registrations. Fraudulent registration is the act of scanning and registering a barcode of a product other than the original product (the product taken out of the shopping basket). For example, when the product taken out of the shopping basket is a product that packages multiple products into one (hereinafter referred to as a "multi-pack product"), scanning the barcodes of the individual products that make up the multi-pack product (hereinafter referred to as "single products") instead of scanning the barcode of the multi-pack product constitutes a fraudulent registration.

[0035] The registration and payment device 10 monitors for fraudulent registration when, in the case of a multi-pack product taken out of a shopping basket, the barcode of the individual beer items that make up the multi-pack product is scanned and registered instead of scanning the barcode of the multi-pack product itself. Specifically, when, in the case of a six-pack of beer taken out of a shopping basket, the registration and payment device 10 monitors for fraudulent registration when the barcode of the six-pack of beer is scanned and registered instead of scanning the barcode of the six-pack of beer itself.

[0036] Monitoring for fraudulent registrations includes determining whether a registration is fraudulent and issuing notifications (anomaly or fraud notification) if a registration is deemed fraudulent.

[0037] The registration and settlement device 10 monitors for fraudulent registration of a 6-pack of beer (the act of scanning the barcodes of individual beer products that make up the 6-pack of beer, rather than scanning the barcode of the 6-pack of beer itself) by, for example, the following procedure (steps 1 to 6).

[0038] (Procedure 1) The registration and settlement device 10 acquires an image (an image of the product taken out of the shopping basket) using the imaging unit 110. (Step 2) The registration and settlement device 10 recognizes the appearance of the product from the image acquired in Step 1. (Step 3) The registration and payment device 10 acquires product identification information (product identification information obtained from the barcode of the product taken out of the shopping basket) using the scanner unit 104. (Step 4) The registration and settlement device 10 determines whether the appearance recognized in Step 2 corresponds to the appearance of a 6-pack of beer. For example, the registration and settlement device 10 determines whether the appearance recognized in Step 2 is a rectangular parallelepiped of a predetermined size that corresponds to a 6-pack of beer.

[0039] The registration and payment device 10 may determine whether the appearance recognized in step 2 corresponds to a predetermined size equivalent to a 6-pack of beer based on the size of the image area recognized from the captured image (the image area of ​​the product removed from the shopping basket). For example, the size of the image area (information indicating the size) when a 6-pack of beer is captured may be stored as a size for comparison (information indicating the size) in a location accessible to the registration and payment device 10 (for example, the storage unit 102 of the device itself, the storage unit of the store controller 60, the storage unit of the headquarters server 70, or the storage unit of the cloud server 71), and if the difference between the size of the image area recognized from the captured image and the size for comparison is within a predetermined range, the registration and payment device 10 may determine that the appearance recognized in step 2 corresponds to a predetermined size equivalent to a 6-pack of beer. The size of the image area (information indicating the size) may also be the volume. The volume may be determined by recognizing the faces and edges of the product from the captured image. The registration and payment device 10 may determine whether the appearance recognized in step 2 is a rectangular parallelepiped by recognizing the faces and edges from the image area of ​​the product recognized from the captured image.

[0040] The method by which the registration and payment device 10 determines whether an item has the appearance of a 6-pack of beer is not limited to determining whether it is a rectangular prism of a predetermined size corresponding to a 6-pack of beer. For example, if it recognizes that individual beer products are lined up (if multiple sides (curved surfaces) or tops (curved surfaces) of individual beer cans are recognized through the gaps), it may determine that the item has the appearance of a 6-pack of beer. In addition to the above, the registration and payment device 10 may also recognize the customer's posture and how they are holding the product (for example, whether they are holding it with both hands) to determine whether the item has the appearance of a 6-pack of beer. If the registration and payment device 10 recognizes that the item is being held with both hands (if it recognizes the left and right hands as objects overlapping the product), it may determine that the item is not an individual beer product.

[0041] (Procedure 5) If the registration and settlement device 10 determines in Procedure 4 that the product has the appearance of a 6-pack of beer, it determines whether the product for which product identification information was obtained in Procedure 3 is an alcoholic beverage priced below a predetermined amount. The predetermined amount is the selling price of the cheapest 6-pack of beer among the various 6-packs of beer sold at the store. An alcoholic beverage priced below the predetermined amount is, for example, a single item of beer (assuming a single item of beer that makes up a 6-pack of beer). The selling price of the cheapest 6-pack of beer is higher than the selling price of the most expensive single item of beer.

[0042] Alternatively, the system may store a predetermined amount (the lowest selling price of a 6-pack of beer) in advance (stored in a location accessible to the registration and settlement device 10), and when the registration and settlement device 10 determines whether the product for which product identification information was obtained in step 3 is an alcoholic beverage priced below the predetermined amount, it may refer to the predetermined amount (the lowest selling price of a 6-pack of beer) that has been stored in advance to determine whether the product for which product identification information was obtained in step 3 is an alcoholic beverage priced below the predetermined amount. Alternatively, when the registration and settlement device 10 determines whether the product for which product identification information was obtained in step 3 is an alcoholic beverage priced below the predetermined amount, it may refer to the product master to determine whether the product for which product identification information was obtained in step 3 is an alcoholic beverage priced below the predetermined amount. The product master may indicate a threshold line based on the predetermined amount, and the predetermined amount may be automatically set from the product master. If a user is setting the threshold, to make it easier to set, the selling price of the highest-priced single product, such as beer, may be automatically extracted from the product master, or the product master may be sorted by price.

[0043] Alternatively, the registration and settlement device 10 may pre-store product identification information for alcoholic beverages as predetermined product identification information (stored in a location accessible to the registration and settlement device 10), and when the registration and settlement device 10 determines whether the product for which product identification information was obtained in step 3 is an alcoholic beverage priced below a predetermined amount, it may refer to the pre-stored predetermined product identification information (product identification information for alcoholic beverages) to determine whether the product for which product identification information was obtained in step 3 is an alcoholic beverage. Or, when the registration and settlement device 10 determines whether the product for which product identification information was obtained in step 3 is an alcoholic beverage priced below a predetermined amount, it may refer to the product master to determine whether the product for which product identification information was obtained in step 3 is an alcoholic beverage.

[0044] (Step 6) If the registration and settlement device 10 determines in Step 5 that the alcoholic beverages are below the predetermined price, it will report an error (abnormality, fraud).

[0045] In other words, the registration and payment device 10 determines whether a registration is fraudulent by looking at the captured image to see if the item removed from the shopping basket is a 6-pack of beer, or if the scanned item is an alcoholic beverage (a single item of beer) that is priced lower than the cheapest 6-pack of beer. If it determines that a registration is fraudulent (if it determines that the item removed from the shopping basket is a 6-pack of beer and the scanned item is an alcoholic beverage that is priced lower than the cheapest 6-pack of beer), it notifies the user of an error.

[0046] Note that the order of steps 1 to 3 can be changed. In other words, instead of acquiring the captured image → recognizing the appearance of the product in the captured image → acquiring the product identification information, the order could be, for example, acquiring the captured image → acquiring the product identification information → recognizing the appearance of the product in the captured image, or acquiring the product identification information → acquiring the captured image → recognizing the appearance of the product in the captured image.

[0047] Figure 3 shows an example of the configuration of the product master, etc. Figure 3(A) shows an example of the configuration of the product master (product file) referenced by the registration and settlement device 10. Figures 3(B) and 3(C) will be described later.

[0048] The product master, as shown in Figure 3(A), is a file that stores product names and prices, associated with product identification information (for example, JAN codes or product codes pre-set for each product preset button). For example, in the product master shown in Figure 3(A), one record stores the product name "XX Beer" and price "280 (yen)" associated with product identification information "S0001" (symbol a). "XX Beer" is a single product. Also, in the product master shown in Figure 3(A), one record stores the product name "XX Beer 6-Can Pack" and price "1680 (yen)" associated with product identification information "S0201" (symbol b). "XX Beer 6-Can Pack" is a multi-pack product containing six cans of "XX Beer" packaged together.

[0049] The product master (for example, the product master in Figure 3(A); the same applies to the product master in Figure 3(B) described later) only needs to be stored in a location accessible to the registration and settlement device 10. The same applies to the classification master (for example, the classification master in Figure 3(C) described later).

[0050] Figure 4 is a flowchart illustrating an example of the operation of the registration and payment device 10. Specifically, the flowchart in Figure 4 shows the operation until each product is registered (until each individual product removed from the shopping basket is imaged and registered).

[0051] It was explained that the order of steps 1 to 3 in monitoring fraudulent registrations (steps 1 to 6) could be: acquiring captured images → recognizing the appearance of the product in the captured images → acquiring product identification information; acquiring captured images → acquiring product identification information → recognizing the appearance of the product in the captured images; or acquiring product identification information → acquiring captured images → recognizing the appearance of the product in the captured images. However, the flowchart in Figure 4 acquires product identification information after acquiring the captured images, so it corresponds to the order of acquiring captured images → recognizing the appearance of the product in the captured images and the order of acquiring captured images → acquiring product identification information → recognizing the appearance of the product in the captured images. Note that if the order of steps S1 and S2 in the flowchart in Figure 4 is swapped, it corresponds to acquiring product identification information → acquiring captured images → recognizing the appearance of the product in the captured images.

[0052] Step S1: The registration and settlement device 10 acquires an image (an image of the product registered by the customer) using the imaging unit 110. The process then proceeds to step S2. Step S2: The registration and settlement device 10 acquires product identification information using the scanner unit 104. The process then proceeds to step S3.

[0053] Step S3: The registration and payment device 10 determines whether the appearance (appearance of the product) recognized from the captured image acquired in Step S1 is a predetermined appearance. Specifically, the registration and payment device 10 determines whether the appearance recognized from the captured image is an appearance equivalent to a 6-pack of beer (a rectangular parallelepiped of a predetermined size). If it is a predetermined appearance (Step S3: YES), proceed to Step S4. If it is not a predetermined appearance (Step S3: NO), proceed to Step S9.

[0054] Step S4: The registration and settlement device 10 determines whether the product identification information obtained in Step S2 is product identification information for alcoholic beverages. If it is product identification information for alcoholic beverages (Step S4: YES), proceed to Step S5. If it is not product identification information for alcoholic beverages (Step S4: NO), proceed to Step S9.

[0055] Step S5: The registration and settlement device 10 determines whether the product identified by the product identification information obtained in Step S2 is less than a predetermined amount (less than the price of the cheapest 6-pack of beer). If it is less than the predetermined amount (Step S5: YES), proceed to Step S6. If it is not less than the predetermined amount (Step S5: NO), proceed to Step S9.

[0056] Step S6: The registration and settlement device 10 does not register the product identified by the product identification information acquired in step S2 (interrupts product registration) and reports an error. For example, the registration and settlement device 10 displays a message on the display unit 103 asking the customer to confirm whether the product to be scanned (registered) is a 6-pack of beer. The process then proceeds to step S7.

[0057] Step S7: The registration and settlement device 10 executes the staff call process. As part of the staff call process, the registration and settlement device 10 may, for example, display a message on the display unit 103 indicating that a staff member is being called, illuminate the light-emitting unit 112 in a light emission pattern indicating that a staff member is being called, and send a staff call notification via the communication unit 107 to a device used by a staff member (for example, a transaction status management device 50, a mobile terminal used by a staff member (not shown)) (one or more of these actions are sufficient). The process then proceeds to step S8. The registration and settlement device 10 may also display the message indicating that a staff member is being called on the display unit 103 along with the error notification in step S6 (for example, a message asking the customer to confirm whether it is a 6-pack of beer).

[0058] Step S8: The registration and settlement device 10 acquires product identification information using the scanner unit 104 based on the operation of the store clerk (the clerk who responds to the call for a store clerk) (for example, by acquiring product identification information from the barcode attached to a 6-pack of beer), registers the product identified by the acquired product identification information (for example, a 6-pack of beer), and cancels the interruption (returns from the interruption). Then this flowchart ends (processing for the next product starts from step S1).

[0059] (Supplement to Step S8) When a store employee is called, the employee enters their employee code into the registration and settlement device 10, which is displaying an error (for example, by having the employee code on the employee's name tag or card read), and displays a screen that allows for rescanning. The rescanning screen has release buttons (release button, OK button) to undo the interruption of product registration (return from interruption of product registration). In other words, if an employee code is entered while an error is being reported, the registration and settlement device 10 stops (clears) the error report and displays the rescanning screen. The employee who has displayed the rescanning screen scans the barcode of the product (for example, a 6-pack of beer) (this is the second scan if the scan in step S2 is counted as the first, so it is a rescan), then operates the release button to register the product (for example, a 6-pack of beer) and undo the interruption of product registration (return from interruption of product registration). In other words, the registration and settlement device 10 registers a product if a barcode of a product is scanned while the rescannable screen is displayed, then clears the rescannable screen based on the operation of the release button, and resumes the interruption of product registration (recovers from the interruption of product registration). As mentioned above, even during the interruption of product registration (before the release button is operated), product registration (product registration by a store employee) is possible while the rescannable screen is displayed.

[0060] In other words, if it is determined that the registration is fraudulent (and the process does not proceed to step S9), the process proceeds as follows: Error notification (product registration interrupted) → Call for a store employee → Input of employee code → Display of a screen where rescanning is possible → Rescan by the store employee (product registration) → Operation of the release button by the store employee → Disappearance of the rescanning screen / Release of product registration interruption (resumption).

[0061] Step S9: The registration and settlement device 10 registers the product identified by the product identification information obtained in Step S2. Then this flowchart ends (processing for the next product starts from Step S1).

[0062] Figure 5 shows an example of the display of the registration and settlement device 10 (an example of the display on the display unit 103). Before a transaction begins (while waiting for a customer), the registration and settlement device 10 displays a transaction start screen (not shown). The transaction start screen is equipped with a registration start button to start a transaction (it is displayed). In addition, a staff call button is provided at the top of the transaction start screen (and other screens as well).

[0063] When the registration start button on the transaction start screen (not shown) is pressed, the registration and settlement device 10 displays the initial registration screen (the registration screen before registering one product item) as shown in Figure 5(A). The initial registration screen (and the registration screen) is provided with a total amount display area HR10 that displays total information (total number of items purchased, total amount) and a product information display area HR11 that displays information about the registered product. The initial registration screen (and the registration screen) is also provided with a settlement button BT12 that is operated when proceeding to settlement (i.e., when completing registration). Note that the settlement button BT12 is a button that can be operated after the registration of one product item, so on the initial registration screen, which is the registration screen before the registration of the first product item, the settlement button BT12 is displayed in a way that indicates it cannot be operated (grayed out; shown as a dashed line in the figure).

[0064] Furthermore, the initial registration screen (and the registration screen as well) is equipped with various buttons (vegetable button, fruit button, fresh fish button, meat button) for registering products that do not have barcodes. For example, when the vegetable button is pressed, the registration and settlement device 10 displays product preset buttons for registering individual vegetables (cucumber, carrot, etc.). The same applies to the fruit, fresh fish, and meat buttons.

[0065] When a customer takes a bottle of XX Juice from a shopping basket placed on the upstream counter 12 and scans the barcode attached to the XX Juice, the registration and settlement device 10 displays a registration screen as shown in Figure 5(B) from the initial registration screen shown in Figure 5(A). The product information display area HR11 of the registration screen in Figure 5(B) displays information about the XX Juice (product name, quantity, price). The total amount display area HR10 of the registration screen in Figure 5(B) displays the total amount for the XX Juice.

[0066] In other words, the registration and payment device 10 acquires an image of the XX Juice taken out of the shopping basket (step S1 in the flowchart of Figure 4), obtains product identification information "S0089" from the barcode attached to the XX Juice (step S2), determines that the appearance of the XX Juice is not the predetermined appearance (appearance equivalent to a 6-pack of beer) (step S3: NO), and registers the XX Juice based on the product identification information "S0089" (step S9).

[0067] Furthermore, the product information display area HR11 for the product (XX Juice) includes a plus button (labeled "+" in the diagram) to increase the purchase quantity of the product, a minus button (labeled "-" in the diagram) to decrease the purchase quantity of the product, and a cancel button (labeled "Cancel" in the diagram) to cancel the purchase of the product. Note that the minus button is only operable when the purchase quantity is 2 or more, so in the scene shown in Figure 5(B), the minus button is displayed in a way that indicates it is not operable.

[0068] When a customer takes a 6-pack of XX beer from the shopping basket placed on the upstream counter 12 and reads the barcode on the individual XX beer product through the gap in the 6-pack, the registration and payment device 10 will display a small screen SG1 as a pop-up in front of the registration screen in Figure 5(B), as shown in Figure 5(C). The small screen SG1 displays the message, "Isn't this a 6-pack and not an individual product? A store employee will come to you. Please wait a moment."

[0069] The message "Isn't this a 6-pack, not a single item?" is an example of an error notification (step S6) in the flowchart of Figure 4. Note that "Isn't this a 6-pack, not a single item?" also suggests the correct product, but it could also be a message that simply points out the mistake, such as "Have you registered a different product?" The message "A store employee will be with you. Please wait a moment." is an example of a store employee calling process (step S7) in the flowchart of Figure 4.

[0070] In other words, the registration and payment device 10 acquires an image of the 6-pack of XX beer taken out of the shopping basket (step S1 in the flowchart of Figure 4), acquires product identification information "S0001" from the barcode on the individual XX beer products that make up the 6-pack of XX beer, rather than the barcode on the 6-pack of XX beer itself (step S2), determines that the appearance of the 6-pack of XX beer is consistent with that of a 6-pack of beer (step S3: YES), and acquires product identification information "S000 The system determines that "1" is product identification information for alcoholic beverages (Step S4: YES), determines that the XX beer identified by product identification information "S0001" is below a predetermined price (Step S5: YES), notifies an error (displays the message "Is this a 6-pack and not a single item?") (Step S6), and executes the process of calling a store employee (displays the message "A store employee will be coming. Please wait a moment," and illuminates the light-emitting unit 112 in a manner that indicates that a store employee is being called) (Step S7).

[0071] When a store clerk reads the barcode on a 6-pack of XX beer from the registration screen shown in Figure 5(C) and the interruption is released, the registration device 10 displays the registration screen shown in Figure 5(D). The product information display area HR11 on the registration screen in Figure 5(D) displays information (product name, quantity, price) for the XX juice and the 6-pack of XX beer. In addition, the total amount display area HR10 on the registration screen in Figure 5(D) displays the total amount for the XX juice and the 6-pack of XX beer.

[0072] In other words, the registration and settlement device 10, at the request of the store clerk, obtains product identification information "S0201" from the barcode attached to the 6-pack of XX beer, registers the 6-pack of XX beer, and then cancels the interruption (step S8).

[0073] (Other actions taken if the registration is deemed fraudulent) The actions taken when an invalid registration is determined (actions taken when the system does not proceed to step S9) are explained as follows: error notification (interruption of product registration) → calling a store employee → input of employee code → display of a screen where rescanning is possible → rescanning by the store employee (product registration) → operation of the release button by the store employee → disappearance of the rescanning screen / release of the interruption of product registration (resumption). However, this is just one example, and any of the alternative actions A to D below may also be taken.

[0074] Alternatively, (Alternative action A) Error notification (interruption of product registration) → Call for staff → Input of staff code → Display of staff-only screen (with a rescan button) → Staff member operates the rescan button → Display of rescannable screen (with a back button (equivalent to a cancel button)) → Staff member rescans (product registration) → Staff member operates the back button → Screen of rescannable screen is cleared / Product registration interruption is canceled (return).

[0075] (Alternative action B) Error notification (interruption of product registration) → Call for staff → Input of staff code → Staff-only screen (with a release button) → Staff operates the release button → Staff (or customer) rescans (registers the product) → Staff-only screen is cleared / Product registration interruption is reversed (returned). Note that the part "Staff operates the release button → Staff (or customer) rescans (registers the product) → Staff-only screen is cleared / Product registration interruption is reversed (returned)" may be in the order of "Staff operates the release button → Staff-only screen is cleared / Product registration interruption is reversed (returned) → Staff (or customer) rescans (registers the product)".

[0076] (Alternative action C) Error notification (interrupt / rescan instruction) → Customer rescans → Successful registration of product (6-pack of beer) (If the 6-pack of beer is rescanned, step S5 (NO)) → Error notification is cleared → Release (return) is also possible.

[0077] (Alternative action D) Error notification (instruction to operate the interrupt / confirm button) → Customer operates the confirmation button → Error notification is cleared → Release (return). If the release is performed by alternative action D, the process returns to step S2 after the release.

[0078] Furthermore, including separate operations A through D, error notifications may be erased even without any further operation after a predetermined time has elapsed since the notification.

[0079] (Other methods for identifying fraudulent registrations (monitoring methods)) In the flowchart of Figure 4, if the appearance of the product is a predetermined appearance (Step S3 (YES)), the product identification information is for alcoholic beverages (Step S4 (YES)), and the price is less than a predetermined amount (Step S5 (YES)), it is determined to be an invalid registration and an error is reported (proceed to Step S6). In other words, the determination of whether it is an invalid registration (decision to proceed to Step S9 or Step S6) is explained as being whether the appearance of the product is a predetermined appearance (Step S3), whether the product identification information obtained by scanning is for alcoholic beverages (Step S4), and whether the product identified by the product identification information obtained by scanning is less than a predetermined amount (Step S5). However, this is just an example, and it may also be any of the alternative determinations A to C below.

[0080] Figures 6 to 8 are flowcharts illustrating other examples of operations of the registration and settlement device 10. The flowchart in Figure 6 shows alternative decision A. The flowchart in Figure 7 shows alternative decision B. The flowchart in Figure 8 shows alternative decision C. Note that Figures 6 to 8 each represent the dashed box P (steps S3, S4, and S5) of the flowchart in Figure 4 (showing the parts to be replaced).

[0081] (Alternative decision A / Flowchart in Figure 6) Step S3: The registration and settlement device 10 determines whether the appearance (appearance of the product) recognized from the image acquired in step S1 is a predetermined appearance, similar to step S3 in the flowchart of Figure 4. If it is a predetermined appearance (step S3: YES), proceed to step S4. If it is not a predetermined appearance (step S3: NO), proceed to step S9 in the flowchart of Figure 4.

[0082] Step S4: The registration and settlement device 10 determines whether the product identification information obtained in step S2 is product identification information for alcoholic beverages, similar to step S4 in the flowchart of Figure 4. If it is product identification information for alcoholic beverages (step S4: YES), proceed to step S15. If it is not product identification information for alcoholic beverages (step S4: NO), proceed to step S9 in the flowchart of Figure 4.

[0083] Step S15: The registration and settlement device 10 determines whether the product identification information obtained in Step S2 is product identification information for an individual product that makes up a pack (for example, a 6-pack of beer). If it is product identification information for an individual product that makes up a pack (Step S15: YES), proceed to Step S6 in the flowchart of Figure 4. If it is not product identification information for an individual product that makes up a pack (Step S15: NO), proceed to Step S9 in the flowchart of Figure 4.

[0084] (Supplement to Step S15) In sales system 1, the product master (product file) shown in Figure 3(B) may be managed (stored) instead of the product master (product file) shown in Figure 3(A). In other words, the product master shown in Figure 3(B) may be stored in a location accessible to the registration and settlement device 10 instead of the product master shown in Figure 3(A), and the registration and settlement device 10 may refer to the product master shown in Figure 3(B) instead of the product master shown in Figure 3(A). The product master shown in Figure 3(B) is a file that stores the major category, minor category, product name, and price, associated with product identification information.

[0085] The registration and settlement device 10 may refer to the product master shown in Figure 3(B) to make the determination in step S15 (determining whether the product identification information obtained in step S2 is the product identification information of the individual products that make up the pack). For example, if the product identification information obtained in step S2 is the product identification information ("S0001", "S0003", "S0089") of the individual products ("○○ Beer", "△△ Beer", "×× Juice") corresponding to the products in the subcategory "6-pack" (in the example in Figure 3(B), "○○ Beer 6-pack", "△△ Beer 6-pack", "×× Juice 6-pack"), the registration and settlement device 10 may determine that it is the product identification information of the individual products that make up the pack.

[0086] In sales system 1, in addition to the product master (product file) shown in Figure 3(A), a classification master (classification file) shown in Figure 3(C) may also be managed (stored). In other words, instead of the product master shown in Figure 3(B), the product master shown in Figure 3(A) and the classification master shown in Figure 3(C) may be managed. The registration and settlement device 10 may refer to the product master shown in Figure 3(A) and the classification master shown in Figure 3(C) instead of the product master shown in Figure 3(B) to make the decision in step S15. Note that since a file similar to the product master shown in Figure 3(B) can be generated from the product master shown in Figure 3(A) and the classification master shown in Figure 3(C) (combined), the registration and settlement device 10 can make the decision in step S15 in the same way as when referring to the product master shown in Figure 3(B).

[0087] Alternatively, the product identification information of individual products constituting the pack (for example, "S0001", "S0003", "S0089") may be stored in advance as predetermined product identification information (stored in a location accessible to the registration and settlement device 10). When the registration and settlement device 10 determines whether the product identification information obtained in step S2 is the product identification information of an individual product constituting the pack, it may refer to the predetermined product identification information (product identification information of individual products constituting the pack) that has been stored in advance to determine whether the product identification information obtained in step S2 is the product identification information of an individual product constituting the pack. As mentioned above, if the product identification information obtained in step S2 is "S0089", the product "XX Juice" with product identification information "S0089" is not an alcoholic beverage, so the process does not proceed to step S15 (step S4(NO)).

[0088] In other words, in the flowchart of Figure 6 (alternative judgment A), if the appearance of the product is the predetermined appearance (step S3 (YES)), and it is product identification information for alcoholic beverages (step S4 (YES)), and it is product identification information for individual products that make up the pack (step S15 (YES)), then it is determined that it is an invalid registration and an error is reported (proceed to step S6).

[0089] (Alternative decision B / Flowchart in Figure 7) Step S3: The registration and settlement device 10 determines whether the appearance (appearance of the product) recognized from the image acquired in step S1 is a predetermined appearance, similar to step S3 in the flowchart of Figure 4. If it is a predetermined appearance (step S3: YES), proceed to step S24. If it is not a predetermined appearance (step S3: NO), proceed to step S9 in the flowchart of Figure 4.

[0090] Step S24: The registration and settlement device 10 determines whether the product identification information obtained in step S2 is product identification information for a single alcoholic beverage product. If it is product identification information for a single alcoholic beverage product (Step S24: YES), proceed to step S6 in the flowchart of Figure 4. If it is not product identification information for a single alcoholic beverage product (Step S24: NO), proceed to step S9 in the flowchart of Figure 4.

[0091] (Supplement to Step S24) The registration and settlement device 10 may refer to the product master shown in Figure 3(B) to make the determination in step S24 (determining whether the product identification information obtained in step S2 is product identification information for a single alcoholic beverage product). For example, if the product identification information obtained in step S2 is product identification information ("S0001", "S0002", "S0003") for a product in the major category "Alcoholic Beverages" and the subcategory "Single Item" ("○○ Beer", "×× Beer", "△△ Beer"), the registration and settlement device 10 may determine that it is product identification information for a single alcoholic beverage product.

[0092] The registration and settlement device 10 may also make the decision in step S15 by referring to the product master shown in Figure 3(A) and the classification master shown in Figure 3(C), instead of the product master shown in Figure 3(B).

[0093] Alternatively, the registration and settlement device 10 may pre-store predetermined product identification information (for example, "S0001", "S0002", "S0003") as product identification information for individual alcoholic beverages (stored in a location accessible to the registration and settlement device 10), and when the registration and settlement device 10 determines whether the product identification information obtained in step S2 is product identification information for individual alcoholic beverages, it may refer to the pre-stored predetermined product identification information (product identification information for individual alcoholic beverages) to determine whether the product identification information obtained in step S2 is product identification information for individual alcoholic beverages.

[0094] In other words, in the flowchart of Figure 7 (alternative judgment B), if the appearance of the product is the predetermined appearance (step S3 (YES)) and the product identification information is for a single alcoholic beverage product (step S24 (YES)), it is determined that the registration is fraudulent and an error is reported (proceed to step S6).

[0095] (Alternative decision C / Flowchart in Figure 8) Step S3: The registration and settlement device 10 determines whether the appearance (appearance of the product) recognized from the image acquired in step S1 is a predetermined appearance, similar to step S3 in the flowchart of Figure 4. If it is a predetermined appearance (step S3: YES), proceed to step S15. If it is not a predetermined appearance (step S3: NO), proceed to step S9 in the flowchart of Figure 4.

[0096] Step S15: The registration and settlement device 10 determines whether the product identification information obtained in step S2 is product identification information for an individual product that makes up the pack, similar to step S15 in the flowchart of Figure 6. If it is product identification information for an individual product that makes up the pack (step S15: YES), the device proceeds to step S6 in the flowchart of Figure 4. If it is not product identification information for an individual product that makes up the pack (step S15: NO), the device proceeds to step S9 in the flowchart of Figure 4.

[0097] In other words, in the flowchart of Figure 8 (alternative judgment C), if the appearance of the product is the predetermined appearance (step S3 (YES)) and the product identification information is that of an individual product constituting the pack (step S15 (YES)), it is determined that the registration is invalid and an error is reported (proceed to step S6).

[0098] The flowcharts in Figures 4, 6, 7, and 8 have been explained above. The part of each flowchart used to determine fraudulent registration can be summarized as follows: In other words, the registration settlement device 10 determines that a registration is fraudulent if the following conditions A and B are met.

[0099] The registration and payment device 10 determines whether the conditions (condition A) based on the captured image of the product taken out of the shopping basket are met. Condition A is the determination in step S3 of the flowchart in Figures 4, 6, 7, and 8 (determination of whether the appearance of the product is a predetermined appearance (appearance of a 6-pack of beer)).

[0100] If the registration and settlement device 10 determines that condition A (condition based on the captured image) is met (in the case of step S3 (YES) in the flowcharts of Figures 4, 6, 7, and 8), it determines whether the condition (condition B) related to the scanned product is met. Condition B is determined by the decisions in steps S4 and S5 of the flowchart in Figure 4 (determining whether the scanned product is an alcoholic beverage below a predetermined price (for example, a single beer product less than a 6-pack)), the decisions in steps S4 and S15 of the flowchart in Figure 6 (determining whether the scanned product is an alcoholic beverage that makes up a pack (for example, a single beer product that makes up a 6-pack)), the decision in step S24 of the flowchart in Figure 7 (determining whether the scanned product is an alcoholic beverage product that makes up a single beer product) and the decision in step S15 of the flowchart in Figure 8 (determining whether the scanned product is a product that makes up a pack (for example, a single beer product that makes up a 6-pack)).

[0101] The registration and settlement device 10 determines that the registration is fraudulent if it determines that condition B (conditions relating to the product whose barcode was scanned) is met (in the case of step S4 (YES) → S5 (YES) in the flowchart of Figure 4, in the case of step S4 (YES) → S15 (YES) in the flowchart of Figure 6, in the case of step S24 (YES) in the flowchart of Figure 7, and in the case of step S15 (YES) in the flowchart of Figure 8).

[0102] (Notification that the appearance is as specified) If the registration and settlement device 10 determines that the appearance of the product is a predetermined appearance (the appearance of a 6-pack of beer), it may notify that the product has the predetermined appearance.

[0103] Regarding the order of steps 1 to 3 in monitoring fraudulent registration (steps 1 to 6), it was explained that the order could be either acquiring an image → acquiring product identification information → recognizing the appearance of the product in the image, or acquiring product identification information → acquiring an image → recognizing the appearance of the product in the image, rather than acquiring an image → recognizing the appearance of the product in the image → acquiring product identification information. However, if the order is acquiring an image → recognizing the appearance of the product in the image → acquiring product identification information (recognizing the appearance of the product before scanning the barcode), the registration and settlement device 10 may determine if the appearance of the product is a predetermined appearance, and if it determines that the appearance of the product is a predetermined appearance, it may notify that it is a predetermined appearance. For example, the registration and settlement device 10 may notify that it is a predetermined appearance before acquiring product identification information.

[0104] If the registration and payment device 10 determines that the appearance of the product is a predetermined appearance, it may, in lieu of or in addition to stating that the product has a predetermined appearance, notify that the 6-pack of beer needs to be registered, or that the single-item beer product should not be registered. This notification is intended to inform the customer that the product has a predetermined appearance, but the registration and payment device 10 may, in lieu of or in addition to informing the customer, also inform the store clerk that the product has a predetermined appearance (by illuminating the light-emitting unit 112, transmitting the information to a device used by the store clerk, etc.).

[0105] The timing for recognizing the appearance of the product can be when it is in the shopping basket (before taking it out), when it is taken out of the shopping basket, when it is being moved, or when it is scanned. Furthermore, if the order of steps 1 to 3 is arbitrary, it can also be done at a time after scanning.

[0106] (Other devices) Other devices besides those described above may be installed in the store. For example, a registration and settlement device (not shown) that performs registration and settlement processing based on the actions of a store employee may be installed. Alternatively, a registration and settlement device (face-to-face semi-self-service device; not shown) that performs registration processing based on the actions of a store employee and settlement processing based on the actions of a customer may be installed. Furthermore, as a semi-self-service system, a registration device 20 (not shown) that performs registration processing based on the actions of a store employee, and a settlement device (not shown) that acquires registration information from the registration device 20 and performs settlement processing based on the actions of a customer may be installed. Additionally, a registration and settlement device 30 (not shown) that can switch between an operating mode that performs registration processing based on the actions of a store employee and settlement processing based on the actions of a customer (face-to-face semi-self-service mode) and an operating mode that performs registration processing and settlement processing based on the actions of a customer, similar to the registration and settlement device 10 (full self-service mode), may be installed. Furthermore, a portable registration terminal 40 (a terminal that can be attached to a shopping cart, a terminal integrated with a shopping cart, etc., not shown) that performs registration processing based on customer operations at the product sales area may be operated, and a payment device (not shown) that acquires registration information from the portable registration terminal 40 and performs payment processing based on customer operations may be installed.

[0107] While the monitoring of fraudulent registrations in the registration and settlement device 10 has been described, other devices may similarly monitor for fraudulent registrations. For example, the full self-service registration and settlement device 30 and the portable registration terminal 40 may operate as shown in the flowchart in Figure 4 (or Figure 6, 7, or 8).

[0108] (Sales data processing system, product processing system) For example, a device capable of performing registration processing (e.g., registration and settlement device 10, registration device 20, registration and settlement device 30, portable registration terminal 40) processes information related to the sale of goods (such as generating product registration information), so a device capable of performing registration processing may be called a sales data processing device. Also, a device capable of performing registration processing processes products (processing information related to the sale of goods), so a device capable of performing registration processing may be called a product processing device.

[0109] Furthermore, as shown in Modification 10 described later, some of the functions of each device may be implemented in other devices different from the device described as the main device for that function. For example, some of the functions of the registration and settlement device 10 may be implemented in a device different from the registration and settlement device 10. In other words, the functions of the registration and settlement device 10 may be implemented by multiple devices. Accordingly, the sales data processing device may be referred to as a sales data processing system, and the product processing device may be referred to as a product processing system.

[0110] As described above, the embodiments can reduce fraud (fraudulent registration) at self-checkout counters. For example, the registration and payment device 10 monitors for fraudulent registrations, such as scanning the barcode of a single beer item instead of the barcode of a 6-pack of beer, that is, to monitor typical fraudulent activity. This can effectively reduce fraud at self-checkout counters.

[0111] Furthermore, since the registration and settlement device 10 simply recognizes the appearance of the product from the captured image, it can easily reduce fraud. For example, when identifying a product from a captured image (in a configuration where the product identified from the captured image is compared with the product identification information obtained from the barcode, or in a configuration where the product identified from the captured image is registered without obtaining product identification information from the barcode), maintenance of the product image data (registration, re-registration, etc.) is required when the product is changed or discontinued or the design is changed. However, when simply recognizing the appearance of the product from the captured image, such maintenance is basically unnecessary, and fraud can be easily reduced. In addition, while high-precision recognition is required when identifying a product from a captured image, the requirements are not so high when simply recognizing the appearance of the product from the captured image, so the design is relatively easy.

[0112] (Variations, etc.) Although embodiments have been described above, the configuration of the equipment, the data configuration, the processing flow, the display and output modes, etc., can be changed as appropriate, for example, as shown in the following Modifications 1 to 10. Furthermore, Modifications 1 to 10 can be combined as appropriate.

[0113] (Modification 1) In the above embodiment, an example was described in which the shape of the product is recognized as the appearance of the product. However, in addition to the shape of the product, the color (including patterns and designs) may also be recognized as the appearance of the product. In other words, in the above embodiment, when determining whether the appearance recognized from the captured image (the appearance of the product) corresponds to the appearance of a 6-pack of beer, it was explained that it is determined whether it is a rectangular prism of a predetermined size. However, it is also possible to determine whether it is a rectangular prism of a predetermined size and a predetermined color (the color of a 6-pack of beer).

[0114] The colors of various 6-pack beer containers are stored as predetermined colors (stored in a location accessible to the registration and settlement device 10), and the registration and settlement device 10 may refer to the predetermined colors (colors of various 6-pack beer containers) that have been stored in advance when determining whether a rectangular prism of a predetermined size is of a predetermined color.

[0115] (Modification 2) In the above embodiment, an example was described in which a message is displayed on the display unit 103 as notification of an error indicating that the registration is fraudulent (Figure 5(C)). However, notification of an error indicating that the registration is fraudulent is not limited to display on the display unit 103. For example, instead of or in addition to displaying on the display unit 103, one or more of the following may be performed: audio output of the error by the audio output unit 109, illumination of an error by the light-emitting unit 112 in a specific illumination pattern, or transmission of a fraudulent registration notification by the communication unit 107 to a device used by the store clerk (e.g., transaction status management device 50, mobile terminal used by the store clerk (not shown)).

[0116] (Modification 3) In the above embodiment, an example was described in which fraudulent activity occurs when a single beer product is registered without registering a 6-can beer pack. However, it is not limited to 6 cans. For example, fraudulent activity may also occur when a single beer product is registered without registering a 4-can beer pack. In other words, the number of single products that make up a multi-pack product does not have to be 6.

[0117] (Modification 4) In the above embodiment, an example was described of monitoring fraudulent activity in which a single beer product is registered without registering a 6-pack of beer. However, the target of fraudulent activity is not limited to beer. The target of fraudulent activity may be other alcoholic beverages (e.g., shochu cocktails, sake). For example, fraudulent activity in which a single shochu cocktail is registered without registering a 6-pack of shochu cocktails may be monitored. Note that the product does not have to be in a can.

[0118] The items monitored for fraud may also include beverages other than alcohol (e.g., juice, carbonated drinks). For example, fraud such as registering a single bottle of juice instead of a 6-pack of juice may be monitored. Furthermore, the product does not necessarily have to be in a can.

[0119] The items subject to monitoring for fraud are not limited to beverages. For example, it could be a four-pack of candy (hanging candy) where four individual bags of candy are connected together and sold. It could also be a pack product where several individual bags of candy are packaged together (or in a box) and sold. It could also be a pack product where several individual packs of sliced ​​ham are bundled together and sold. Furthermore, it could be a pack product where several packs of rice that are heated in a microwave are sold together.

[0120] The scope of fraud monitoring is not limited to food products. For example, it could include everyday necessities sold as individual items or in multi-packs (such as hand warmers and tissues).

[0121] The determination of whether the appearance is predetermined in step S3 of the flowchart in Figure 4 (and similarly in the flowcharts in Figures 6 to 8) can be made differently depending on the product being monitored for fraud. In other words, in this embodiment, since the product being monitored for fraud is canned beer, in step S3 it is determined whether the appearance recognized from the captured image corresponds to the appearance of a 6-pack of beer (the appearance of a rectangular parallelepiped of a predetermined size). However, if the product being monitored for fraud is a 4-pack of confectionery, for example, in step S3 it may be determined whether the appearance recognized from the captured image corresponds to the appearance of a 4-pack of confectionery (the appearance of a strip of a predetermined size).

[0122] Furthermore, the determination of whether the product identification information in step S4 of the flowchart in Figure 4 (and similarly in the flowchart in Figure 6) is for alcoholic beverages can be made differently depending on the product being monitored for fraud. In other words, in this embodiment, since the product being monitored for fraud is canned beer, in step S4 it is determined whether the product whose barcode has been scanned is an alcoholic beverage. However, if, for example, the product being monitored for fraud is a four-pack of confectionery, then in step S4 it may be determined whether the product whose barcode has been scanned is confectionery. The same applies to step S24 of the flowchart in Figure 7.

[0123] Furthermore, the determination in step S5 of the flowchart in Figure 4 regarding whether the price is below a predetermined amount can be varied depending on the product being monitored for fraud. In other words, in this embodiment, since the product being monitored for fraud is canned beer, step S5 determines whether the price is below the price of the cheapest 6-pack of beer. However, if the product being monitored for fraud is, for example, a 4-pack of confectionery, then step S5 may determine whether the price is below the price of the cheapest 4-pack of confectionery.

[0124] Furthermore, the determination in step S15 of the flowchart in Figure 6 (and similarly in the flowchart in Figure 8) as to whether the barcode is product identification information for an individual product constituting the pack can be made differently depending on the product being monitored for fraud. In other words, in this embodiment, since the product being monitored for fraud is canned beer, in step S15 it is determined whether the scanned barcode is product identification information for a can of beer, which is an individual product constituting a 6-pack of beer. However, if the product being monitored for fraud is a 4-pack of confectionery, for example, in step S15 it may be determined whether the scanned barcode is product identification information for a confectionery, which is an individual product constituting a 4-pack of confectionery.

[0125] This section provides a brief explanation (verification) of the operation in cases where the monitoring target for fraud involves something other than beer. (Case a) When scanning the barcode of a single juice item in a 6-can juice set, in the flowchart of Figure 4, "Alcoholic Beverages" in step S4 becomes "Beverages," and the predetermined price becomes the lowest price for the 6-can juice set. Therefore, the registration and settlement device 10 proceeds from step S3 (YES) to step S4 (YES) to step S5 (YES) and determines that the registration is fraudulent. Similarly, in the flowchart of Figure 6, the registration and settlement device 10 also determines that the registration is fraudulent. (Case b) If a single candy item (which is part of a package product) is placed on top of a product (for example, a set of 6 cans of beer) and the barcode of the candy is scanned, the registration and settlement device 10 will, for example, proceed from step S3 (YES) to step S35 (YES) in the flowchart of Figure 8 and determine that the registration is invalid.

[0126] (Modification 5) In the above embodiment, the registration and settlement device 10 determines whether condition A (condition based on the captured image) is met, and if it determines that condition A is met, it determines whether condition B (condition related to the product whose barcode was scanned) is met, and if it determines that condition B is met, it determines that the registration is fraudulent. However, the order of condition A and condition B may be reversed.

[0127] For example, in the flowchart of Figure 4, the registration and settlement device 10 determines whether the appearance recognized from the captured image is a predetermined appearance (step S3), and if it determines that the appearance recognized from the captured image is a predetermined appearance, it determines whether the product whose barcode was scanned is an alcoholic beverage (step S4), and if it determines that it is an alcoholic beverage, it determines whether the price of the product whose barcode was scanned is less than a predetermined amount (step S5), and if it determines that the price is less than a predetermined amount, it determines that it is a fraudulent registration. Alternatively, the registration and settlement device 10 may determine whether the product whose barcode was scanned is an alcoholic beverage, and if it determines that it is an alcoholic beverage, it determines whether the price of the product whose barcode was scanned is less than a predetermined amount, and if it determines that the price is less than a predetermined amount, it determines whether the appearance recognized from the captured image is a predetermined appearance, and if it determines that the appearance recognized from the captured image is a predetermined appearance, it determines that it is a fraudulent registration.

[0128] In the flowchart of Figure 6, the registration and settlement device 10 determines whether the appearance recognized from the captured image is a predetermined appearance (step S3), and if it determines that the appearance recognized from the captured image is a predetermined appearance, it determines whether the product whose barcode was scanned is an alcoholic beverage (step S4), and if it determines that it is an alcoholic beverage, it determines whether the product whose barcode was scanned is a single item that makes up a pack (step S15), and if it determines that it is a single item that makes up a pack, it determines that the registration is fraudulent. Alternatively, the registration and settlement device 10 may determine whether the product whose barcode was scanned is an alcoholic beverage, and if it determines that it is an alcoholic beverage, it determines whether the product whose barcode was scanned is a single item that makes up a pack, and if it determines that it is a single item that makes up a pack, it determines whether the appearance recognized from the captured image is a predetermined appearance, and if it determines that the appearance recognized from the captured image is a predetermined appearance, it determines that the registration is fraudulent.

[0129] In the flowchart of Figure 7, the registration and settlement device 10 determines whether the appearance recognized from the captured image is a predetermined appearance (step S3), and if it determines that the appearance recognized from the captured image is a predetermined appearance, it determines whether the product whose barcode was scanned is a single-item alcoholic beverage (step S24), and if it determines that it is a single-item alcoholic beverage, it determines that the registration is fraudulent. Alternatively, the registration and settlement device 10 may determine whether the product whose barcode was scanned is a single-item alcoholic beverage, and if it determines that it is a single-item alcoholic beverage, it may determine whether the appearance recognized from the captured image is a predetermined appearance, and if it determines that the appearance recognized from the captured image is a predetermined appearance, it may determine that the registration is fraudulent.

[0130] In the flowchart of Figure 8, the registration and settlement device 10 determines whether the appearance recognized from the captured image is a predetermined appearance (step S3), and if it determines that the appearance recognized from the captured image is a predetermined appearance, it determines whether the product whose barcode was scanned is a single product that makes up a pack (step S15), and if it determines that it is a single product that makes up a pack, it determines that the registration is fraudulent. Alternatively, the registration and settlement device 10 may determine whether the product whose barcode was scanned is a single product that makes up a pack, and if it determines that it is a single product that makes up a pack, it may determine whether the appearance recognized from the captured image is a predetermined appearance, and if it determines that the appearance recognized from the captured image is a predetermined appearance, it may determine that the registration is fraudulent.

[0131] Furthermore, in the order of Condition A (condition based on the captured image) → Condition B (condition related to the product whose barcode was scanned), the appearance of the product is recognized from the captured image for all products. However, in the order of Condition B → Condition A, the appearance of the product may not be recognized from the captured image if it is determined that Condition B is not met, and may be recognized from the captured image if it is determined that Condition B is met. In other words, by using the order Condition B → Condition A, the frequency (load) of recognizing the appearance of the product from the captured image can be reduced.

[0132] In an embodiment in which the appearance of a product is recognized from an captured image when it is determined that condition B is met, the captured image may be acquired after it is determined that condition B is met and the appearance may be recognized from the captured image, or the captured image may be acquired before it is determined that condition B is met (captured images for all products), and the appearance may be recognized from the captured image when it is determined that condition B is met.

[0133] Condition B (the condition relating to the product whose barcode is scanned) may also be that the product is prone to fraud. In other words, the registration and settlement device 10 may determine the condition (condition A) based on the captured image when the barcode of a product prone to fraud is scanned (condition B is met). For example, if fraud often involves placing a single candy (regardless of whether it is part of a pack) on top of a 6-can beer set and scanning the candy's barcode, the device pre-stores the candy's product identification information and its appearance (information indicating its characteristics). When the candy's barcode is scanned (condition B is met), the registration and settlement device 10 recognizes the product's appearance from the captured image. If the recognized appearance differs from the candy's appearance (for example, if it is the appearance of a 6-can beer set), the registration and settlement device 10 determines that it is a fraudulent registration.

[0134] (Modified Example 6) This includes the content described in Modified Example 5, but the order of processing in steps S3, S4, and S5 may be swapped in the flowchart of Figure 4. The order of processing in steps S3, S4, and S15 may be swapped in the flowchart of Figure 6. The order of processing in steps S3 and S24 may be swapped in the flowchart of Figure 7. The order of processing in steps S3 and S15 may be swapped in the flowchart of Figure 8.

[0135] In other words, the order in which conditions A and B are met can be arbitrary (modified example 5), and if condition B consists of multiple subconditions (as in the flowchart in Figure 4 and the flowchart in Figure 6), the order in which each subcondition within condition B is met can be arbitrary. For example, one could first determine whether one subcondition of condition B is met, then whether condition A is met, and then whether the other subconditions of condition B are met.

[0136] (Modification 7) In the above embodiment, the registration and settlement device 10 was described in which the appearance of the product is captured by the imaging unit 110. However, instead of or in addition to the imaging unit 110, the appearance of the product may be captured by the built-in camera (scanner camera) of the scanner unit 104.

[0137] If the scanner unit 104 (scanner camera) is configured to capture images of the product's appearance on its own (without the imaging unit 110 capturing images of the product's appearance), the scanner unit 104 acquires the code (i.e., acquires the information necessary for the product's unit price and classification). Therefore, the registration and settlement device 10 can acquire both information about the product's appearance and information necessary for the product's unit price and classification using only the scanner unit 104.

[0138] In a configuration where the imaging unit 110 and the scanner unit 104 (scanner camera) capture images of the product's appearance, the product's appearance based on the image captured by the imaging unit 110 may be compared with the product's appearance based on the image captured by the scanner unit 104. Furthermore, the accuracy of appearance recognition may be improved by using both captured images.

[0139] (Modification 8) In the above embodiment (Figure 4), the registration and settlement device 10 determines that the registration is fraudulent and reports an error (proceeds to step S6) if the appearance of the product is a predetermined appearance (step S3 (YES)), the product identification information is for alcoholic beverages (step S4 (YES)), and the amount is less than a predetermined amount (step S5 (YES)). However, the registration and settlement device 10 may also determine that the registration is fraudulent and report an error (proceeds to step S6) if the appearance of the product is a predetermined appearance (step S3 (YES)) and the amount is less than a predetermined amount (step S5 (YES)). In other words, the processing in step S4 is not required in the flowchart of Figure 4. In this case, for example, the registration and settlement device 10 determines that the registration is fraudulent and reports an error if the appearance of the product is a predetermined appearance (for example, a 6-pack of beer) and the scanned product is less than a predetermined amount (for example, less than the amount of the highest-priced single item, beer).

[0140] (Modification 9) In the above embodiment, an example of notifying the customer in order to monitor fraud during customer registration was described. However, since store employees may also mistakenly register individual cans of beer that make up a 6-pack of beer, the store employees may also be notified. For example, in a registration and settlement device (not shown) that performs registration and settlement processing based on the operation of a store employee, or a registration and settlement device (face-to-face semi-self-service device, not shown) that performs registration processing based on the operation of a store employee and settlement processing based on the operation of a customer, or in a registration device 20 (not shown) that constitutes a semi-self-service system, or in a registration and settlement device 30 (not shown) when in face-to-face semi-self-service mode, if a store employee mistakenly scans the barcode of an individual can of beer that makes up a 6-pack of beer, a small screen such as small screen SG1 (a message such as "A store employee will be coming. Please wait a moment" is unnecessary as it is a notification to the store employee) may be displayed on the display unit on the store employee's side. Also, when a store employee is registering, the store employee calling process (step S7) is unnecessary.

[0141] (Modification 10) Some or all of the functions (input / output, storage, processing (including judgment)) in each device (registration and settlement device 10, registration and settlement device 30, portable registration terminal 40, etc.) may be implemented in a device other than the device described as the main entity of that function.

[0142] For example, the system may be configured to include an imaging processing device (a device having an imaging unit 110 and a processing unit) as a separate device from the registration and settlement device 10. This imaging processing device may recognize the product removal operation of taking products out of the shopping basket (or implement a part corresponding to the product removal operation recognition function), recognize the product movement operation of moving the products taken out of the shopping basket to the front of the main unit 11 (or implement a part corresponding to the product movement operation recognition function), or recognize the appearance of the products from the captured image. In other words, the imaging processing device may perform at least a part of the image processing functions described as functions of the registration and settlement device 10. Alternatively, for example, the cloud server 71 may recognize the appearance of the products from the captured image, or determine whether the appearance of the products is a predetermined appearance. In other words, the cloud server 71 may perform steps S3, S4, and S5 in Figure 4 (the flowcharts in Figures 6, 7, and 8 are similar). Furthermore, each device may function as a so-called thin client specialized in input / output interfaces, etc. For example, the cloud server 71 may execute a program to control the operation of each device, and each device may display a screen using its browsing function.

[0143] <Summary of Embodiments> [Technical field] This invention relates to a product processing system and program. [Background technology] Self-checkout systems where customers register their products are known (see, for example, Patent Document 1). [Prior art document] [Patent] [Patent Document 1] Japanese Unexamined Patent Publication No. 2020-017075 [Overview of the prefecture] [Problems the invention aims to solve] However, self-checkout systems, which are operated by customers, are susceptible to fraud, which is a challenge for customer-operated devices. This invention has been made in view of these circumstances and aims to provide a technology that reduces fraud in customer-operated devices. [Means for solving the problem]

[0144] (1) One embodiment for solving the above-mentioned problems is a product processing system characterized by comprising: an acquisition unit for acquiring the product code of a product to be registered; a recognition unit for recognizing the appearance of a product from an image captured in an area where the product is to be registered; a determination unit for determining whether predetermined conditions related to the product information of a product identified by the product code acquired by the acquisition unit and the appearance of the product recognized by the recognition unit are met; and a notification unit for notifying the operator based on the determination result of the determination unit. The configuration in (1) can reduce fraud. For example, fraud in devices operated by customers can be reduced simply and efficiently.

[0145] (2) The product processing system according to (1), characterized in that the determination unit determines, as a predetermined condition, whether the appearance of the product recognized by the recognition unit is a predetermined appearance and the product identified by the product code acquired by the acquisition unit is a product of a predetermined price or less, and the notification unit notifies the operator of the fraud when the determination unit determines that the appearance of the product recognized by the recognition unit is a predetermined appearance and the product identified by the product code acquired by the acquisition unit is a product of a predetermined price or less (see, for example, Modification 8). According to the configuration of (2), for example, fraud in devices operated by customers can be reduced in a simple and efficient manner.

[0146] (3) The product processing system according to (1), characterized in that the determination unit determines, as a predetermined condition, whether the appearance of the product recognized by the recognition unit is a predetermined appearance and the product identified by the product code acquired by the acquisition unit is an alcoholic beverage with a selling price of less than a predetermined amount, and the notification unit notifies the operator of the fraud when the determination unit determines that the appearance of the product recognized by the recognition unit is a predetermined appearance and the product identified by the product code acquired by the acquisition unit is an alcoholic beverage with a selling price of less than a predetermined amount (see, for example, Figure 4). According to the configuration of (3), for example, fraud in devices operated by customers can be reduced in a simple and efficient manner.

[0147] (4) The product processing system according to (1), characterized in that the determination unit determines, as a predetermined condition, whether the appearance of the product recognized by the recognition unit is a predetermined appearance and the product identified by the product code acquired by the acquisition unit is a single alcoholic beverage, and the notification unit notifies the operator of the fraud when the determination unit determines that the appearance of the product recognized by the recognition unit is a predetermined appearance and the product identified by the product code acquired by the acquisition unit is a single alcoholic beverage (see, for example, Figure 7). According to the configuration of (4), for example, fraud in devices operated by customers can be reduced in a simple and efficient manner.

[0148] (5) The determination unit determines, as a predetermined condition, whether the appearance of the product recognized by the recognition unit is a predetermined appearance and whether the product identified by the product code acquired by the acquisition unit is a single product that constitutes a pack of multiple products sold as such. The notification unit notifies the operator of the fraud when the determination unit determines that the appearance of the product recognized by the recognition unit is a predetermined appearance and the product identified by the product code acquired by the acquisition unit is a single product that constitutes a pack of multiple products sold as such (see, for example, Figures 6 and 8). This is the product processing system according to (1). According to the configuration of (5), for example, fraud in devices operated by customers can be reduced simply and efficiently.

[0149] (6) Another embodiment for solving the above-mentioned problems is a program that causes a computer used in a product processing system to function as an acquisition means for acquiring a product code of a product to be registered, a recognition means for recognizing the appearance of a product from an image captured in an area where the product is to be registered, a determination means for determining whether predetermined conditions related to the product information of a product identified by the product code acquired by the acquisition means and the appearance of the product recognized by the recognition means are met, and a notification means for notifying the operator based on the determination result of the determination means. According to the configuration in (6), fraud can be reduced.

[0150] Furthermore, the processing of each of the above-described devices (registration and settlement device 10, registration and settlement device 30, portable registration terminal 40, transaction status management device 50, etc.) may be performed by recording the program for realizing the functions of each device on a computer-readable recording medium, loading the program recorded on this recording medium into a computer system, and executing it. Here, "loading the program recorded on the recording medium into a computer system and executing it" includes installing the program into the computer system. Here, "computer system" includes hardware such as the OS and peripheral devices. Also, "computer system" may include multiple computer devices connected via a network including communication lines such as the Internet, WAN, LAN, and dedicated lines. Also, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into the computer system. Thus, the recording medium storing the program may also be a non-transient recording medium such as a CD-ROM. Also, the recording medium includes internal or external recording media that can be accessed from the distribution server for distributing the program. The program code stored on the distribution server's recording medium may be different from the program code in a format executable by the terminal device. In other words, the format in which the program is stored on the distribution server is irrelevant, as long as it can be downloaded from the distribution server and installed in a form that can be executed on the terminal device. Furthermore, the program may be divided into multiple parts, each downloaded at a different time and then combined on the terminal device, and each divided program may be distributed by a different distribution server. In addition, "computer-readable recording medium" includes volatile memory (RAM) within computer systems that act as servers or clients when a program is transmitted over a network, which retains the program for a certain period of time. Moreover, the above program may be intended to implement only a part of the functions described above.Furthermore, the above-mentioned functions may be realized in combination with programs already recorded in the computer system, such as so-called differential files (differential programs). [Explanation of Symbols]

[0151] 1...Sales system, 10...Registration and settlement device, 30...Registration and settlement device, 40...Mobile registration terminal, 50...Transaction status management device, 60...Store controller, 70...Head office server (cloud server), 71...Cloud server

Claims

1. The acquisition unit retrieves the product code of the product to be registered, A recognition unit that recognizes the appearance of a product from an image of the area where the product is registered, A determination unit that determines whether predetermined conditions related to the product information of a product identified by the product code acquired by the acquisition unit and the appearance of the product recognized by the recognition unit are met, A notification unit that notifies the operator based on the determination result of the determination unit. A product processing system characterized by being equipped with the following features.

2. The determination unit, As a predetermined condition, the recognition unit determines whether the appearance of the product recognized by the recognition unit is a predetermined appearance, and whether the product identified by the product code acquired by the acquisition unit is a product with a price less than a predetermined amount. The aforementioned notification department, If the determination unit determines that the appearance of the product recognized by the recognition unit is a predetermined appearance, and that the product identified by the product code acquired by the acquisition unit is a product priced below a predetermined amount, the determination unit notifies the operator of the fraud. The product processing system according to feature 1.

3. The determination unit, As a predetermined condition, the recognition unit determines whether the appearance of the product recognized by the recognition unit is a predetermined appearance, and whether the product identified by the product code acquired by the acquisition unit is an alcoholic beverage with a selling price below a predetermined amount. The aforementioned notification department, If the determination unit determines that the appearance of the product recognized by the recognition unit is a predetermined appearance, and that the product identified by the product code acquired by the acquisition unit is an alcoholic beverage with a selling price below a predetermined amount, the determination unit will notify the operator of the fraud. The product processing system according to feature 1.

4. The determination unit, As a predetermined condition, the recognition unit determines whether the appearance of the product recognized by the recognition unit is a predetermined appearance, and whether the product identified by the product code acquired by the acquisition unit is a single alcoholic beverage. The aforementioned notification department, If the determination unit determines that the appearance of the product recognized by the recognition unit is a predetermined appearance, and that the product identified by the product code acquired by the acquisition unit is a single alcoholic beverage, the determination unit will notify the operator of the fraud. The product processing system according to feature 1.

5. The determination unit, As a predetermined condition, the recognition unit determines whether the appearance of the product recognized by the recognition unit is a predetermined appearance, and whether the product identified by the product code acquired by the acquisition unit is a single product that constitutes a pack of multiple products sold as such. The aforementioned notification department, If the determination unit determines that the appearance of the product recognized by the recognition unit is a predetermined appearance, and that the product identified by the product code acquired by the acquisition unit is a single product that constitutes a pack of multiple products sold together, the determination unit will notify the operator of the fraud. The product processing system according to feature 1.

6. The computer used in the product processing system A means of obtaining the product code of the product to be registered. Recognition means for recognizing the appearance of a product from an image captured in the area where the product is registered. A determination means for determining whether predetermined conditions related to the product information of a product identified by the product code acquired by the acquisition means and the appearance of the product recognized by the recognition means are met. Notification means for notifying the operator based on the determination result of the determination means. A program that makes it function as such.

Images