Article registration system

Abstract

To suitably register an article by the use of a portable terminal.SOLUTION: An article registration system for registering an article to be purchased by a customer at a selling space, comprises: a registration part which registers the article to be purchased; an imaging part which can image an article secured by the customer as the article to be purchased; a recognition part which recognizes a specific action related to the securing of the article by the customer based on a captured image acquired from the imaging part; and a notification part which gives notice of information based on a registration state by the registration part and a recognition state by the recognition part, wherein the recognition part has a first recognition part, and a second recognition part which has higher accuracy of recognition than the first recognition part.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a product registration system. [Background technology]

[0002] BACKGROUND ART A system is known in which customers themselves register products using mobile terminals (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-055700 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is room for improvement in the above-described system, for example, in terms of measures against malicious or inexperienced customers.

[0005] The present invention has been made in view of the above circumstances, and has an object to provide a technique for suitably registering products using a mobile terminal. [Means for solving the problem]

[0006] One aspect of the product registration system for solving the above-mentioned problems is a product registration system that registers products purchased by customers in a sales area, and is equipped with a registration unit that registers the products to be purchased, an imaging unit that can capture images of products secured by customers as products to be purchased, a recognition unit that recognizes specific behaviors of customers regarding the securing of products based on images acquired from the imaging unit, and an alarm unit that notifies information based on the registration status by the registration unit and the recognition status by the recognition unit, wherein the recognition unit has a first recognition unit and a second recognition unit that has higher recognition accuracy than the first recognition unit. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a conceptual diagram of a network for explaining a sales system. [Figure 2] FIG. 10 is an explanatory diagram illustrating a cart to which a mobile terminal can be attached. [Figure 3] FIG. 10 is a sequence diagram illustrating an outline of the operations of the mobile terminal, the cloud server, and the settlement device. [Figure 4] 10 is a display example of a mobile terminal. [Figure 5] FIG. 10 is an explanatory diagram illustrating notification of fraud, etc. [Figure 6] FIG. 10 is an explanatory diagram illustrating notification of fraud, etc. [Figure 7] 10A and 10B are explanatory diagrams illustrating the executing entity and execution timing of two types of product insertion / removal recognition. [Figure 8] 10 is a display example of a payment device. [Figure 9] 10 is an example of a captured image for explaining recognition of inserting and removing a commodity. [Figure 10] 10 is an example of a captured image for explaining recognition of inserting and removing a commodity. [Figure 11] 10 is an example of a captured image for explaining recognition of inserting and removing a commodity. DETAILED DESCRIPTION OF THE INVENTION

[0008] Fig. 1 is a conceptual diagram of a network for explaining a sales system 1, which is an example of an embodiment. The sales system 1 (which may also be called a "product registration system" or a "sales data processing system") shown in Fig. 1 includes a headquarters server 10, a cloud server 20, a store controller 30, a transaction status management device 40 (for example, a personal computer), a settlement device 50, and a mobile terminal 60 (which may also be called an "information communication terminal," "mobile communication terminal," "mobile registration terminal," "portable registration terminal," etc.).

[0009] The sales system 1 can be introduced into various business types (supermarkets, convenience stores, drug stores, home centers, electronics retailers, etc.). In FIG. 1, the dashed line T indicates the inside of the store (product display area, checkout area, back-of-house area, office, etc.). As shown in FIG. 1, the store controller 30, transaction status management device 40, and payment device 50 are installed in the store. These are communicatively connected via a LAN 19 (which may be wired or wireless). A mobile terminal 60 is also installed in the store. The mobile terminal 60 is basically used while attached to (set in) a cart (shopping cart) 70. The headquarters server 10 and cloud server 20 are installed outside the store (for example, at a headquarters or data center). The devices installed in the store (the store controller 30, the transaction status management device 40, and the payment device 50) can communicate (directly or via the store controller 30) with the devices installed outside the store (the headquarters server 10 and the cloud server 20).

[0010] Sales system 1 provides a self-service (hereinafter, sometimes referred to as service A) in which a customer registers a product using a mobile terminal 60 and settles the payment using a payment settlement device 50. In other words, service A is a self-service in which the customer registers the product and settles the payment themselves, using mobile terminal 60 for product registration and settlement using payment settlement device 50.

[0011] Note that the number of devices (terminals) in FIG. 1 is an example. For example, while "..." is shown to the right of the payment device 50 in FIG. 1, the number of payment devices 50 installed in the store may be one or more. Also, for example, while two mobile terminals 60 are shown in FIG. 1, the number of mobile terminals 60 used in the store varies depending on the number of customers. Other devices may also be installed in the store. For example, the store may be equipped with a group of semi-self-service devices (registration device, payment device) that register products based on the operation of a store clerk and settle the payment based on the operation of a customer, or a full-self-service device (registration payment device) that registers products based on the operation of a customer and settles the payment based on the operation of the customer.

[0012] (Mobile terminal 60) The mobile terminal 60 is used by a customer. Specifically, the mobile terminal 60 is used when the customer registers a product in the store. The mobile terminal 60 is, for example, a smartphone, but may also be a tablet terminal. The mobile terminal 60 is basically owned by the customer.

[0013] Although not shown in the figure, the mobile terminal 60 includes, for example, a CPU, a memory unit, a display unit, an operation unit, an audio output unit (speaker), an audio input unit (microphone), an imaging unit (camera), a detection unit (sensor), and a communication unit.

[0014] The CPU is a central processing unit that controls the entire mobile terminal 60 by reading and executing programs stored in the storage unit. The storage unit stores various information (for example, information referenced or generated by the CPU, information obtained from the outside, etc.). The storage unit is a general term for one or more various storage components (for example, memory or storage, such as RAM, ROM, hard disk, SSD (Solid State Drive), etc.).

[0015] For example, the mobile terminal 60 may acquire (receive) an application program (hereinafter, App A) for using Service A from an external source (which may be a predetermined server, such as the cloud server 20) and store (install) it in a storage unit. The mobile terminal 60 may use App A to display various screens and execute various processes related to Service A. As described above, Service A is a self-service service in which the mobile terminal 60 is used for product registration and the settlement device 50 is used for payment, and App A is an application program that handles the mobile terminal 60 (product registration) portion of Service A.

[0016] The display unit is, for example, a liquid crystal display, and displays various information (including image information). The display unit is configured as a touch panel that can accept operations on the display screen. The operation unit accepts operations from an operator (customer). The operation unit is a collective term for one or more various operation members (various operators (hard buttons) provided on the housing. Note that, since the display unit is configured as a touch panel, it may be considered as part of the operation unit. The audio output unit outputs audio. The audio input unit inputs audio. The imaging unit acquires (generates) captured images (still images, moving images). The detection unit (measurement unit) detects or measures various events, etc. The detection unit is a collective term for, for example, one or more various detection members (GPS sensor, acceleration sensor, gyro sensor, etc.). Note that the operation unit, audio input unit, and imaging unit may also be considered as part of the detection unit. The communication unit transmits and receives various information. The communication unit is a collective term for one or more various communication members (various communication members such as 5G, wireless LAN, NFC (Near Field Communication), Bluetooth (registered trademark), etc.).

[0017] Needless to say, the mobile terminal 60 (as well as other devices) has a timekeeping unit (timekeeping function). The timekeeping unit enables the mobile terminal 60 to manage the current time, the elapsed time from a certain point in time, and the remaining time until a certain point in time.

[0018] For example, the mobile terminal 60 can manage the time that has elapsed since a product was added to a shopping basket or the time that has elapsed since a product was removed from the shopping basket. The mobile terminal 60 can manage the time that has elapsed since a product was registered (scan registration (described later), button registration (described later)). The mobile terminal 60 can manage the time that has elapsed since a product was canceled using the product cancel button (described later). The mobile terminal 60 can manage the time that has elapsed since a quantity change operation (a quantity decrease operation using the minus button (described later), or a quantity increase operation using the plus button (described later)) was performed. The mobile terminal 60 can manage the time that has elapsed since a certain screen (including a small screen) was displayed (started to be displayed) or since a certain screen (including a small screen) was erased.

[0019] The mobile terminal 60 can display an image captured by the imaging unit as a so-called through image (live view image) on the display unit. When the mobile terminal 60 is attached to the holder unit 72 (described later) and an image is captured by the imaging unit, the mobile terminal 60 obtains an image captured in a direction looking down on the shopping basket from the position of the holder unit 72 and displays the image on the display unit as a through image. do.

[0020] The mobile terminal 60 acquires a captured image by the imaging unit whether or not the image is displayed on the display unit as a through image. That is, the mobile terminal 60 acquires a captured image by the imaging unit whether or not the mobile terminal 60 is displaying a scan registration screen (described later) that displays a through image, or a button registration screen (described later) that does not display a through image.

[0021] The mobile terminal 60 transmits the captured image to the cloud server 20. The mobile terminal 60 may store the captured image in a storage unit.

[0022] The mobile terminal 60 may display the captured image captured by the imaging unit on the display unit and transmit it to the cloud server 20 without storing it in the storage unit, or may display the captured image captured by the imaging unit on the display unit and store it in the storage unit before transmitting it to the cloud server 20, or may display the captured image captured by the imaging unit on the display unit and store it (temporarily) in the storage unit, and later transmit all of the stored captured images together to the cloud server 20. Furthermore, the mobile terminal 60 may transmit the captured image captured by the imaging unit to the cloud server 20 without displaying it on the display unit or storing it in the storage unit, or may transmit the captured image captured by the imaging unit to the cloud server 20 without displaying it on the display unit or storing it in the storage unit before transmitting it to the cloud server 20, or may store the captured image captured by the imaging unit (temporarily) in the storage unit without displaying it on the display unit, and later transmit all of the stored captured images together to the cloud server 20.

[0023] The mobile terminal 60 has various image recognition functions (image processing functions). For example, the mobile terminal 60 has an object recognition function (object determination function) that recognizes objects in a captured image. In addition, the mobile terminal 60 has the following functions related to the object recognition function.

[0024] The mobile terminal 60 has a barcode reading function that recognizes a barcode in a captured image, reads the recognized barcode, and acquires a product code (for example, a JAN code or an in-store code).

[0025] The mobile terminal 60 has a product insertion / removal recognition function (product insertion / removal determination function) that recognizes (determines) a product insertion action (product insertion) by a customer to insert a product into a shopping basket and a product removal action (product removal) by a customer to remove a product from the shopping basket based on the captured image. The product insertion / removal recognition function will be described in detail later.

[0026] The mobile terminal 60 has a scanning action recognition function that recognizes a scanning action (holding action) made by a customer to scan a product (barcode) based on a captured image. Note that the scanning action recognition function may, for example, recognize (by pattern matching) whether something resembling a barcode (for example, an object with a predetermined pattern consisting of multiple line segments) has newly appeared in a predetermined area (for example, a code scan area, the area indicated by symbol A in FIG. 4(A)) in the captured image.

[0027] (Monitoring fraud, etc. using mobile terminal 60) The mobile terminal 60 monitors for fraudulent acts (fraud or mistakes) by customers based on the status of product registration, product cancellation, etc., and the status of product insertion and removal using the product insertion / removal recognition function. For example, the mobile terminal 60 monitors for unregistered insertion, unregistered insertion, uncancelled removal, and uncancelled removal.

[0028] An unregistered input is a state in which there is no product registration (including an increase in quantity) but there is product input (a state in which there is product input (product input is recognized) but the product is not registered). An unregistered input results in a loss for the store because the secured product (the input product) is not subject to settlement.

[0029] "Not registered and not added" means that the product has been registered (including an increase in quantity) but not added (the product has been registered but not added (the product addition has not been recognized)). "Not registered and not added" results in a loss for the customer because the product to be settled (registered product) has not been secured.

[0030] Uncancelled retrieval is a state in which there is no product cancellation (including a decrease in quantity) and product retrieval (a state in which a product has been retrieved (it has been recognized as being retrieved) but not cancelled). Uncancelled retrieval results in a loss for the customer because the product that has not been secured (the retrieved product) is subject to settlement (it has not been cancelled).

[0031] Cancellation not taken out is a state in which a product has been cancelled (including a reduction in quantity) but the product has not been taken out (a state in which a product has been cancelled but the product has not been taken out (a state in which the product has not been recognised)). Cancellation not taken out results in a loss for the store because the product that is not subject to settlement (the cancelled product) has been reserved (it has not been taken out).

[0032] (Scan registration, button registration) The mobile terminal 60 has two product registration methods (methods for acquiring product codes): scan registration and button registration. Scan registration is a product registration method based on scanning a barcode. Button registration is a product registration method based on operating (touching) a product button (a preset button for product registration that corresponds to each product displayed on the display).

[0033] (operation information) The mobile terminal 60 transmits operation information (information that a product has been registered (scan registration, button registration), information that a product has been cancelled (operation of the product cancel button (described later)), information that a quantity has been changed (quantity decrease operation using the minus button (described later), quantity increase operation using the plus button (described later)) and the like) at the mobile terminal 60 to the cloud server 20. The mobile terminal 60 may store the operation information in a storage unit.

[0034] The mobile terminal 60 may transmit the operation information to the cloud server 20 without storing it in the memory unit, may transmit the operation information to the cloud server 20 while storing it in the memory unit, or may store the operation information in the memory unit (temporarily store it) and later transmit the stored operation information together to the cloud server 20.

[0035] Fig. 2 is an explanatory diagram illustrating a cart 70 to which a mobile terminal 60 can be attached. As shown in Fig. 2, the cart 70 has a shopping basket placement section 71 on which a shopping basket (not shown) can be placed, and a holder section 72 to which the mobile terminal 60 can be attached. Although it has been explained that the mobile terminal 60 is basically used while attached to the cart 70, specifically, it is used while attached to the holder section 72.

[0036] 2, no shopping basket is placed on the shopping basket placement section 71, and no mobile terminal 60 is attached to the holder section 72, but a customer who comes to the store prepares by placing a shopping basket on the shopping basket placement section 71 and attaching the mobile terminal 60 to the holder section 72. In addition, a dedicated bag 90 is attached to the cart 70 for temporarily storing products that require confirmation by a store clerk at the time of checkout.

[0037] When the mobile terminal 60 is attached to the holder unit 72, as described above, it captures an image of the shopping basket viewed from the position of the holder unit 72. When a customer finds a desired product (product to purchase) in the product display area (display shelf), they pick up the product and move it below the holder unit 72 (the space between the holder unit 72 and the shopping basket) so that the barcode attached to the product can be read by the imaging unit.

[0038] (Payment device 50) The settlement device 50 is mainly used by customers. Specifically, the settlement device 50 settles the payment for products registered by the mobile terminal 60 based on the customer's operation. In other words, the sales system 1 provides a service in which the customer registers the products themselves using the mobile terminal 60 and settles the payment for them themselves using the settlement device 50.

[0039] Although not shown in the figures, settlement device 50 includes a CPU, a memory unit, a display unit, a scanner unit, a card payment unit (non-cash payment unit), a change dispenser (cash payment unit), a printing unit, an audio output unit, an imaging unit, a communication unit, and a sign pole. Note that settlement device 50 may further include a key operation unit consisting of one or more various operation members, a sensor (such as a human presence sensor) that detects (sensing, recognizing) the presence of a person, etc.

[0040] The CPU is a central processing unit that controls the entire settlement device 50 by reading and executing programs stored in the memory unit. The memory unit stores various information. The memory unit is a general term for one or more various storage components. The display unit is, for example, an LCD display that displays various information. The display unit is configured as a touch panel that can accept operations on the display screen. The scanner unit scans various codes (for example, settlement codes (described below), product codes, and clerk codes attached to gift certificates, point cards, clerk name tags, etc.) and reads each code.

[0041] The card payment unit is a payment mechanism that accepts various cards (credit cards, prepaid cards such as transportation cards, cash cards for debit payments, point cards for point payments, etc.). The change dispenser is a cash payment mechanism. The printing unit prints and issues various media (receipts, etc.). The audio output unit outputs audio (e.g., confirmation sounds, warning sounds, audio guidance, etc.). The imaging unit (camera) acquires captured images (still images, moving images). The communication unit is a collective term for one or more various communication components. The sign pole has one or more light-emitting units at its tip, and notifies various information (calling a store clerk, operating status, etc.) depending on the light-emitting mode of the light-emitting units.

[0042] (Headquarters Server 10) The headquarters server 10 is a server that manages the entire sales system 1. The headquarters server 10 stores various information (e.g., a product master, a product button file, etc.). The product master is a file that stores product names, prices, etc. in association with product codes. The product button file stores, for example, the product code of a product designated (specified) by the product button, category identification information identifying the category to which the product button belongs (i.e., category identification information identifying the category to which the product designated by the product button belongs), and an image (captured image, illustration, etc.) to be displayed on the product button, in association with product button identification information that identifies the product button. The headquarters server 10 also supplies various information to, for example, the cloud server 20 and the store controller 30. Although not shown, the headquarters server 10 includes at least a CPU, a memory unit, and a communication unit. The headquarters server 10 may be composed of multiple servers.

[0043] (Cloud Server 20) The cloud server 20 is a server that mainly supports product registration by the mobile terminal 60. The cloud server 20 stores various information (for example, product master, product button file, transaction information (cart information), etc.). The cloud server 20 also supplies various information to, for example, the mobile terminal 60 and the settlement device 50. Although not shown in the figure, the cloud server 20 includes at least a CPU, a storage unit, and a communication unit. The cloud server 20 may be composed of multiple servers.

[0044] Cloud server 20 has various image recognition functions, similar to mobile terminal 60. For example, cloud server 20 has an object recognition function that recognizes an object based on a captured image acquired (received) from mobile terminal 60.

[0045] In connection with the object recognition function, the cloud server 20 has a product insertion / removal recognition function that recognizes product insertion and product removal based on captured images acquired from the mobile terminal 60. The product insertion / removal recognition function will be described in detail later. The cloud server 20 has a barcode reading function that recognizes barcodes in captured images acquired from the mobile terminal 60, reads the recognized barcodes, and acquires the product codes.

[0046] The cloud server 20 may have a scan action recognition function that recognizes a scan action based on a captured image acquired from the mobile terminal 60.

[0047] (Monitoring of fraud, etc. by cloud server 20) Cloud server 20 monitors fraudulent behavior by customers on mobile terminal 60 (for example, unregistered input, unregistered input, uncancelled takeout, uncancelled takeout) based on the status of product registration, product cancellation, etc. on mobile terminal 60 and the status of product input and product takeout on mobile terminal 60 recognized by the product input / takeout recognition function. In other words, cloud server 20 monitors fraudulent behavior by customers on mobile terminal 60 based on the status of product registration, product cancellation, etc. on mobile terminal 60 based on operation information acquired from mobile terminal 60 and the status of product input and product takeout on mobile terminal 60 recognized by the product input / takeout recognition function based on captured images acquired from mobile terminal 60.

[0048] The mobile terminal 60 transmits the captured image and operation information to the cloud server 20 so that the cloud server 20 can monitor for fraudulent activities by customers. For example, the mobile terminal 60 transmits the captured image and operation information to the cloud server 20 so that the captured image and operation information can be compared on a time axis (for example, in the case of an unregistered input, it is possible to determine whether a product has been registered or the quantity has increased before and after the input of the product, and in the case of an unregistered input, it is possible to determine whether a product has been input before and after the product has been registered or the quantity has increased). As an example, the mobile terminal 60 may transmit the captured image to the cloud server 20 together with the time of image capture (or transmit the captured image to the cloud server 20 at the time of image capture), and transmit operation information to the cloud server 20 together with the time of operation (or transmit operation information to the cloud server 20 at the time of operation).

[0049] (Store Controller 30) The store controller 30 is a server that is mainly responsible for communicating information (e.g., product master, product button file, etc.) between devices installed inside the store (transaction status management device 40, settlement device 50) and devices installed outside the store (headquarters server 10, cloud server 20). Although not shown in the figure, the store controller 30 includes at least a CPU, a memory unit, and a communication unit.

[0050] (Transaction status management device 40) The transaction status management device 40 is a device (e.g., a personal computer) that manages (monitors and controls processing status, operating status, etc.) devices (store controller 30, settlement device 50) installed in the store. Although not shown in the figure, the transaction status management device 40 includes at least a CPU, a memory unit, a display unit (which may be configured as a touch panel), an operation unit (keyboard, mouse, etc.), and a communication unit. The transaction status management device 40 is capable of communicating with a printing device (not shown), and various information may be output from the printing device.

[0051] In addition to managing the devices installed in the store, the transaction status management device 40 may also access the headquarters server 10, etc., and perform maintenance (such as product settings) on the product master stored in the headquarters server 10, etc., or access the cloud server 20, and display or modify transaction information, etc. stored in the cloud server 20.

[0052] (Overview of operations of the mobile terminal 60, cloud server 20, and settlement device 50) Figure 3 is a sequence diagram outlining the operations of the mobile terminal 60, cloud server 20, and settlement device 50. The sequence diagram in Figure 3 shows the operations of each device (mobile terminal 60, cloud server 20, settlement device 50) from when the mobile terminal 60 displays an initial registration screen (a registration screen before the registration of one product item; the registration screen in Figure 4(A)) until the settlement device 50 displays a product list screen (Figure 8). Note that the sequence diagram in Figure 3 mainly explains the cooperation between each device, and therefore omits parts related to the recognition of product insertion, etc. (product insertion, product removal) by the product insertion / removal recognition function and the monitoring of fraud, etc.

[0053] It is assumed that the mobile terminal 60 displays a service start screen (menu screen, not shown) based on a customer's operation, and displays an initial registration screen based on the operation of a shopping start instruction button on the service start screen. It is also assumed that when the initial registration screen is displayed (based on the operation of the shopping start instruction button), transaction identification information that identifies a transaction made by the mobile terminal 60 (or transaction identification information of the transaction) is acquired (received) from the cloud server 20.

[0054] Step S1: The mobile terminal 60 acquires a product code. For example, the mobile terminal 60 acquires a product code by scanning a barcode (scan registration). The mobile terminal 60 also acquires a product code (a product code corresponding to the product button identification information of the product button) by operating a product button (button registration). The mobile terminal 60 may store the product code in a storage unit. Step S2: The mobile terminal 60 transmits to the cloud server 20 the transaction identification information of the transaction and the product code acquired in step S1.

[0055] Step S3: The cloud server 20 receives the transaction identification information and the product code from the mobile terminal 60. Step S4: Cloud server 20 updates the transaction information for the transaction (the transaction corresponding to the transaction identification information received in step S3). Specifically, cloud server 20 acquires (extracts) product information (product name, price, etc.) corresponding to the product code received in step S3 from the product master. If the product code contains some or all of the product information, cloud server 20 may acquire the product information contained in the product code from the product code without referencing the product master. Next, cloud server 20 stores (adds) the product code and the product information to the transaction information for the transaction as registration information for the product. Step S5: The cloud server 20 transmits the product information (the product information acquired in step S4) to the mobile terminal 60.

[0056] Step S6: The mobile terminal 60 receives the product information from the cloud server 20. The mobile terminal 60 may store the product information in a storage unit. Step S7: The mobile terminal 60 displays a registration screen. Specifically, in the case of the first product, the mobile terminal 60 generates and displays, instead of the initial registration screen, a registration screen (for example, the registration screen in FIG. 4(C)) in which the product information of the first product received in step S6 is reflected as the registration information of the first product, and in the case of the second and subsequent products, generates and displays a registration screen in which the product information of the second and subsequent products received in step S6 is reflected as the registration information of the second and subsequent products (updates the registration screen).

[0057] After step S7, the system may either acquire a product code based on the customer's scan or product button operation (return to step S1), or generate and display a settlement code based on the customer's operation of the checkout instruction button BT631 (described below) (proceed to step S8). In other words, steps S1 to S7 are repeatedly executed each time a product code is acquired.

[0058] In step S5, cloud server 20 may transmit the updated transaction information (the transaction information updated in step S4) to mobile terminal 60, and in step S7, mobile terminal 60 may generate and display a registration screen reflecting the contents of the updated transaction information (a registration screen reflecting product information for each product as registration information). Also, in step S5, cloud server 20 may generate a registration screen reflecting the contents of the updated transaction information, transmit screen information of the registration screen to mobile terminal 60, and in step S7, mobile terminal 60 may display the registration screen based on the screen information.

[0059] Step S8: The mobile terminal 60 generates and displays a settlement code in response to the customer's operation of the checkout instruction button BT631. The settlement code is settlement information that has been coded (e.g., a two-dimensional code). The settlement information is information required to obtain the transaction information of the transaction from the cloud server 20 (e.g., transaction identification information that identifies the transaction (transaction information of the transaction), or location information of the transaction information of the transaction (a storage location within the sales system 1, such as a URI (Uniform Resource Identifier))).

[0060] Note that mobile terminal 60 may notify cloud server 20 that the checkout instruction button BT631 has been operated, and may receive and display a settlement code from cloud server 20. In other words, when cloud server 20 receives a notification from mobile terminal 60 that the checkout instruction button BT631 has been operated, it generates a settlement code and sends it to mobile terminal 60. Mobile terminal 60 displays the settlement code received from cloud server 20.

[0061] Furthermore, mobile terminal 60 may notify cloud server 20 that the checkout instruction button BT631 has been operated, and may generate and display a settlement code after receiving a generation instruction from cloud server 20. In other words, when cloud server 20 receives a notification from mobile terminal 60 that the checkout instruction button BT631 has been operated, cloud server 20 may not generate a settlement code itself, but may instead control the timing at which mobile terminal 60 generates and displays the settlement code using the generation instruction.

[0062] The customer causes the settlement device 50 to read the settlement code displayed on the mobile terminal 60. Note that the settlement device 50 may display a guidance screen in the standby state to guide the customer in reading the settlement code (reading operation).

[0063] Step S9: Based on the customer's reading operation, the settlement device 50 reads the settlement code displayed on the mobile terminal 60. That is, the settlement device 50 obtains transaction identification information that identifies the transaction. Step S10: The settlement device 50 requests transaction information for the transaction from the cloud server 20. For example, the settlement device 50 transmits request information (transaction information acquisition request information) including transaction identification information that identifies the transaction to the cloud server 20.

[0064] Step S11: The cloud server 20 receives the request (transaction information acquisition request information). Step S12: The cloud server 20 transmits to the settlement device 50 the transaction information of the transaction (the transaction corresponding to the transaction identification information included in the transaction information acquisition request information received in step S11).

[0065] Step S13: The settlement device 50 receives the transaction information from the cloud server 20. The settlement device 50 may store the transaction information in a storage unit. Step S14: The settlement device 50 displays a product list screen (not shown). Specifically, the settlement device 50 generates and displays a product list screen that reflects the contents of the transaction information received in step S13 (a product list screen that reflects the product information of each product as registration information).

[0066] In step S12, the cloud server 20 may generate a product list screen that reflects the contents of the transaction information for the transaction, and transmit the screen information of the product list screen to the settlement device 50, and in step S14, the mobile terminal 60 may display the product list screen based on the screen information.

[0067] (Example of display on mobile terminal 60) Fig. 4 is a display example of the mobile terminal 60. It is assumed that the mobile terminal 60 is already attached to the cart 70 (holder unit 72). When a scanning operation is not being performed, the image captured when looking down on the shopping basket from the position of the holder unit 72 captures the shopping basket and items already added to the shopping basket, and when a scanning operation is being performed, the image captured when looking down on the shopping basket from the position of the holder unit 72 captures the items that are the target of the scanning operation, the shopping basket, and items that have already been added to the shopping basket, but for simplicity's sake, Fig. 4 does not show items other than the items that are the target of the scanning operation (the shopping basket and items already added to the shopping basket, etc.).

[0068] 4(A) shows a registration screen (initial registration screen). For example, the mobile terminal 60 displays a service start screen (menu screen, not shown) based on a customer's operation, and displays the registration screen (initial registration screen) based on the customer's operation of a shopping start instruction button on the service start screen.

[0069] The registration screen in Fig. 4(A) is provided with a captured image display area HR620, a registration information display area HR630, etc. The captured image display area HR620 is an area that mainly displays a captured image. The registration information display area HR630 is an area that mainly displays product registration information (transaction details). The captured image display area HR620 (above the captured image display area HR620) is provided with a registration method selection button display area HR621, in which a recommendation button BT624 is displayed. The registration method selection button display area HR621 is an area that displays a registration method selection button.

[0070] The registration method selection button is a button that allows selection (switching) of the product registration method (scan registration / button registration). In the registration method selection button display area HR621, the scan registration button BT622 (a button displayed as "Scan" in this example) and the button registration button BT623 (a button displayed as "No barcode" in this example) are displayed as registration method selection buttons. The customer operates (touches) the scan registration button BT622 when switching the product registration method from button registration to scan registration, and operates the button registration button BT623 when switching the product registration method from scan registration to button registration.

[0071] When scan registration is selected, the mobile terminal 60 displays the scan registration button BT622 in a display mode indicating a selected state (shown with a thick frame in the figure), and displays the button registration button BT623 in a display mode indicating a non-selected state (shown with a dashed line in the figure). When button registration is selected, the mobile terminal 60 displays the button registration button BT623 in a display mode indicating a selected state, and displays the scan registration button BT622 in a display mode indicating a non-selected state. On the registration screen in Fig. 4(A), the scan registration button BT622 is displayed with a thick frame, and the button registration button BT623 is displayed with a dashed line (scan registration is selected).

[0072] When the recommendation button BT624 is operated (touched), the mobile terminal 60 displays a recommended product screen (not shown) that presents recommended products. Note that the "12" in "★12" in this example indicates that there are 12 recommended products.

[0073] A registration screen on which scan registration is selected may be referred to as a scan registration screen, and a registration screen on which button registration is selected may be referred to as a button registration screen.

[0074] Fig. 4(B) is a display example of a scene following Fig. 4(A). When the barcode of the loaf of bread is being imaged by the customer's action, the mobile terminal 60 displays a registration screen (scan registration screen) as shown in Fig. 4(B).

[0075] FIG. 4(C) is a display example of a scene following FIG. 4(B). When the mobile terminal 60 acquires the product code of the bread from the barcode, it displays a registration screen (scan registration screen) as shown in FIG. 4(C). The registration information display area HR630 on the registration screen in FIG. 4(C) displays the registration information of the scanned and registered product (bread). That is, the mobile terminal 60 acquires the product code, transmits the product code, etc. to the cloud server 20 (step S2 in FIG. 3), and displays a registration screen as shown in FIG. 4(C) (step S7 in FIG. 3). After registering the product (one item), the mobile terminal 60 operably displays a checkout instruction button BT631 (a button displaying "Proceed to Checkout" in this example) that accepts checkout instruction operations, as shown in FIG. 4(C). The quantity button BT632 displays the purchase quantity and is operated to change the purchase quantity or cancel the purchase.

[0076] Fig. 4(D) is a display example of a scene following Fig. 4(C). When the button registration button BT623 is operated on the registration screen of Fig. 4(C), the mobile terminal 60 displays a registration screen (button registration screen) as shown in Fig. 4(D). The registration screen of Fig. 4(D) is provided with a registration method selection button display area HR621, a tab display area HR640, a registration information display area HR630, a product button display area HR660, etc. A search button BT625 is displayed to the right of the registration method selection button display area HR621.

[0077] On the registration screen of FIG. 4(D), the scan registration button BT622 is displayed in a dashed line, and the button registration button BT623 is displayed in a thick line (button registration is selected).

[0078] When the search button BT625 is operated, the mobile terminal 60 accepts character input for searching for a product button. For example, when the search button BT625 is operated, the mobile terminal 60 displays a character input area and accepts character input from the customer in the character input area.

[0079] The tab display area HR640 is an area that displays five selectable (switchable) tabs (all tab, fruit tab, vegetable tab, fresh fish tab, prepared food tab). The four tabs other than the all tab (fruit tab, vegetable tab, fresh fish tab, prepared food tab) correspond to the four categories (fruit, vegetable, fresh fish, prepared food) in which products to be registered by button registration exist. The all tab corresponds to all of the above four categories. In the registration screen of Figure 4(D), the all tab is selected. The underline on the all tab indicates that the all tab is selected.

[0080] As described above, the registration information display area HR630 is an area that mainly displays the registration information of a product. The registration information display area HR630 on the registration screen (button registration screen) of FIG. 4(D) displays the registration information of the product (loaf of bread) that was scanned and registered.

[0081] The product button display area HR660 is an area that displays product buttons for products that belong to the category corresponding to the tab selected (touched) in the tab display area HR640. In the registration screen of Fig. 4(D), the All tab is selected, so all product buttons (product buttons for products belonging to Fruits, Vegetables, Fresh Fish, and Prepared Foods) are displayed in the product button display area HR660.

[0082] If a customer selects the Fruits tab in the tab display area HR640, the mobile terminal 60 displays in the tab display area HR640 an indication that the Fruits tab has been selected (by underlining in this example), and displays product buttons for products belonging to Fruits in the product button display area HR660. For example, if a customer selects the Fruits tab in the tab display area HR640, the mobile terminal 60 identifies the category (fruits) corresponding to the Fruits tab. Next, the mobile terminal 60 references the product button file stored in the cloud server 20, and displays product buttons corresponding to the category identification information that identifies fruits (i.e., product buttons for products belonging to Fruits) in the product button display area HR660. The same applies if a customer selects the Vegetables tab, Fresh Fish tab, or Prepared Food tab.

[0083] The mobile terminal 60 may acquire the product button file together with the app A when acquiring the app A, or may acquire the product button file as part of the app A when acquiring the app A and store it in the storage unit. In this case, the mobile terminal 60 refers to the product button file stored therein and displays the product button in the product button display area HR660.

[0084] Fig. 4(E) is a display example of a scene following Fig. 4(D). When the pear product button (pear button) is operated on the registration screen of Fig. 4(D), the mobile terminal 60 displays a registration screen (button registration screen) as shown in Fig. 4(E). The registration information display area HR630 of the registration screen (button registration screen) of Fig. 4(E) displays the registration information of the scanned and registered product (bread) and the registration information of the button-registered product (pear).

[0085] When the pear button (pear button) is operated on the registration screen of Fig. 4(D), the mobile terminal 60 may display a small screen (pop-up screen, not shown) on top of the registration screen of Fig. 4(D) on which the purchase quantity can be specified, and when an operation to determine the purchase quantity is performed on the small screen, the small screen may be cleared and a registration screen such as that shown in Fig. 4(E) may be displayed. In other words, in the case of button registration, the purchase quantity may be confirmed with the customer immediately after the operation of the product button.

[0086] Fig. 4(F) is a display example of a scene following Fig. 4(E). When the scan registration button BT622 is operated from the scene of Fig. 4(E), the mobile terminal 60 displays a registration screen (scan registration screen) as shown in Fig. 4(F). The registration information display area HR630 of the registration screen (scan registration screen) of Fig. 4(F) displays the registration information of the scan-registered product (loaf of bread) and the registration information of the button-registered product (pear).

[0087] FIG. 4(G) is an example display of a scene following FIG. 4(F). When the checkout instruction button BT631 is operated from the scene of FIG. 4(F), the mobile terminal 60 displays the settlement code display screen as shown in FIG. 4(G). The settlement code display screen of FIG. 4(G) displays the settlement code GA641, a message MS642 that instructs the settlement machine (payment device 50) to read the settlement code GA641, and an explanatory image GA643 that explains how to read the settlement code GA641 from the settlement machine. The settlement code display screen of FIG. 4(G) also displays an operable Return to Shopping button BT649. When the Return to Shopping button BT649 is operated, the registration screen before the checkout instruction button BT631 was operated is displayed. Note that the mobile terminal 60 also displays the settlement code display screen of FIG. 4(G) when the checkout instruction button BT631 is operated from each of the scenes of FIGS. 4(C) to 4(E).

[0088] FIG. 4(H) is a display example of another scene following FIG. 4(C). When the quantity button BT632 is operated, the mobile terminal 60 displays a small screen (pop-up screen) SG1 in front of the registration screen of FIG. 4(C), as shown in FIG. 4(H). The mobile terminal 60 tone-downs the back (except for the registration information portion of the product) to make the small screen SG1 on the front and the product stand out, but can accept a touch on any location on the back as an operation to clear the small screen SG1. In other words, while the small screen SG1 is displayed, the area behind the small screen SG1 is the area that accepts an operation to clear the small screen SG1. Note that the mobile terminal 60 also displays the small screen SG1 in front of the respective registration screen when the checkout instruction button BT631 is operated from each of the scenes of FIG. 4(D) to FIG. 4(F).

[0089] The small screen SG1 displays the purchase quantity of the registered product (bread) (displayed as "xn") and the total price for the purchase quantity of that product, and also has a product cancellation button (in this example, a button displayed as "Cancel Product"), a minus button (in this example, a button displayed as "-"), and a plus button (in this example, a button displayed as "+").

[0090] The product cancel button is a button that is operated when canceling the registration (all items) of the registered product. The minus button is a button that is operated when decreasing the purchase quantity of the registered product. A single operation of the minus button (a quantity decrease operation) decreases the purchase quantity by one. In the example shown in FIG. 4(H), the purchase quantity is "1", so the minus button cannot be operated. The plus button is a button that is operated when increasing the purchase quantity of the registered product. A single operation of the plus button (a quantity increase operation) increases the purchase quantity by one. For example, from the scene in FIG. 4(H), if the plus button on the small screen SG1 is operated twice, followed by an operation on the area behind the small screen SG1, the mobile terminal 60 increases the quantity of the registered product from 1 to 3. Also, from the scene in FIG. 4(H), if the product cancel button on the small screen SG1 is operated, followed by an operation on the area behind the small screen SG1, the mobile terminal 60 cancels the registration of the registered product.

[0091] 5 and 6 are explanatory diagrams for explaining notification of fraud, etc. The mobile terminal 60 notifies the customer when fraud, etc. occurs. Note that in Figs. 5(B) to 5(D) and Figs. 6(A) to 6(F), the back of the small screen (registration screen, etc.) is not shown, and only the small screen on the front is shown.

[0092] Fig. 5(A) is an example of a notification in the case of an unregistered input. In the case of an unregistered input, the mobile terminal 60 displays the small screen SG20 as shown in Fig. 5(A). Specifically, the mobile terminal 60 displays the small screen SG20 if there is no product registration or quantity increase before or after recognizing the input of a product. As an example, the mobile terminal 60 may display the small screen SG20 if there is no scan registration, button registration, or quantity increase within a predetermined time (e.g., 5 seconds) before the timing of recognizing the input of the product, and if there is no button registration or quantity increase within a predetermined time (e.g., 5 seconds) after the timing of recognizing the input of the product.

[0093] The small screen SG20 in FIG. 5(A) displays a message ("Have you registered the product?" in the illustrated example) that alerts the customer to product registration. In other words, because there is a possibility that "registration is forgotten (forgetting to increase the quantity)," the mobile terminal 60 pops up the small screen SG20 to alert the customer to product registration. In the scene in FIG. 5(A), there is a possibility that registration of two products (e.g., pears) is forgotten, so the mobile terminal 60 alerts the customer to product registration of the two products. The mobile terminal 60 may output audio in addition to displaying the small screen SG20. In other words, the mobile terminal 60 may alert the customer to unregistered input (forgetting to register) by displaying and audio. The mobile terminal 60 erases the small screen SG20 when a predetermined time (e.g., 3 seconds) has elapsed since the small screen SG20 started to be displayed.

[0094] Fig. 5(B) is an example of a notification when a cancellation has not been made and a product has not been removed. When a cancellation has not been made and a product has not been removed, the mobile terminal 60 displays the small screen SG30 as shown in Fig. 5(B). Specifically, the mobile terminal 60 displays the small screen SG30 when it does not recognize a product removal before or after a product cancellation or a quantity reduction. As an example, the mobile terminal 60 may display the small screen SG30 when it does not recognize a product removal within a predetermined time (e.g., 5 seconds) before the timing of the product cancellation (or quantity reduction) and when it does not recognize a product removal within a predetermined time (e.g., 5 seconds) after the timing of the product cancellation (or quantity reduction).

[0095] The small screen SG30 in FIG. 5(B) displays a message ("Have you taken out the product?" in the illustrated example) to alert the customer to taking out the product. In other words, since there is a possibility that the customer may have "forgotten to take out" the product, the mobile terminal 60 pops up the small screen SG30 to alert the customer to taking out the product. In the scene in FIG. 5(B), for example, since there is a possibility that the customer may have forgotten to take out the first canceled product (loaf of bread), the mobile terminal 60 alerts the customer to take out the first canceled product. The mobile terminal 60 may output audio in addition to displaying the small screen SG30. In other words, the mobile terminal 60 may alert the customer to the fact that the canceled product has not been taken out (forgotten to take out) by displaying and audio. The mobile terminal 60 erases the small screen SG30 when a predetermined time (for example, 3 seconds) has elapsed since the small screen SG30 started to be displayed.

[0096] Fig. 5(C) is an example of a notification when registration has not been performed. When registration has not been performed, the mobile terminal 60 displays the small screen SG40 as shown in Fig. 5(C). Specifically, the mobile terminal 60 displays the small screen SG40 when it does not recognize the insertion of a product before or after the product registration or quantity increase. As an example, the mobile terminal 60 may display the small screen SG40 when it does not recognize the insertion of a product within a predetermined time (e.g., 5 seconds) after the scan registration or button registration or quantity increase timing, and when it does not recognize the insertion of a product within a predetermined time (e.g., 5 seconds) before the button registration or quantity increase timing.

[0097] The small screen SG40 in FIG. 5(C) displays a message ("Have you inserted the product?" in the illustrated example) that alerts the customer to inserting the product. In other words, since there is a possibility that the product may have been "forgotten to be inserted," the mobile terminal 60 pops up the small screen SG40 to alert the customer to inserting the product. In the scene in FIG. 5(C), for example, since there is a possibility that the product registered as the first item (e.g., bread) may have been forgotten to be inserted, the mobile terminal 60 alerts the customer to insert the first item. The mobile terminal 60 may output audio in addition to displaying the small screen SG40. In other words, the mobile terminal 60 may alert the customer to the non-registered insertion (forgotten to be inserted) by displaying and audio. The mobile terminal 60 erases the small screen SG40 when a predetermined time (e.g., 3 seconds) has elapsed since the small screen SG40 started to be displayed.

[0098] Fig. 5(D) is an example of notification in the case of uncancelled removal. In the case of uncancelled removal, the mobile terminal 60 displays the small screen SG50 as shown in Fig. 5(D). Specifically, the mobile terminal 60 displays the small screen SG50 if there is no product cancellation or quantity reduction before or after recognizing the product removal. As an example, the mobile terminal 60 may display the small screen SG50 if there is no product cancellation or quantity reduction within a predetermined time (e.g., 5 seconds) before the timing of recognizing the product removal, and if there is no product cancellation or quantity reduction within a predetermined time (e.g., 5 seconds) after the timing of recognizing the product removal.

[0099] The small screen SG50 in FIG. 5(D) displays a message ("Did you cancel the product?" in the illustrated example) that alerts the customer to the cancellation of the product. In other words, since there is a possibility that "cancellation has been forgotten (forgetting to reduce the quantity)," the mobile terminal 60 pops up the small screen SG50 to alert the customer. In the scene in FIG. 5(C), for example, since there is a possibility that cancellation of one item of product (loaf of bread) that has already been taken out has been forgotten, the mobile terminal 60 alerts the customer to the cancellation of the one item of product. The mobile terminal 60 may output audio in addition to displaying the small screen SG50. In other words, the mobile terminal 60 may alert the customer to the uncancelled takeout (forgetting to cancel) by displaying and audio. The mobile terminal 60 erases the small screen SG50 when a predetermined time (for example, 3 seconds) has elapsed since the small screen SG50 started to be displayed.

[0100] 5A shows an example in which the small screen SG20 is displayed in front of the registration information display area HR630 of the scan registration screen, but the small screen SG20 may also be displayed in front of the captured image display area HR620. For example, if a large amount of registration information is displayed in the registration information display area HR630 and the registration information is hidden by the display of the small screen SG20, the small screen SG20 may be displayed in front of the captured image display area HR620. The same applies to the small screens SG30, SG40, and SG50. The same applies to the small screens SG10, SG11, SG22, SG32, SG42, and SG52 in FIG.

[0101] Furthermore, on the small screen SG20, instead of or in addition to a message that draws the customer's attention to product registration, a message indicating that confirmation by a store clerk may be required for product registration (for example, "At the checkout machine, a store clerk may check whether the product is registered") may be displayed. On the small screen SG30, instead of or in addition to a message that draws the customer's attention to product removal, a message indicating that confirmation by a store clerk may be required for product removal (for example, "At the checkout machine, a store clerk may check whether the product has been removed from the shopping basket") may be displayed. On the small screen SG40, instead of or in addition to a message that draws the customer's attention to product insertion, a message indicating that confirmation by a store clerk may be required for product insertion (for example, "At the checkout machine, a store clerk may check whether the product has been inserted into the shopping basket") may be displayed. On the small screen SG50, instead of or in addition to a message alerting the customer about canceling the product, a message may be displayed indicating that confirmation by a store clerk may be required before canceling the product (for example, "A store clerk may confirm at the checkout machine whether the product has been canceled").

[0102] Also, although it has been explained that small screen SG20 is erased when a predetermined time has elapsed since the start of display, an OK button may be provided on small screen SG20, and small screen SG20 may be erased when the OK button is pressed. The same applies to small screens SG30, SG40, and SG50. The same applies to small screens SG22, SG32, SG42, and SG52 in FIG. 6.

[0103] Furthermore, although the small screen SG20 in Fig. 5(A) is displayed in front of the scan registration screen (the registration screen in Fig. 4(C)), the small screen SG20 can also be displayed in front of the button registration screen (for example, the registration screens in Fig. 4(D) and Fig. 4(E)) or the settlement code display screen (Fig. 4(G)). The same is true for the small screens SG30, SG40, and SG50. The same is true for the small screens SG10, SG11, SG22, SG32, SG42, and SG52 in Fig. 6.

[0104] In the case of an unregistered input, the mobile terminal 60 may display a small screen SG10 as shown in FIG. 6(A) instead of the small screen SG20 of FIG. 5(A). The small screen SG10 is a small screen that prompts the customer to answer (response) as to whether or not the product has been input into the shopping basket. The small screen SG10 of FIG. 6(A) displays a message confirming whether or not the product has been input into the shopping basket (in the illustrated example, "Have you added the product to the basket?"), and displays a Yes button and a No button that can be operated. If the product has been input into the shopping basket, the customer operates the Yes button, and if the product has not been input into the shopping basket, the customer operates the No button. When the Yes button is pressed on the small screen SG10 of FIG. 6(A) (when the customer responds that they have inserted the product), the mobile terminal 60 displays a small screen SG22 as shown in FIG. 6(B) instead of the small screen SG10 of FIG. 6(A). When the No button is pressed (when the customer responds that they have not inserted the product), the mobile terminal 60 simply erases the small screen SG10 of FIG. 6(A). The small screen SG22 of FIG. 6(B) displays a message instructing the customer to register the inserted product (in the illustrated example, "Please register the inserted product"). The mobile terminal 60 erases the small screen SG22 when a predetermined time (e.g., 3 seconds) has elapsed since the small screen SG22 started to be displayed. As described above, an OK button may be provided on the small screen SG22, and the small screen SG22 may be erased when the OK button is pressed.

[0105] In the case of cancellation and non-removal, the mobile terminal 60 may display a small screen SG11 as shown in FIG. 6(C) instead of the small screen SG30 of FIG. 5(B). The small screen SG11 is a small screen that prompts the customer to answer (response) whether or not the product has been removed from the shopping basket. The small screen SG11 of FIG. 6(C) displays a message confirming whether or not the product has been removed from the shopping basket (in the illustrated example, "Did you remove the product from the basket?"), and displays a Yes button and a No button that can be operated. If the product has been removed from the shopping basket, the customer operates the Yes button, and if the product has not been removed from the shopping basket, the customer operates the No button. When the NO button is pressed on the small screen SG11 of FIG. 6(C) (when the customer responds that they have not removed the product), the mobile terminal 60 displays the small screen SG32 shown in FIG. 6(D) instead of the small screen SG11 of FIG. 6(C). When the YES button is pressed (when the customer responds that they have removed the product), the mobile terminal 60 simply erases the small screen SG11 of FIG. 6(C). The small screen SG32 of FIG. 6(D) displays a message instructing the customer to remove the canceled product (in the illustrated example, "Please remove the canceled product"). The mobile terminal 60 erases the small screen SG32 when a predetermined time (e.g., 3 seconds) has elapsed since the display of the small screen SG32 began. As described above, an OK button may be provided on the small screen SG32, and the small screen SG32 may be erased when the OK button is pressed.

[0106] If the product has not been registered, the mobile terminal 60 may display the small screen SG10 of FIG. 6(A) instead of the small screen SG40 of FIG. 5(C). When the No button is pressed on the small screen SG10 of FIG. 6(A) (when the customer responds that they have not inserted the product), the mobile terminal 60 displays the small screen SG42 shown in FIG. 6(E) instead of the small screen SG10 of FIG. 6(A). When the Yes button is pressed (when the customer responds that they have inserted the product), the mobile terminal 60 simply erases the small screen SG10 of FIG. 6(A). The small screen SG42 of FIG. 6(E) displays a message instructing the customer to insert the registered product (in the illustrated example, "Please insert the registered product"). The mobile terminal 60 erases the small screen SG42 when a predetermined time (e.g., 3 seconds) has elapsed since the small screen SG42 started to be displayed. As described above, an OK button may be provided on the small screen SG42, and the small screen SG42 may be erased when the OK button is pressed.

[0107] In the case of an uncancelled removal, the mobile terminal 60 may display the small screen SG11 of FIG. 6(C) instead of the small screen SG50 of FIG. 5(D). When the Yes button is pressed on the small screen SG11 of FIG. 6(C) (when the customer responds that the product has been removed), the mobile terminal 60 displays the small screen SG52 shown in FIG. 6(F) instead of the small screen SG11 of FIG. 6(C). When the No button is pressed (when the customer responds that the product has not been removed), the mobile terminal 60 simply erases the small screen SG11 of FIG. 6(C). The small screen SG52 of FIG. 6(F) displays a message instructing the customer to cancel the removed product (in the illustrated example, "Please cancel the removed product"). The mobile terminal 60 erases the small screen SG52 when a predetermined time (e.g., 3 seconds) has elapsed since the small screen SG52 started to be displayed. As described above, an OK button may be provided on the small screen SG52, and the small screen SG52 may be erased when the OK button is pressed.

[0108] A small screen intended for the customer to confirm something (a small screen for confirmation only) may be called, for example, a "confirmation screen" or a "message screen." A small screen that requests a response from the customer may be called, for example, a "response required screen" or a "response acceptance screen."

[0109] For example, in the case of the small screens in Figures 5 and 6, the small screen SG20 in Figure 5 and the small screen SG22 in Figure 6 are "confirmation screens (message screens)." Note that for small screens SG20 and SG22 when an OK button is provided, the OK operation does not require an answer to a question, so they may be called "confirmation screens (message screens)," or they may be called "answer-required screens (response acceptance screens)" because they do require a response. The same applies to the small screens SG30, SG40, and SG50 in Figure 5, and the small screens SG32, SG42, and SG52 in Figure 6.

[0110] 6 and SG11 are "answer required screens (response acceptance screens)." As described above, the small screen SG20 and the like in the case where the OK button is arranged may also be called "answer required screens (response acceptance screens)."

[0111] (Product input / removal recognition) Hereinafter, the recognition (determination) of the insertion of a product, etc., by the product insertion / removal recognition function may be referred to as product insertion / removal recognition (product insertion / removal determination). The product insertion / removal recognition function (product insertion / removal recognition) may be realized by comparing a captured image when a product is actually inserted into a shopping basket with a captured image when a product is actually removed from the shopping basket.

[0112] For example, a large number of captured images of products actually being added to a shopping basket are collected in advance, and their feature amounts are calculated and stored as feature amounts for product addition determination. Similarly, a large number of captured images of products actually being removed from a shopping basket are collected in advance, and their feature amounts are calculated and stored as feature amounts for product removal determination. Note that captured images of products being added to a shopping basket include scenes in which the product (or a hand holding a product) appears from the periphery (edge) and scenes in which an empty hand (hand not holding a product) disappears from the periphery, and captured images of products being removed from a shopping basket include scenes in which an empty hand appears from the periphery and scenes in which a hand holding a product disappears from the periphery (FIGS. 9 to 11).

[0113] When recognizing the insertion of a product, the mobile terminal 60 calculates the feature amounts of the captured image acquired from the imaging unit. The mobile terminal 60 calculates the similarity between the captured image acquired from the imaging unit and the captured image when the product is actually added to the shopping basket by comparing the feature amounts of the captured image acquired from the imaging unit with the feature amounts for determining whether a product has been inserted, and calculates the similarity between the captured image acquired from the imaging unit and the captured image when the product is actually removed from the shopping basket by comparing the feature amounts of the captured image acquired from the imaging unit with the feature amounts for determining whether a product has been removed.

[0114] The mobile terminal 60 determines that a product has been added if the similarity between the captured image acquired from the imaging unit and the captured image of the product actually being added to the shopping basket is equal to or greater than a predetermined threshold (determines that a product has not been added if the similarity is less than the predetermined threshold).The mobile terminal 60 determines that a product has been removed if the similarity between the captured image acquired from the imaging unit and the captured image of the product actually being removed from the shopping basket is equal to or greater than a predetermined threshold (determines that a product has not been removed if the similarity is less than the predetermined threshold).

[0115] The mobile terminal 60 may output a determination result of three or more levels regarding the insertion of a commodity, instead of a determination result of two levels (commodity inserted, commodity not inserted). For example, the mobile terminal 60 may output a determination result of three levels regarding the insertion of a commodity (commodity inserted (high accuracy), commodity inserted (low accuracy), commodity not inserted). Specifically, the mobile terminal 60 may determine that a commodity has been inserted (high accuracy) when the similarity between the captured image acquired from the imaging unit and the captured image when the commodity is actually inserted into the shopping basket is equal to or greater than a second threshold that is greater than the first threshold, determine that a commodity has been inserted (low accuracy) when the similarity is equal to or greater than the first threshold but less than the second threshold, and determine that a commodity has not been inserted when the similarity is less than the first threshold.

[0116] Similarly, the mobile terminal 60 may output a determination result of three or more levels regarding product removal instead of a determination result of two levels (product removed, product not removed). For example, the mobile terminal 60 may output a determination result of three levels regarding product removal (product removed (high accuracy), product removed (low accuracy), product not removed). Specifically, the mobile terminal 60 may determine that product has been removed (high accuracy) when the similarity between the captured image acquired from the imaging unit and the captured image of the product actually removed from the shopping basket is equal to or greater than a second threshold that is greater than the first threshold, determine that product has been removed (low accuracy) when the similarity is equal to or greater than the first threshold and less than the second threshold, and determine that product has not been removed when the similarity is less than the first threshold.

[0117] When adopting a mode that outputs a three-level judgment result for product input (product input (high probability), product input (low probability), product input not input) (the same applies when adopting a mode that outputs a three-level judgment result for product input using a product input judgment model (described below)), unregistered input may be notified as follows. (1) If a product has been inserted (high accuracy) but has not been registered (if the product has been recognized as inserted (high accuracy) but has not been registered), the small screen SG20 in Figure 5(A) or the small screen SG22 in Figure 6(B) is displayed. (2) If a product has been inserted (low accuracy) and has not been registered (if the product has been recognized as inserted (low accuracy) but has not been registered), the small screen SG10 of FIG. 6(A) is displayed. (3) If no product has been introduced and no product has been registered, neither small screen SG20, small screen SG22 nor small screen SG10 is displayed.

[0118] When adopting a mode that outputs a three-level judgment result for product input (the same applies when adopting a mode that outputs a three-level judgment result for product input using a product input judgment model (described below)), non-registration input may be notified as follows. (1) If no product is inserted and registered (registered) (if the product is registered but the insertion of the product is not recognized), the small screen SG40 of FIG. 5(C) or the small screen SG42 of FIG. 6(E) is displayed. (2) In the case where a product has been inserted (low accuracy) and registered (registered) (when the product has been registered but the accuracy of the recognition that the product has been inserted is low), the small screen SG10 of FIG. 6(A) is displayed. (3) When a product is inserted (high probability) and registered (registered), none of the small screens SG40, SG42, and SG10 are displayed.

[0119] When adopting a mode that outputs a three-level judgment result for product removal (product removed (high probability), product removed (low probability), product not removed) (the same applies when adopting a mode that outputs a three-level judgment result for product removal using a product removal judgment model (described below)), cancellations that have not been removed may be notified as follows. (1) In the case where no product is taken out and cancellation (cancellation is present) (the product is canceled but the product is not recognized as being taken out), the small screen SG30 of FIG. 5(B) or the small screen SG32 of FIG. 6(D) is displayed. (2) In the case where a product has been taken out (low accuracy) and cancelled (cancelled) (the product has been cancelled but the accuracy of the recognition that the product has been taken out is low), the small screen SG11 of FIG. 6(C) is displayed. (3) When product removal is confirmed (high probability) and cancellation is confirmed (cancellation is confirmed), neither small screen SG30, small screen SG32 nor small screen SG11 is displayed.

[0120] When adopting a mode that outputs a three-level judgment result for product removal (the same applies when adopting a mode that outputs a three-level judgment result for product removal using a product removal judgment model (described below)), uncancelled removals may be notified as follows. (1) If a product has been removed (high accuracy) and not canceled (if a product has been removed (high accuracy) but not canceled), the small screen SG50 of Figure 5(D) or the small screen SG52 of Figure 6(F) is displayed. (2) If a product has been taken out (low accuracy) and has not been canceled (if a product has been recognized as being taken out (low accuracy) but has not been canceled), the small screen SG11 of FIG. 6C is displayed. (3) If no product has been taken out and the order has not been canceled, none of the small screens SG50, SG52, and SG11 is displayed.

[0121] Product insertion / removal recognition (product insertion / removal recognition function) may be realized using a trained model generated by machine learning. For example, a trained model for determining whether a product has been inserted or removed may be used. Hereinafter, the trained model for determining whether a product has been inserted is referred to as a "product insertion determination model," and the trained model for determining whether a product has been removed is referred to as a "product removal determination model."

[0122] The product insertion determination model and the product removal determination model may be trained models represented by using a neural network, for example. The product insertion determination model is a trained model that outputs, for example, the probability (likelihood) that a product has been inserted and the probability that a product has not been inserted when a captured image (or feature values ​​of the captured image) acquired from the imaging unit is input. The product removal determination model is a trained model that outputs, for example, the probability that a product has been removed and the probability that a product has not been removed when a captured image (or feature values ​​of the captured image) acquired from the imaging unit is input.

[0123] The product input judgment model (and the product removal judgment model) are generated by a learning model generation device. The learning model generation device may be a device within the sales system 1 (such as the headquarters server 10 or the cloud server 20) or a device outside the sales system 1 (such as another server or a personal computer).

[0124] The learning model generation device generates a product insertion determination model based on, for example, a large number of individual captured images (or the features of each captured image) when a product is added to a shopping basket and corresponding training data (trainer data indicating that each is an image with a product added), and a large number of individual captured images (or the features of each captured image) when a product is not added to the shopping basket and corresponding training data (trainer data indicating that each is an image with no product added).

[0125] The learning model generation device generates a product removal judgment model based on, for example, a large number of individual captured images (or feature values ​​of each captured image) when a product is removed from a shopping basket and corresponding training data (trainer data indicating that each is an image with a product removed), and a large number of individual captured images (or feature values ​​of each captured image) when a product is not removed from the shopping basket and corresponding training data (trainer data indicating that each is an image with a product not removed).

[0126] When recognizing the insertion of a product, etc., the mobile terminal 60 inputs the captured image (or the calculated features of the captured image) obtained from the imaging unit into the product insertion determination model (input layer of the product insertion determination model) and into the product removal determination model (input layer of the product removal determination model), thereby obtaining the probability that a product has been inserted and the probability that a product has not been inserted from the product insertion determination model (output layer of the product insertion determination model), and the probability that a product has been removed and the probability that a product has not been removed from the product removal determination model (output layer of the product removal determination model).

[0127] In a mode in which a two-stage determination result is output for product insertion, the mobile terminal 60 determines that a product has been inserted if the probability that a product has been inserted obtained from the product insertion determination model is equal to or greater than a predetermined probability (e.g., 50%), and determines that a product has not been inserted if the probability is less than the predetermined probability. In a mode in which a two-stage determination result is output for product removal, the mobile terminal 60 determines that a product has been removed if the probability that a product has been removed obtained from the product removal determination model is equal to or greater than a predetermined probability (e.g., 50%), and determines that a product has not been removed if the probability is less than the predetermined probability.

[0128] In a mode in which a three-level determination result for product insertion is output, the mobile terminal 60 determines that a product has been inserted (high accuracy) if the probability of product insertion obtained from the product insertion determination model is equal to or greater than a second probability (e.g., 50%) that is greater than a first probability (e.g., 20%), determines that a product has been inserted (low accuracy) if the probability is equal to or greater than the first probability and less than the second probability, and determines that a product has not been inserted if the probability is less than the first probability. In a mode in which a three-level determination result for product removal is output, the mobile terminal 60 determines that a product has been removed (high accuracy) if the probability of product removal obtained from the product removal determination model is equal to or greater than the second probability that is greater than the first probability, determines that a product has been removed (low accuracy) if the probability is equal to or greater than the first probability and less than the second probability, and determines that a product has not been removed if the probability is less than the first probability.

[0129] (Two types of product input / output recognition functions with different recognition accuracy) Sales system 1 is equipped with a product insertion / removal recognition function with standard accuracy for recognizing (determining) product insertion, etc. (product insertion, product removal), and a high-precision product insertion / removal recognition function with higher accuracy than the standard accuracy for recognizing product insertion, etc. In other words, although it has been explained that the determination result of product insertion, etc. may be output in two stages or in three or more stages, sales system 1 is equipped with two types of product insertion / removal recognition functions with different recognition accuracy (determination accuracy) for product insertion, etc. (a standard-precision product insertion / removal recognition function and a high-precision product insertion / removal recognition function).

[0130] Hereinafter, the recognition (determination) of product insertion, etc. using the standard-precision product insertion / removal recognition function will be referred to as "standard-precision product insertion / removal recognition," and the recognition (determination) of product insertion, etc. using the high-precision product insertion / removal recognition function will be referred to as "high-precision product insertion / removal recognition." The determination results of product insertion, etc. using standard-precision product insertion / removal recognition (and the same applies to high-precision product insertion / removal recognition) may be output in two stages, or in three or more stages.

[0131] Standard-accuracy product input / output recognition (standard-accuracy product input / output recognition function) may be realized without using a trained model, or may be realized using a trained model. The same applies to high-accuracy product input / output recognition (high-accuracy product input / output recognition function).

[0132] High-precision product insertion / removal recognition has a higher accuracy of recognition results (judgment results) than standard-precision product insertion / removal recognition, but it takes time to output the recognition results. Standard-precision product insertion / removal recognition outputs recognition results in a shorter time than high-precision product insertion / removal recognition, but the accuracy of the recognition results is lower. In other words, standard-precision product insertion / removal recognition is inferior in accuracy (reliability) to high-precision product insertion / removal recognition, but is superior in speed (response) to high-precision product insertion / removal recognition. On the other hand, high-precision product insertion / removal recognition is inferior in speed to standard-precision product insertion / removal recognition, but is superior in accuracy to standard-precision product insertion / removal recognition.

[0133] The amount of information in the captured image used for high-precision product insertion / removal recognition is greater than the amount of information in the captured image used for standard-precision product insertion / removal recognition. For example, the number of captured images used for the former determination (e.g., the number of frames in the case of a video) may be greater than the number of captured images used for the latter determination. Alternatively or in addition to the above, the number of colors (the number of possible pixel values) in the captured image used for the former determination may be greater than the number of colors in the captured image used for the latter determination (e.g., the former may be color and the latter may be grayscale). Also, alternatively or in addition to the above, the size (volume) of the captured image used for the former determination may be greater than the size of the captured image used for the latter determination. Details will be described later.

[0134] 7 is an explanatory diagram illustrating the executing entity and execution timing of two types of product insertion / removal recognition (standard-accuracy product insertion / removal recognition and high-accuracy product insertion / removal recognition). As explained above, both the mobile terminal 60 and the cloud server 20 have the product insertion / removal recognition function, but the mobile terminal 60 executes standard-accuracy product insertion / removal recognition and the cloud server 20 executes high-accuracy product insertion / removal recognition.

[0135] For example, as shown in FIG. 7(A), the mobile terminal 60 performs standard-accuracy product insertion / removal recognition during the period from the display of the initial registration screen to the payment instruction (operation of the payment instruction button BT631) (between steps S1 and S7 in FIG. 3). The cloud server 20 performs high-accuracy product insertion / removal recognition during the period from receiving a request for transaction information from the settlement device 50 to sending the transaction information to the settlement device 50 (between steps S11 and S12 in FIG. 3). After completing high-accuracy product insertion / removal recognition, the cloud server 20 sends the transaction information to the settlement device 50. In other words, the settlement device 50 displays the product list screen after the cloud server 20 completes high-accuracy product insertion / removal recognition.

[0136] The mobile terminal 60 acquires captured images from the imaging unit each time (for example, 15 images per second), but for example, performs preprocessing (processing to reduce the amount of information, such as extracting, grayscaling, or compressing some captured images) on the captured images acquired from the imaging unit, and recognizes the insertion of merchandise, etc. using the preprocessed captured images (standard-accuracy merchandise insertion / removal recognition). In other words, the mobile terminal 60 reduces the amount of information in the captured images while displaying the registration screen (scan registration screen, button registration screen), and recognizes (determines) the insertion of merchandise, etc. with standard accuracy in real time using the captured images with the reduced amount of information.

[0137] The mobile device 60 transmits to the cloud server 20 information (original captured images without preprocessing) necessary for the cloud server 20 to execute high-precision product insertion / removal recognition at least (at the latest) before the cloud server 20 starts executing high-precision product insertion / removal recognition. For example, if the period during which the cloud server 20 executes high-precision product insertion / removal recognition is the period after the checkout operation on the mobile device 60 (the period in FIG. 7(A) or 7(B)), the mobile device 60 may transmit the captured image to the cloud server 20 each time it is captured, or may transmit the captured images within that period to the cloud server 20 at predetermined time intervals (e.g., every 5 seconds), or may transmit all the captured images to the cloud server 20 after the checkout operation. Furthermore, if the period during which the cloud server 20 executes high-precision product insertion / removal recognition is the period before the checkout operation on the mobile device 60 (the period in FIG. 7(C)), the mobile device 60 may transmit the captured image to the cloud server 20 each time it is captured, or may transmit the captured images within that period to the cloud server 20 at predetermined time intervals (e.g., every 5 seconds).

[0138] Cloud server 20 stores (accumulates) information (original captured images) transmitted from mobile terminal 60. Cloud server 20 recognizes the insertion of merchandise, etc. (high-precision merchandise insertion / removal recognition) using information stored in itself during the period from when transaction information is requested by settlement device 50 until when the cloud server 20 transmits the transaction information to settlement device 50. In other words, after registration is complete, cloud server 20 uses the original captured images to recognize (determine) the insertion of merchandise, etc., with high precision after the fact.

[0139] Alternatively, as shown in FIG. 7(B), cloud server 20 may perform high-precision product insertion / removal recognition during the period from when a checkout instruction is made on mobile terminal 60 (operation of checkout instruction button BT631) to when a settlement code is displayed on mobile terminal 60. In step S8 of FIG. 3, it was explained that mobile terminal 60 may notify cloud server 20 that the checkout instruction button BT631 has been operated, but cloud server 20 performs high-precision product insertion / removal recognition after receiving the above notification from mobile terminal 60. After receiving the above notification from mobile terminal 60, cloud server 20 may generate a settlement code itself and send it to mobile terminal 60, or it may not generate a settlement code itself, and instead mobile terminal 60 may generate and control the timing of displaying the settlement code using a generation instruction. In the former case, the cloud server 20 transmits the settlement code to the mobile terminal 60 after the completion of the high-precision product insertion / removal recognition (the settlement code may be generated before the completion of the high-precision product insertion / removal recognition, or may be generated after the completion of the high-precision product insertion / removal recognition, but the settlement code is transmitted to the mobile terminal 60 after the completion of the high-precision product insertion / removal recognition), and in the latter case, transmits a generation instruction to the mobile terminal 60 after the completion of the high-precision product insertion / removal recognition. In other words, in either the former case or the latter case, the mobile terminal 60 displays the settlement code after the cloud server 20 completes the high-precision product insertion / removal recognition.

[0140] Alternatively, as shown in FIG. 7(C), the cloud server 20 may execute high-precision product insertion / removal recognition during the period when the mobile terminal 60 executes standard-precision product insertion / removal recognition. Because the cloud server 20 executes high-precision product insertion / removal recognition ahead of time without waiting for product registration to be completed (operation of the checkout instruction button BT631), there is no wait time when the cloud server 20 executes high-precision product insertion / removal recognition after product registration is complete, and processing progresses faster after product registration is complete. In reality, however, because the cloud server 20 executes high-precision product insertion / removal recognition after the mobile terminal 60 executes standard-precision product insertion / removal recognition (due to a time lag), it appears that the cloud server 20 is executing high-precision product insertion / removal recognition immediately after product registration is complete, but the process as a whole is almost complete (i.e., the wait time is significantly reduced).

[0141] (Standard accuracy fraud monitoring, high accuracy fraud monitoring) The mobile terminal 60 monitors for fraudulent behavior by customers (e.g., unregistered input, unregistered input, uncancelled withdrawal, uncancelled withdrawal) based on the status of product registration, product cancellation, etc., and the status of product insertion, etc. (recognition results from standard-accuracy product insertion / removal recognition). Monitoring of fraudulent behavior based on standard-accuracy product insertion / removal recognition is referred to as "standard-accuracy fraud monitoring" or "standard-accuracy fraud recognition." The mobile terminal 60 performs standard-accuracy fraud monitoring in real time during the period when standard-accuracy product insertion / removal recognition is performed (the period in Figures 7(A) to 7(C)), and if it determines that fraud has occurred, it notifies the customer as shown in Figures 5 to 6.

[0142] Cloud server 20 monitors for fraudulent behavior by customers (e.g., unregistered insertion, unregistered insertion, uncancelled removal, uncancelled removal) based on the status of product registration and product cancellation on mobile terminal 60 (operation information acquired from mobile terminal 60) and the status of product insertion on mobile terminal 60 (recognition results from high-precision product insertion / removal recognition). Monitoring of fraudulent behavior based on high-precision product insertion / removal recognition is referred to as "high-precision fraud monitoring" or "high-precision fraud recognition." Cloud server 20 performs high-precision fraud monitoring during the period when high-precision product insertion / removal recognition is performed (any of the periods in Figures 7(A) to 7(C)).

[0143] The mobile device 60 sends operation information to the cloud server 20 at least (at the latest) before the cloud server 20 starts performing high-precision fraud monitoring. For example, if the period during which the cloud server 20 performs high-precision fraud monitoring is a period after a checkout operation on the mobile device 60 (e.g., the same period as the high-precision product insertion / removal recognition in FIG. 7(A) or the same period as the high-precision product insertion / removal recognition in FIG. 7(B)), the mobile device 60 may send operation information to the cloud server 20 each time an operation is performed, or may send operation information for that period to the cloud server 20 every predetermined time (e.g., every 5 seconds), or may send all operation information to the cloud server 20 after the checkout operation. Furthermore, if the period during which the cloud server 20 performs high-precision fraud monitoring is a period before a checkout operation on the mobile device 60 (e.g., the same period as the high-precision product insertion / removal recognition in FIG. 7(C)), the mobile device 60 may send operation information to the cloud server 20 each time an operation is performed, or may send operation information for that period to the cloud server 20 every predetermined time (e.g., every 5 seconds).

[0144] (Storage of recognition results such as product input) The recognition results of product insertion, etc. by the mobile terminal 60 (recognition results by standard-accuracy product insertion / removal recognition) and the recognition results of product insertion, etc. by the cloud server 20 (recognition results by high-accuracy product insertion / removal recognition) are stored in the sales system 1 (for example, the cloud server 20). For example, the mobile terminal 60 transmits the recognition results of product insertion, etc. by the mobile terminal 60 to the cloud server 20, and the cloud server 20 stores the recognition results of product insertion, etc. by the mobile terminal 60 and the recognition results of product insertion, etc. by the cloud server 20.

[0145] The recognition result of merchandise insertion, etc. by the mobile terminal 60 and the recognition result of merchandise insertion, etc. by the cloud server 20 for a certain object (the same object) are stored in a comparable linked manner. That is, the recognition result of merchandise insertion, etc. by the mobile terminal 60 based on one or more captured images (captured images after preprocessing) during a certain period (a period from XX:00:XX:△ second to XX:00:XX:▽ second) and the recognition result of merchandise insertion, etc. by the cloud server 20 based on one or more captured images (original captured images) during the same period are stored in a comparable linked manner. The linking may be performed based on the captured images (capture times) that served as the basis for the recognition of each merchandise insertion, etc., or based on the elapsed time or order from the start of the transaction.

[0146] (Memory of the results of the judgment on fraud, etc.) The results of the determination of fraud, etc. by the mobile terminal 60 (determination results by standard-accuracy fraud monitoring, monitoring results by standard-accuracy fraud monitoring) and the results of the determination of fraud, etc. by the cloud server 20 (determination results by high-accuracy fraud monitoring, monitoring results by high-accuracy fraud monitoring) are stored in the sales system 1 (for example, cloud server 20). For example, the mobile terminal 60 transmits the results of the determination of fraud, etc. by the mobile terminal 60 to the cloud server 20, and the cloud server 20 stores the results of the determination of fraud, etc. by the mobile terminal 60 and the results of the determination of fraud, etc. by the cloud server 20. As with the recognition results of product insertion, etc., the results of the determination of fraud, etc. by the mobile terminal 60 and the results of the determination of fraud, etc. by the cloud server 20 for a certain target are stored in a linked manner so as to be comparable.

[0147] (When the recognition of product launch, etc. is overturned, or when the judgment of fraud, etc. is overturned) There are cases where the recognition result of product insertion, etc. by the mobile terminal 60 (recognition result by standard-accuracy product insertion / removal recognition) differs from the recognition result of product insertion, etc. by the cloud server 20 (recognition result by high-accuracy product insertion / removal recognition). When the recognition result of product insertion, etc. by the mobile terminal 60 differs from the recognition result of product insertion, etc. by the cloud server 20, the judgment result of fraud, etc. by the mobile terminal 60 (judgment result by standard-accuracy fraud monitoring) differs from the judgment result of fraud, etc. by the cloud server 20 (judgment result by high-accuracy fraud monitoring).

[0148] When the recognition result of the product insertion, etc. by the mobile terminal 60 differs from the recognition result of the product insertion, etc. by the cloud server 20, the recognition result of the product insertion, etc. by the cloud server 20 takes precedence. In other words, when the recognition result of the product insertion, etc. by the mobile terminal 60 differs from the recognition result of the product insertion, etc. by the cloud server 20, the recognition of the product insertion, etc. by the mobile terminal 60 is overturned in the cloud server 20. When the recognition of the product insertion, etc. by the mobile terminal 60 is overturned in the cloud server 20, the determination of fraud, etc. by the mobile terminal 60 is overturned in the cloud server 20. In other words, when the recognition result of the product insertion, etc. by the mobile terminal 60 differs from the recognition result of the product insertion, etc. by the cloud server 20, the latter is given priority, and as a result, the determination of fraud, etc. by the mobile terminal 60 is overturned in the cloud server 20. For example, the determination of fraud, etc. is overturned as shown in (a) to (h) below.

[0149] (a) Applies to unregistered input → Does not apply to unregistered input In a situation where no product is registered (or no increase in quantity), if the mobile terminal 60 (standard accuracy fraud monitoring) recognizes that a product has been inserted and determines that this corresponds to an unregistered insertion, but the cloud server 20 (high accuracy fraud monitoring) recognizes that no product has been inserted, it is determined that this does not correspond to an unregistered insertion. In other words, in a situation where no product is registered (or no increase in quantity), if the recognition of a product being inserted on the mobile terminal 60 is changed to a recognition of no product being inserted on the cloud server 20, the determination that this corresponds to an unregistered insertion is reversed to a determination that this does not correspond to an unregistered insertion.

[0150] In the case of (a) above, the cloud server 20 stores "no product registered" based on the operation information, stores "product inserted" as the recognition result by standard-accuracy product insertion / removal recognition, stores "no product inserted" as the recognition result by high-accuracy product insertion / removal recognition, stores "unregistered insertion" as the judgment result by standard-accuracy fraud monitoring, and stores "not corresponding to fraud, etc." as the judgment result by high-accuracy fraud monitoring.

[0151] (b) Not applicable to unregistered input → Applicable to unregistered input In a situation where no product is registered (or the quantity is not increased), if the mobile terminal 60 (standard accuracy fraud monitoring) recognizes that no product has been inserted and determines that this does not correspond to an unregistered insertion, but the cloud server 20 (high accuracy fraud monitoring) recognizes that a product has been inserted, then this is determined to correspond to an unregistered insertion. In other words, in a situation where no product is registered (or the quantity is not increased), if the recognition of no product insertion on the mobile terminal 60 is changed to a recognition of product insertion on the cloud server 20, then the determination that this does not correspond to an unregistered insertion is reversed to a determination that this does correspond to an unregistered insertion.

[0152] In the case of (b) above, the cloud server 20 stores "no product registered" based on the operation information, stores "no product inserted" as the recognition result by standard-accuracy product insertion / removal recognition, stores "product inserted" as the recognition result by high-accuracy product insertion / removal recognition, stores "not applicable to fraud, etc." as the judgment result by standard-accuracy fraud monitoring, and stores "apparently unregistered insertion" as the judgment result by high-accuracy fraud monitoring.

[0153] (c) Applies to unregistered items → Does not apply to unregistered items In a situation where a product has been registered (or the quantity has increased), if the mobile terminal 60 (standard accuracy fraud monitoring) recognizes that no product has been inserted and determines that the product has not been registered, but the cloud server 20 (high accuracy fraud monitoring) recognizes that the product has been inserted, it is determined that the product has not been registered. In other words, in a situation where a product has been registered (or the quantity has increased), if the recognition of no product insertion on the mobile terminal 60 is changed to that of product insertion on the cloud server 20, the determination that the product has not been registered is reversed to a determination that the product has not been registered.

[0154] In the case of (c) above, the cloud server 20 stores "product registered" based on the operation information, stores "no product inserted" as the recognition result by standard-accuracy product insertion / removal recognition, stores "product inserted" as the recognition result by high-accuracy product insertion / removal recognition, stores "not registered inserted" as the judgment result by standard-accuracy fraud monitoring, and stores "not corresponding to fraud, etc." as the judgment result by high-accuracy fraud monitoring.

[0155] (d) Not applicable to not being registered → Applicable to not being registered In a situation where a product has been registered (or the quantity has increased), if the mobile terminal 60 (standard accuracy fraud monitoring) recognizes that a product has been inserted and determines that this does not correspond to a case where the product has not been inserted, but the cloud server 20 (high accuracy fraud monitoring) recognizes that the product has not been inserted, then this will be determined to correspond to a case where the product has not been inserted. In other words, in a situation where a product has been registered (or the quantity has increased), if the recognition of the product having been inserted on the mobile terminal 60 is changed to the cloud server 20 that the product has not been inserted, then the determination that this does not correspond to a case where the product has not been inserted is reversed to a determination that this corresponds to a case where the product has not been inserted.

[0156] In the case of (d) above, the cloud server 20 stores "product registered" based on the operation information, stores "product inserted" as the recognition result by standard-accuracy product insertion / removal recognition, stores "product not inserted" as the recognition result by high-accuracy product insertion / removal recognition, stores "not corresponding to fraud, etc." as the judgment result by standard-accuracy fraud monitoring, and stores "corresponding to unregistered insertion" as the judgment result by high-accuracy fraud monitoring.

[0157] (e) Applies to uncancelled withdrawal → Does not apply to uncancelled withdrawal In a situation where no product has been cancelled (or no quantity has been reduced), if the mobile terminal 60 (standard-accuracy fraud monitoring) recognizes that a product has been taken out and determines that this corresponds to an uncancelled takeout, but the cloud server 20 (high-accuracy fraud monitoring) recognizes that no product has been taken out, it will be determined that this does not correspond to an uncancelled takeout. In other words, in a situation where no product has been cancelled (or no quantity has been reduced), if the recognition on the mobile terminal 60 that a product has been taken out is changed to a recognition on the cloud server 20 that a product has not been taken out, the determination that this corresponds to an uncancelled takeout will be reversed to a determination that this does not correspond to an uncancelled takeout.

[0158] In the case of (e) above, the cloud server 20 stores "no product cancellation" based on the operation information, stores "product removal" as the recognition result by standard-accuracy product insertion / removal recognition, stores "no product removal" as the recognition result by high-accuracy product insertion / removal recognition, stores "corresponds to uncancelled removal" as the judgment result by standard-accuracy fraud monitoring, and stores "not corresponding to fraud, etc." as the judgment result by high-accuracy fraud monitoring.

[0159] (f) Not applicable to uncancelled withdrawal → Applicable to uncancelled withdrawal In a situation where no product has been cancelled (or no quantity has been reduced), if the mobile terminal 60 (standard-accuracy fraud monitoring) recognizes that no product has been taken out and determines that this does not correspond to an uncancelled takeout, but the cloud server 20 (high-accuracy fraud monitoring) recognizes that a product has been taken out, then this is determined to correspond to an uncancelled takeout. In other words, in a situation where no product has been cancelled (or no quantity has been reduced), if the recognition on the mobile terminal 60 that no product has been taken out is changed to a recognition on the cloud server 20 that a product has been taken out, then the determination that this does not correspond to an uncancelled takeout is reversed to a determination that this does correspond to an uncancelled takeout.

[0160] In the case of (f) above, the cloud server 20 stores "no product cancellation" based on the operation information, stores "no product removal" as the recognition result by standard-accuracy product insertion / removal recognition, stores "product removal" as the recognition result by high-accuracy product insertion / removal recognition, stores "not applicable to fraud, etc." as the judgment result by standard-accuracy fraud monitoring, and stores "apparently applicable to uncancelled removal" as the judgment result by high-accuracy fraud monitoring.

[0161] (g) Applies to cancellation and non-retrieval → Does not apply to cancellation and non-retrieval In a situation where a product has been cancelled (or the quantity has been reduced), if the mobile terminal 60 (standard-accuracy fraud monitoring) recognizes that the product has not been taken out and determines that it corresponds to a cancellation that has not been taken out, but the cloud server 20 (high-accuracy fraud monitoring) recognizes that the product has been taken out, it is determined that it does not correspond to a cancellation that has not been taken out. In other words, in a situation where a product has been cancelled (or the quantity has been reduced), if the recognition on the mobile terminal 60 that the product has not been taken out is changed to a recognition on the cloud server 20 that the product has been taken out, the determination that it corresponds to a cancellation that has not been taken out is reversed to a determination that it does not correspond to a cancellation that has not been taken out.

[0162] In the case of (g) above, the cloud server 20 stores "product canceled" based on the operation information, stores "product not removed" as the recognition result by standard-accuracy product insertion / removal recognition, stores "product removed" as the recognition result by high-accuracy product insertion / removal recognition, stores "corresponding to cancellation but not removed" as the judgment result by standard-accuracy fraud monitoring, and stores "not corresponding to fraud, etc." as the judgment result by high-accuracy fraud monitoring.

[0163] (h) Not applicable to cancellation and not retrieved → Applicable to cancellation and not retrieved In a situation where a product has been cancelled (or the quantity has been reduced), if the mobile terminal 60 (standard-accuracy fraud monitoring) recognizes that the product has been taken out and determines that this does not correspond to a case of cancellation but not being taken out, but the cloud server 20 (high-accuracy fraud monitoring) recognizes that the product has not been taken out, then this will be determined to correspond to a case of cancellation but not being taken out. In other words, in a situation where a product has been cancelled (or the quantity has been reduced), if the recognition on the mobile terminal 60 that the product has been taken out is overturned to a judgment that the product has not been taken out by the cloud server 20, then the determination that this does not correspond to a case of cancellation but not being taken out will be overturned to a determination that this corresponds to a case of cancellation but not being taken out.

[0164] In the case of (h) above, the cloud server 20 stores "product canceled" based on the operation information, stores "product removed" as the recognition result by standard-accuracy product insertion / removal recognition, stores "product not removed" as the recognition result by high-accuracy product insertion / removal recognition, stores "not corresponding to fraud, etc." as the judgment result by standard-accuracy fraud monitoring, and stores "corresponding to cancellation but not removed" as the judgment result by high-accuracy fraud monitoring.

[0165] The customer may be notified of information based on the results of the fraudulent activity, etc., judged by the mobile terminal 60 (the results of the standard-accuracy fraud monitoring) or the results of the fraudulent activity, etc., judged by the cloud server 20 (the results of the high-accuracy fraud monitoring).

[0166] 8 is an example of a display on the settlement device 50. For example, when the mobile terminal 60 registers a loaf of bread, recognizes the product insertion, registers a pear, recognizes the product insertion, and completes the registration (when the mobile terminal 60 determines that no fraud has occurred), and when the cloud server 20 also determines that no fraud has occurred, the settlement device 50 reads the settlement code and then displays a product list screen such as that shown in FIG. 8(A).

[0167] The product list screen in Figure 8(A) displays the registration information for the first product (bread), the registration information for the second product (pear), the total price, etc. When the checkout button (bottom right button) is pressed, the checkout device 50 displays, for example, a payment method selection screen (not shown) as the next screen, which allows the user to select a payment method (cash, credit card, etc.).

[0168] In the scene of Figure 8 (A), the cloud server 20 stores, for example, "scan registration confirmed" based on operation information as information regarding fraud, etc. for the first item of product (bread), "product inserted" as the recognition result by standard-accuracy product insertion / removal recognition, "product inserted" as the recognition result by high-accuracy product insertion / removal recognition, "not applicable to fraud, etc." as the judgment result by standard-accuracy fraud monitoring, and "not applicable to fraud, etc." as the judgment result by high-accuracy fraud monitoring, and stores, for example, "button registration confirmed" based on operation information as information regarding fraud, etc. for the second item of product (pear), "product inserted" as the recognition result by standard-accuracy product insertion / removal recognition, "product inserted" as the recognition result by high-accuracy product insertion / removal recognition, "not applicable to fraud, etc." as the judgment result by high-accuracy fraud monitoring. The settlement device 50 displays the settlement list screen of FIG. 8(A) based on the registration information stored in the cloud server 20 and the above information.

[0169] As described above, the settlement device 50 displays the registration information of the first item (loaf of bread) on the product list screen of FIG. 8(A). However, in addition to the registration information (i.e., for the item), it may also display information indicating the result of a determination made by standard-accuracy fraud monitoring that "does not constitute fraud" (e.g., "terminal OK") or information indicating the result of a determination made by high-accuracy fraud monitoring that "does not constitute fraud" (e.g., "server OK"). Furthermore, instead of or in addition to the above, the settlement device 50 may also display information indicating that the result of a determination made by the mobile terminal 60 that "does not constitute fraud" has been maintained in the cloud server 20 (e.g., "does not constitute fraud (maintained)"). The same applies to the second item (pear).

[0170] The cloud server 20 stores information that a product requiring age verification, such as beer (age verification product), is an age verification product, and stores information that a product requiring explanation from a pharmacist, etc. (medicinal product requiring explanation) is a pharmaceutical product requiring explanation. Based on the above information, the settlement device 50 may display information that the product is an age verification product (for example, "age verification product") in addition to the registration information of the age verification product, and may display information that the pharmaceutical product is a pharmaceutical product requiring explanation (for example, "pharmaceutical product requiring explanation") in addition to the registration information of the pharmaceutical product requiring explanation, for pharmaceutical products requiring explanation.

[0171] The settlement device 50 may display information indicating the results of the determination of fraud, etc. using standard-precision fraud monitoring or the results of the determination of fraud, etc. using high-precision fraud monitoring (for example, the above-mentioned "Terminal OK", the above-mentioned "Server OK", the below-mentioned "Terminal NG (unregistered input)", the below-mentioned "Server NG (unregistered input)", etc.), information that the results of the determination of fraud, etc. using standard-precision fraud monitoring have been maintained in high-precision fraud monitoring, or information that the results of the determination of fraud, etc. using standard-precision fraud monitoring have been overturned in high-precision fraud monitoring (described below), in a display format different from that used for "age verification products" or "medicines requiring explanation".

[0172] In another situation, for example, when the mobile terminal 60 recognizes the product insertion, registers a pear, recognizes the product insertion and completes the registration (when the mobile terminal 60 recognizes the product insertion in the scene before the pear and determines that the scene before the pear corresponds to an unregistered insertion), and the cloud server 20 also recognizes the product insertion in the scene before the pear (when the cloud server 20 also determines that the scene before the pear corresponds to an unregistered insertion), the settlement device 50, after reading the settlement code, displays a product list screen as shown in Figure 8 (B) and displays a small screen SG100 in front of the product list screen.

[0173] The small screen SG100 displays a message indicating that a store clerk is being called (in the illustrated example, "Calling a store clerk, please wait a moment"). In other words, since both the mobile terminal 60 and the cloud server 20 have determined that this corresponds to an unregistered input (registration forgotten (quantity increase forgotten)), the settlement device 50 pops up the small screen SG100 to notify the customer that a store clerk will be called. The settlement device 50 may output audio in addition to displaying the small screen SG100. In other words, the settlement device 50 may notify the customer that a store clerk will be called by both display and audio.

[0174] The small screen SG100 is erased based on, for example, an operation by a store clerk. The product list screen on the back displays a message that one product item is unregistered ("Unregistered Product" in the illustrated example), and an image button (button marked with the symbol B in the figure) is arranged. When the image button is operated (basically by a store clerk), a captured image of the unregistered product is displayed (for example, a captured image of the scene in which the unregistered product is being inserted is obtained from the cloud server 20 and displayed). When a purchase is made, the checkout device 50 registers the unregistered product (for example, a loaf of bread) based on, for example, an operation by the store clerk (it is not necessary to purchase the product).

[0175] When the small screen SG100 is displayed in the foreground, operation of the image button on the back or the checkout button (bottom right button) is not accepted. This is also true for the scenes in Figures 8(C) to 8(F). Also, in the scenes in Figures 8(B) and 8(D), even after the small screen in the foreground is cleared, operation of the checkout button is not accepted until a store clerk takes action (for example, until the "unregistered product" is resolved by registering the product). The dashed checkout button in Figures 8(B) and 8(D) indicates that operation is not accepted. In other words, in the case of a transaction that requires the intervention of a store clerk, such as a transaction involving an "unregistered product," the settlement process is allowed to proceed (for example, the payment method selection screen is displayed) after the store clerk's intervention (including simple confirmation) has occurred.

[0176] In the scene of Figure 8 (B), the cloud server 20 stores, for example, "product not registered" based on operation information as information regarding fraud, etc. for the first item of product (unregistered product), "product inserted" as the recognition result by standard-accuracy product insertion / removal recognition, "product inserted" as the recognition result by high-accuracy product insertion / removal recognition, "corresponding to unregistered insertion" as the judgment result by standard-accuracy fraud monitoring, and "corresponding to unregistered insertion" as the judgment result by high-accuracy fraud monitoring, and as information regarding fraud, etc. for the second item of product (pear), for example, "button registered" based on operation information is stored, "product inserted" as the recognition result by standard-accuracy product insertion / removal recognition, "product inserted" as the recognition result by high-accuracy product insertion / removal recognition, "not corresponding to fraud, etc." as the judgment result by standard-accuracy fraud monitoring, and "not corresponding to fraud, etc." as the judgment result by high-accuracy fraud monitoring. The settlement device 50 displays the settlement list screen and small screen SG100 in FIG. 8(B) based on the registration information stored in the cloud server 20 and the above information.

[0177] As described above, the settlement device 50 displays "Unregistered Product" and an image button on the product list screen of Figure 8 (B) (product list screen on the back) to indicate that one product is unregistered, but in addition to this information, it may also display information indicating the result of a standard-precision fraud monitoring determination that "corresponds to an unregistered input" (for example, "Terminal NG (unregistered input)") or information indicating the result of a high-precision fraud monitoring determination that "corresponds to an unregistered input" (for example, "Server NG (unregistered input)").Furthermore, instead of or in addition to the above, the settlement device 50 may also display information indicating that the result of a fraud or other determination by the mobile terminal 60 that "corresponds to an unregistered input" has been maintained on the cloud server 20 (for example, "Unregistered Input (Maintained)").

[0178] In the scene of Figure 8(B), the mobile terminal 60 determines that the input corresponds to an unregistered input, and displays the small screen SG20 of Figure 5(A). However, if a customer who sees the small screen registers a product (e.g., a loaf of bread that they forgot to register) (e.g., registers it within a predetermined time after the small screen is cleared), the cloud server 20 stores, as in the case of Figure 8(A), information regarding fraud, etc., for the first product (loaf of bread), such as "scan registration confirmed" based on the operation information, "product inserted" as the recognition result by standard-accuracy product input / removal recognition, "product inserted" as the recognition result by high-accuracy product input / removal recognition, "not corresponding to fraud, etc." as the judgment result by standard-accuracy fraud monitoring, and "not corresponding to fraud, etc." as the judgment result by high-accuracy fraud monitoring. Therefore, the settlement device 50 displays a product list screen such as that shown in Figure 8(A).

[0179] In another case, for example, when the mobile terminal 60 recognizes the product insertion, registers a pear, and recognizes the product insertion and completes the registration (when the mobile terminal 60 recognizes the product insertion in the scene before the pear and determines that the scene before the pear corresponds to an unregistered insertion), but the cloud server 20 does not recognize the product insertion in the scene before the pear (when the cloud server 20 determines that the scene before the pear does not correspond to an unregistered insertion), the settlement device 50, after reading the settlement code, displays a product list screen as shown in Figure 8(C) and displays a small screen SG101 in front of the product list screen. Figure 8(C) is an example of a display when the determination by the mobile terminal 60 that the product is an unregistered insertion is overturned by the cloud server 20 that the product is not an unregistered insertion.

[0180] The small screen SG101 displays a message indicating that confirmation by a store clerk is no longer necessary (in the illustrated example, "Staff confirmation is no longer necessary"). In other words, the mobile terminal 60 determined that confirmation by a store clerk was necessary (corresponding to an unregistered deposit), but the cloud server 20 determined that confirmation by a store clerk was no longer necessary (not corresponding to an unregistered deposit). Therefore, the settlement device 50 pops up the small screen SG101 to notify the customer that confirmation by a store clerk is no longer necessary. Note that the settlement device 50 may output audio in addition to displaying the small screen SG101. In other words, the settlement device 50 may notify the customer that calling a store clerk is no longer necessary by visual and audio notification.

[0181] The small screen SG101 is erased after a predetermined time (for example, 3 seconds) has elapsed. The product list screen on the back displays the registered information for pear as the first item. Note that although the product list screen in FIG. 8(C) displays the registered information for pear as the first item, it is also possible to leave the first item blank and display the registered information for pear as the second item.

[0182] In the scene of Figure 8(C), the mobile terminal 60 determines that the product has been inserted without registration, and therefore displays the small screen SG20 of Figure 5(A), but since no product has actually been inserted (the recognition result of the high-precision product insertion / removal recognition is "no product inserted"), the customer who sees the small screen does not register the product.

[0183] In the scene of Figure 8 (C), the cloud server 20 stores, for example, "product not registered" based on operation information as information regarding fraud, etc. for the first item of product (unregistered product), "product inserted" as the recognition result by standard-accuracy product insertion / removal recognition, "product not inserted" as the recognition result by high-accuracy product insertion / removal recognition, "corresponds to unregistered insertion" as the judgment result by standard-accuracy fraud monitoring, and "does not correspond to fraud, etc." as the judgment result by high-accuracy fraud monitoring, and stores, for example, "button registered" based on operation information as information regarding fraud, etc. for the second item of product (pear), "product inserted" as the recognition result by standard-accuracy product insertion / removal recognition, "product inserted" as the recognition result by high-accuracy product insertion / removal recognition, "does not correspond to fraud, etc." as the judgment result by high-accuracy fraud monitoring. The settlement device 50 displays the settlement list screen and small screen SG101 of FIG. 8(C) based on the registration information stored in the cloud server 20 and the above information.

[0184] As described above, the settlement device 50 displays the registration information for the pear as one item on the product list screen (product list screen on the back) of FIG. 8(C) (i.e., no information is displayed in front of (above) the pear), but the information in front of the pear may display information indicating the result of the standard-precision fraud monitoring determination that "corresponds to unregistered input" (e.g., "terminal NG (unregistered input)") or information indicating the result of the high-precision fraud monitoring determination that "does not correspond to fraud, etc." (e.g., "server OK"). Furthermore, instead of or in addition to the above, the settlement device 50 may display information indicating that the result of the fraud, etc. determination by the mobile terminal 60 that "corresponds to unregistered input" has been changed to "does not correspond to fraud, etc." on the cloud server 20 (e.g., "terminal NG (unregistered input) → server OK").

[0185] In another case, for example, when the mobile terminal 60 registers a pear, recognizes the product insertion, and completes the registration (when the mobile terminal 60 determines that there is no fraud, etc.), and the cloud server 20 recognizes the product insertion in the scene before the pear (that is, when the cloud server 20 determines that the scene before the pear corresponds to an unregistered insertion), the settlement device 50, after reading the settlement code, displays a product list screen as shown in Fig. 8(D) and displays a small screen SG102 in front of the product list screen. Fig. 8(D) is an example of a display when the determination in the mobile terminal 60 that the product is not an unregistered insertion is overturned by the cloud server 20 that the product is an unregistered insertion.

[0186] The small screen SG102 displays a message indicating that confirmation by a store clerk is required (in the illustrated example, "Staff confirmation is required (staff member being called)"). In other words, the mobile terminal 60 determined that confirmation by a store clerk was not required (not corresponding to an unregistered deposit), but the cloud server 20 determined that confirmation by a store clerk was required (corresponding to an unregistered deposit). Therefore, the settlement device 50 pops up the small screen SG102 to notify the customer that confirmation by a store clerk is required. Note that the settlement device 50 may output audio in addition to displaying the small screen SG102. In other words, the settlement device 50 may notify the customer that confirmation by a store clerk is required by both visual and audio means.

[0187] The small screen SG102 is erased, for example, based on an operation by a store clerk. The product list screen on the back displays a message that one product item is unregistered ("Unregistered Product" in the illustrated example), and an image button (button marked with the symbol B in the figure) is provided. When the image button is operated (basically by a store clerk), a captured image of the unregistered product is displayed. When a purchase is made, the checkout device 50 registers the unregistered product (for example, a loaf of bread) based on an operation by a store clerk, for example (it is not necessary to purchase the product).

[0188] In the scene of FIG. 8(D), the mobile terminal 60 does not determine that the input corresponds to an unregistered input, and therefore does not display the small screen SG20 of FIG. 5(A) and the like.

[0189] In the scene of Figure 8 (D), the cloud server 20 stores, for example, "product not registered" based on operation information as information regarding fraud, etc. for the first item of product (unregistered product), "product not inserted" as the recognition result by standard-accuracy product insertion / removal recognition, "product inserted" as the recognition result by high-accuracy product insertion / removal recognition, "not corresponding to fraud, etc." as the judgment result by standard-accuracy fraud monitoring, and "corresponding to unregistered insertion" as the judgment result by high-accuracy fraud monitoring, and as information regarding fraud, etc. for the second item of product (pear), for example, "button registered" based on operation information is stored, "product inserted" as the recognition result by standard-accuracy product insertion / removal recognition, "product inserted" as the recognition result by high-accuracy product insertion / removal recognition, "not corresponding to fraud, etc." as the judgment result by high-accuracy fraud monitoring, and "not corresponding to fraud, etc." as the judgment result by high-accuracy fraud monitoring. The settlement device 50 displays the settlement list screen and small screen SG102 in FIG. 8(D) based on the registration information stored in the cloud server 20 and the above information.

[0190] As described above, the settlement device 50 displays "Unregistered product" and an image button on the product list screen of Figure 8 (D) (product list screen on the back) to indicate that one product item is unregistered. In addition to this information, however, it may also display information indicating the result of a standard-precision fraud monitoring assessment that "does not constitute fraud, etc." (for example, "Terminal OK") or information indicating the result of a high-precision fraud monitoring assessment that "corresponds to unregistered input" (for example, "Server NG (unregistered input)"). Furthermore, instead of or in addition to the above, the settlement device 50 may also display information indicating that the result of a fraud, etc. assessment by the mobile terminal 60 that "does not constitute fraud, etc." has been changed to "corresponds to unregistered input" on the cloud server 20 (for example, "Terminal OK → Server NG (unregistered input)").

[0191] Figures 8(B) to 8(D) are examples of displays on settlement device 50 when it is determined that an unregistered deposit has occurred on at least one of mobile terminal 60 and cloud server 20, but the same applies to the cases on settlement device 50 when it is determined that an unregistered deposit, an uncancelled withdrawal, or a cancelled unremoved has occurred on at least one of mobile terminal 60 and cloud server 20. Figures 8(E) and 8(F) will be described later.

[0192] (Notification of the results of fraudulent activity, etc., on devices used by store staff) The device used by the store clerk may access the cloud server 20 and display on a screen the recognition results of the mobile terminal 60 for inserting or removing a product (recognition results obtained through standard-accuracy product insert / removal recognition), the recognition results of the cloud server 20 for inserting or removing a product (recognition results obtained through high-accuracy product insert / removal recognition), the judgment results of the mobile terminal 60 for fraud (judgment results obtained through standard-accuracy fraud monitoring), and the judgment results of the cloud server 20 for fraud (judgment results obtained through high-accuracy fraud monitoring), which are stored in the cloud server 20. The device used by the store clerk may be, for example, the transaction status management device 40, a mobile terminal for the store clerk (smartphone, tablet terminal; not shown), etc. The device used by the store clerk may output audio instead of or in addition to a screen display. The same applies to the information below (information indicating that the recognition results of the mobile terminal 60 for inserting or removing a product have been overturned by the cloud server 20, etc.).

[0193] In addition, the cloud server 20 also stores the captured images (original captured images) that the cloud server 20 uses to perform high-precision product insertion and removal recognition, so the device used by the store clerk displays the recognition results of the high-precision product insertion and removal recognition along with the captured images, and the store clerk can also confirm whether the recognition results of the cloud server 20 for product insertion, etc. (i.e., the judgment results of the cloud server 20 for fraud, etc.) are correct or not.

[0194] The device used by the store clerk may display information that the recognition result of the product insertion, etc. by the mobile terminal 60 has been overturned by the cloud server 20, based on the above information stored in the cloud server 20 (the recognition result of the product insertion, etc. by the mobile terminal 60, the recognition result of the product insertion, etc. by the cloud server 20). The device used by the store clerk may display information that the recognition result of the product insertion, etc. by the mobile terminal 60 has been overturned by the cloud server 20, based on the above information stored in the cloud server 20 (the determination result of fraud, etc. by the mobile terminal 60, the determination result of fraud, etc. by the cloud server 20), instead of or in addition to the information that the recognition result of the product insertion, etc. by the mobile terminal 60 has been overturned by the cloud server 20.

[0195] For example, the device used by the store clerk may display, as the information before the pear in Figure 8 (C), information that the recognition result of the mobile terminal 60, "product inserted", has been changed to "product not inserted" on the cloud server 20 (for example, "terminal (no product inserted) → server (product inserted)"), or information that the judgment result of the mobile terminal 60, "corresponding to unregistered insertion", has been changed to "not corresponding to fraud, etc." on the cloud server 20 (for example, "terminal NG (unregistered insertion) → server OK").

[0196] Also, for example, the device used by the store clerk may display, as information regarding the one item of product in Figure 8 (D), information that the recognition result of product insertion, etc. by the mobile terminal 60, "product not inserted", has been changed to "product inserted" by the cloud server 20 (for example, "terminal (product inserted) → server (product not inserted)"), or information that the judgment result of fraud, etc. by the mobile terminal 60, "not corresponding to fraud, etc.", has been changed to "corresponding to unregistered insertion" by the cloud server 20 (for example, "terminal OK → server NG (unregistered insertion)").

[0197] The device used by the store clerk may display information that the recognition result of the product insertion, etc. by the mobile terminal 60 has been maintained in the cloud server 20, based on the above information stored in the cloud server 20 (the recognition result of the product insertion, etc. by the mobile terminal 60, the recognition result of the product insertion, etc. by the cloud server 20). The device used by the store clerk may display information that the determination result of the fraud, etc. by the mobile terminal 60 has been maintained in the cloud server 20, based on the above information stored in the cloud server 20 (the determination result of the fraud, etc. by the mobile terminal 60, the determination result of the fraud, etc. by the cloud server 20), instead of or in addition to the information that the recognition result of the product insertion, etc. by the mobile terminal 60 has been maintained in the cloud server 20.

[0198] For example, the device used by the store clerk may display, as information regarding the first item of product (loaf of bread) in Figure 8 (A), information that the recognition result of the mobile terminal 60 regarding the insertion of the product, etc., "product inserted", has been maintained in the cloud server 20 (for example, "product inserted (maintained)"), or information that the judgment result of the mobile terminal 60 regarding fraud, etc., "not corresponding to fraud, etc., has been maintained in the cloud server 20 (for example, "not corresponding to fraud, etc., (maintained)").

[0199] For example, the device used by the store clerk may display, as information regarding the one item of product in Figure 8 (B), information that the recognition result of the mobile terminal 60 regarding the insertion of a product, etc., "No product inserted" has been maintained in the cloud server 20 (for example, "No product inserted (maintained)"), or information that the judgment result of the mobile terminal 60 regarding fraud, etc., "Corresponds to unregistered insertion" has been maintained in the cloud server 20 (for example, "Corresponds to unregistered insertion (maintained)").

[0200] As described above, cloud server 20 stores information that age verification products are age verification products and information that explanation-required medicines are medicines that require explanation, so a device used by a store clerk may display information that age verification products are age verification products (e.g., "age verification product"), and information that explanation-required medicines are medicines that require explanation (e.g., "medicine that requires explanation"). The device used by a store clerk may display the recognition results of product insertion, etc. by mobile terminal 60 or cloud server 20, the determination results of fraud, etc. by mobile terminal 60 or cloud server 20, information that the recognition results of product insertion, etc. by mobile terminal 60 have been overturned by cloud server 20, information that the determination results of fraud, etc. by mobile terminal 60 have been overturned by cloud server 20, information that the recognition results of product insertion, etc. by mobile terminal 60 have been maintained by cloud server 20, and information that the determination results of fraud, etc. by mobile terminal 60 have been maintained by cloud server 20, in a display format different from the display format of "age verification product" or "medicine that requires explanation."

[0201] 9 to 11 are examples of captured images for explaining the recognition of product insertion and removal. For the purpose of explanation, FIGS. 9(A) to 9(H) show excerpts of some of the captured images continuously captured by the imaging unit (for example, 15 frames per second) in a scene where a loaf of bread is scanned and registered as one item. That is, time flows from FIG. 9(A) to FIG. 9(B) to FIG. 9(C), ..., FIG. 9(G) to FIG. 9(H), but one or more captured images (not shown) exist between the captured images (between the captured image of FIG. 9(A) and the captured image of FIG. 9(B), between the captured image of FIG. 9(B) and the captured image of FIG. 9(C), ..., between the captured image of FIG. 9(G) and the captured image of FIG. 9(H).

[0202] The captured image in FIG. 9(A) does not include a product. A loaf of bread is captured in the right-hand region of the captured image in FIG. 9(B). That is, the captured image in FIG. 9(B) is captured after the loaf of bread has appeared on the periphery of the captured image (the right-hand region in this example). In the captured image in FIG. 9(C), the barcode of the loaf of bread is captured at a slight angle. The captured image in FIG. 9(C) is, for example, an image captured when the orientation of the barcode is being adjusted. In the captured image in FIG. 9(D), the barcode of the loaf of bread is captured from the front. The captured image in FIG. 9(D) is, for example, an image captured when the barcode is being read. The captured image in FIG. 9(E) is, for example, an image captured when the loaf of bread is about to be placed at the bottom of a shopping basket. The captured image in FIG. 9(F) is, for example, an image captured after the loaf of bread has been placed at the bottom of the shopping basket. In the right-hand region of the captured image in FIG. 9(G), an empty hand (a hand not holding bread) is captured. The captured image in Fig. 9(G) is an image captured when, for example, the empty hand is moved outside the shopping basket after the bread has been placed therein. The captured image in Fig. 9(H) does not capture the empty hand. In other words, the captured image in Fig. 9(H) is an image captured after the empty hand has disappeared from the periphery of the captured image area.

[0203] 10(A) to 10(E) are excerpts for explanation of some of the captured images successively captured by the imaging unit in a scene following Fig. 9(H) where a pear is to be button-registered as the second item. That is, following Fig. 9(H), time flows from Fig. 10(A) → Fig. 10(B) → Fig. 10(C) → Fig. 10(D) → Fig. 10(E), but one or more captured images (not shown) exist between the captured images (between the captured image of Fig. 9(H) and the captured image of Fig. 10(A), between the captured image of Fig. 10(A) and the captured image of Fig. 10(B), ..., between the captured image of Fig. 10(D) and the captured image of Fig. 10(E).

[0204] A hand holding a pear is captured in the right area of ​​the captured image in FIG. 10(A). That is, the captured image in FIG. 10(A) is the captured image after a hand holding a pear appears on the periphery of the captured image area. A hand holding a pear is captured in the captured image in FIG. 10(B). The captured image in FIG. 10(B) is, for example, an image captured when a pear is about to be placed at the bottom of a shopping basket. The captured image in FIG. 10(C) is, for example, an image captured when a pear is being placed at the bottom of the shopping basket. An empty hand (a hand not holding a pear) is captured in the right area of ​​the captured image in FIG. 10(D). The captured image in FIG. 10(D) is, for example, an image captured when the empty hand has been moved outside the shopping basket after the pear has been placed. The empty hand is not captured in the captured image in FIG. 10(E). That is, the captured image in FIG. 10(E) is the captured image after the empty hand has disappeared from the periphery of the captured image area.

[0205] It should be noted that, since there is no barcode reading involved in button registration, there are no scenes in FIGS. 10(A) to 10(E) that correspond to FIGS. 9(C) and 9(D).

[0206] Figures 10(F) to 11(A) are excerpts for explanation of some of the captured images successively captured by the imaging unit in a scene of taking a pear out of a shopping basket, which follows on from Figure 10(E). That is, following Figure 10(E), time flows from Figure 10(F) → Figure 10(G) → Figure 10(H) → Figure 11(A), but one or more captured images (not shown) exist between the captured images (between the captured image of Figure 10(E) and the captured image of Figure 10(F), between the captured image of Figure 10(F) and the captured image of Figure 10(G), ..., between the captured image of Figure 10(H) and the captured image of Figure 11(A).

[0207] An empty hand (a hand not holding a product) is captured in the right area of ​​the captured image in FIG. 10(F). In other words, the captured image in FIG. 10(F) is an image captured after an empty hand has appeared on the periphery of the captured image area. The captured image in FIG. 10(G) is an image captured when, for example, a pear placed in a shopping basket is being grasped. The captured image in FIG. 10(H) is an image captured when, for example, the grasped pear is being moved outside the shopping basket to remove it from the basket. Neither the pear nor the hand is captured in the captured image in FIG. 11(A). In other words, the captured image in FIG. 11(A) is an image captured after the hand holding the product (a pear in this example) has disappeared from the periphery of the captured image area.

[0208] 11(B) to 11(F) are excerpts for explanation of some of the captured images successively captured by the imaging unit in a scene of rearranging items in a shopping basket (in this example, changing the position of a pear in the shopping basket) as another scene following FIG. 10(E) (a different scene from FIG. 10(F) to FIG. 11(A)). That is, following FIG. 10(E), time flows from FIG. 11(B) → FIG. 11(C) → ..., FIG. 11(E) → FIG. 11(F), but one or more captured images (not shown) exist between the captured images before and after (between the captured image of FIG. 10(E) and the captured image of FIG. 11(B), between the captured image of FIG. 11(B) and the captured image of FIG. 11(C), ..., between the captured image of FIG. 11(E) and the captured image of FIG. 11(F)).

[0209] An empty hand (a hand not holding a product) is captured in the right area of ​​the captured image in FIG. 11(B). In other words, the captured image in FIG. 11(B) is the image captured after an empty hand has appeared on the periphery of the captured image area. The captured image in FIG. 11(C) is, for example, an image captured when a pear placed in a shopping basket is being grasped. The captured image in FIG. 11(D) is, for example, an image captured when the position of the pear is being changed. The captured image in FIG. 11(E) is, for example, an image captured when the empty hand has been moved to the outside of the shopping basket after the position of the pear has been changed. The captured image in FIG. 11(F) does not capture an empty hand. In other words, the captured image in FIG. 11(H) is the image captured after the empty hand has disappeared from the periphery of the captured image area.

[0210] (Image taken when putting an item into a shopping basket) When an item is dropped into a shopping basket, the camera captures images of the item (or a hand holding the item) appearing from the periphery (Fig. 9(A) → Fig. 9(B), Fig. 9(H) → Fig. 10(A)) and the empty hand disappearing from the periphery (Fig. 9(G) → Fig. 9(H), Fig. 10(D) → Fig. 10(E)). In other words, if the camera captures images of the item (or a hand holding the item) appearing from the periphery and the empty hand disappearing from the periphery, it can be determined that the item has been dropped.

[0211] 9(C) and 9(D) are captured images related to the scan operation, but the captured images are the same whether the product is scanned and registered or not. Also, if a scan operation is not performed on a scanned and registered product (e.g., bread), naturally there will be no captured images related to the scan operation (FIGS. 9(C) and 9(D)), just like a button-registered product (e.g., pear).

[0212] (Image taken when removing an item from a shopping basket) When an item is removed from a shopping basket, the camera captures an image of an empty hand appearing from the periphery (Fig. 10(E) → Fig. 10(F)) and an image of the hand holding the item disappearing from the periphery (Fig. 10(H) → Fig. 11(A)). In other words, if the camera captures an image of an empty hand appearing from the periphery and an image of the hand holding the item disappearing from the periphery, it can be determined that an item is being removed.

[0213] Although FIGS. 10(F) to 11(A) show an example of removing a button-registered product (e.g., pear) from a shopping basket, the same applies to removing a scan-registered product (e.g., bread) from a shopping basket.

[0214] (Image taken when arranging (tidying up) items in a shopping basket) When rearranging items in a shopping basket, an image is captured of an empty hand appearing from the periphery (Fig. 11(A) → Fig. 11(B)) and an empty hand disappearing (Fig. 11(E) → Fig. 11(F)). In other words, if an image is captured of an empty hand appearing from the periphery and an empty hand disappearing from the periphery, it can be determined that the action is not an item being added.

[0215] If an image captures an empty hand appearing from the periphery and an empty hand disappearing from the periphery, and if the status of the products (the position of each product, the number of products) changes between the scene before the empty hand appears and the scene after the empty hand disappears from the periphery, it can be determined that the products in the shopping basket have been rearranged. A change in the number of products occurs when a product is hidden behind other products due to rearrangement, or when the product appears from behind other products due to rearrangement. Furthermore, if an image captures no image of an empty hand appearing from the periphery or an empty hand disappearing from the periphery, but the status of the products (the position of each product, the number of products captured) changes, this means that the products have shifted position or collapsed due to shaking of cart 70, or have been hidden behind or reappeared from behind other products.

[0216] Another example of product insertion will be described. FIGS. 11(G) and 11(H) are captured images of a different scene from those in FIGS. 9(A) to 11(F). In the captured image of FIG. 11(G), for example, product a is captured but product b is not captured. In the captured image of FIG. 11(H), for example, product a and product b are captured. One or more captured images (not shown) exist between the captured image of FIG. 11(G) and the captured image of FIG. 11(H), but the following description will be given assuming that there are two ways in which the captured image of FIG. 11(G) and the captured image of FIG. 11(H) occur before or after each other. In other words, time flows from the captured image of FIG. 11(G) to one or more captured images (not shown) to the captured image of FIG. 11(H), or from the captured image of FIG. 11(H) to one or more captured images (not shown) to the captured image of FIG. 11(G).

[0217] As time passes, from the captured image in Figure 11(G) to one or more captured images to the captured image in Figure 11(H) (a change from no image of can b to an image being captured), if there is a scene in one or more captured images where a hand holding a product (can b) appears from the periphery and, after the scene where the hand holding the product appears, an empty hand disappears from the periphery, it can be determined that a product has been inserted.

[0218] In the transition over time from the captured image in Figure 11(G) → one or more captured images → the captured image in Figure 11(H), if there is a scene in one or more captured images where an empty hand appears from the periphery and then a scene where the empty hand disappears from the periphery after the scene where the empty hand appears, it can be determined that no product has been inserted. This is considered to be a situation where product b has appeared from behind product a due to tidying up.

[0219] As time passes, from the captured image in Figure 11(G) to one or more captured images to the captured image in Figure 11(H), if there is no scene in which a product or hand appears from the periphery in one or more captured images (and naturally there is no scene in which a product or hand disappears from the periphery), it can be determined that no product has been inserted. This is considered to be a situation in which product b has appeared from behind product a due to shaking or the like.

[0220] As time passes, the captured image in Figure 11(H) changes to one or more captured images, and then to the captured image in Figure 11(G) (a change from an image of can b being captured to no image being captured), and if, in one or more captured images, there is a scene in which an empty hand appears from the periphery and, after the scene in which the empty hand appears, there is a scene in which the hand holding the product (can b) disappears from the periphery, then it can be determined that a product has been removed.

[0221] In the change over time from the captured image in Figure 11(H) → one or more captured images → the captured image in Figure 11(G) (a change from capturing an image of can b → not capturing an image), if there is a scene in one or more captured images where an empty hand appears from the periphery and then a scene where the empty hand disappears from the periphery after the scene where the empty hand appears, it can be determined that no product has been taken out. This is considered to be a situation where product b has been hidden behind product a due to tidying up.

[0222] As time passes, from the captured image in Figure 11(H) to one or more captured images to the captured image in Figure 11(G) (a change from capturing an image of can b to not capturing an image), if there is no scene in one or more captured images where a product or hand appears from the periphery (and of course there is no scene where a product or hand disappears from the periphery), it can be determined that no product has been taken out. This is likely a situation in which product b has been hidden behind product a due to shaking or other reasons.

[0223] (Images used for standard-precision product insertion / removal recognition / high-precision product insertion / removal recognition) The frame rate of the captured image (moving image) by the imaging unit is, for example, 15 fps (frames per second). Also, the captured image 4 seconds after the captured image of Fig. 11(G) is the captured image of Fig. 11(H). That is, the frame 60 frames after the frame of Fig. 11(G) is the frame of Fig. 11(H).

[0224] As explained above, the captured image used for standard-precision product insertion / removal recognition, which prioritizes speed, contains less information than the captured image used for high-precision product insertion / removal recognition, which prioritizes accuracy. However, one or more of the following methods (A) to (E) may be adopted (two or more may be combined).

[0225] (A) Standard-accuracy product insertion / removal recognition uses some of the frames to recognize product insertion, etc., while high-accuracy product insertion / removal recognition uses all of the frames to recognize product insertion, etc. For example, standard-accuracy product insertion / removal recognition may use 1 / 5 of the frames (1 out of every 5 frames) to recognize product insertion, etc. For example, there are 61 frames between Figures 11(G) and 11(H), including both ends. High-accuracy product insertion / removal recognition may use all 61 frames to recognize product insertion, etc., while standard-accuracy product insertion / removal recognition may use 12 frames to recognize product insertion, etc. Note that high-accuracy product insertion / removal recognition only needs to have a larger number of target frames than standard-accuracy product insertion / removal recognition, and high-accuracy product insertion / removal recognition may also use some of the frames to recognize product insertion, etc. For example, standard-accuracy product insertion / removal recognition may use 1 / 10 of the frames to recognize product insertion, etc., while high-accuracy product insertion / removal recognition may use 1 / 3 of the frames to recognize product insertion, etc. In the case of Method A, standard-precision product insertion / removal recognition improves response because the amount of information (number of frames) to be processed is small, while high-precision product insertion / removal recognition improves accuracy because the amount of information to be processed is large.

[0226] (B) Standard-precision product insertion / removal recognition recognizes product insertion, etc. using each frame individually, while high-precision product insertion / removal recognition recognizes product insertion, etc. using multiple frames. In other words, standard-precision product insertion / removal recognition recognizes product insertion, etc. without considering changes (movement) of objects between frames, while high-precision product insertion / removal recognition recognizes product insertion, etc. by considering changes (movement) of objects between frames. High-precision product insertion / removal recognition may recognize product insertion, etc. using three frames (a set of three frames). For example, for five frames (frame 1 → frame 2 → frame 3 → frame 4 → frame 5) over 1 / 3 seconds, standard-accuracy product insertion / removal recognition recognizes product insertion, etc. using frame 1 alone, frame 2 alone, frame 3 alone, ..., frame 5 alone (or, if only some frames are used, product insertion, etc. is recognized using only some of those frames). However, standard-accuracy product insertion / removal recognition recognizes product insertion, etc. using three frames from frames 1 to 3, three frames from frames 2 to 4, and three frames from frames 3 to 5. Note that high-accuracy product insertion / removal recognition may take movement into consideration more than standard-accuracy product insertion / removal recognition, and movement may also be considered in standard-accuracy product insertion / removal recognition. For example, standard-accuracy product insertion / removal recognition may recognize product insertion, etc. using a set of three frames, while high-accuracy product insertion / removal recognition may recognize product insertion, etc. using a set of five frames. In the case of Method B, standard-precision product insertion / removal recognition improves response because the amount of information (number of frames) to be processed is small, while high-precision product insertion / removal recognition improves accuracy because the amount of information to be processed is large.

[0227] (C) In standard-accuracy product insertion / removal recognition, product insertion, etc. is recognized using a grayscale captured image, while in high-accuracy product insertion / removal recognition, product insertion, etc. is recognized using a full-color captured image (approximately 16.77 million colors). That is, in high-accuracy product insertion / removal recognition, a full-color captured image is used as is, as captured by the imaging unit, while in standard-accuracy product insertion / removal recognition, a grayscale captured image is used, in which the captured image as captured by the imaging unit is grayscaled (preprocessed). Therefore, for example, in the example of FIG. 9(G), in high-accuracy product insertion / removal recognition, whether an object disappearing from the periphery is an empty hand is determined using a full-color captured image, while in standard-accuracy product insertion / removal recognition, whether an object disappearing from the periphery is an empty hand is determined using a grayscale captured image. Note that high-accuracy product insertion / removal recognition may use a larger number of image colors than standard-accuracy product insertion / removal recognition; therefore, standard-accuracy product insertion / removal recognition may also use a color captured image to recognize product insertion, etc., while high-accuracy product insertion / removal recognition may also use a color captured image to recognize product insertion, etc. For example, standard-accuracy product insertion / removal recognition may recognize product insertion, etc. using a captured image with 256 colors, while high-accuracy product insertion / removal recognition may recognize product insertion, etc. using a captured image with 4096 colors. Note that standard-accuracy product insertion / removal recognition may use a captured image converted into a different color space (for example, HSV color space). In the case of Method C, standard-accuracy product insertion / removal recognition improves response because it processes a small amount of information (colors), while high-accuracy product insertion / removal recognition improves accuracy because it processes a large amount of information.

[0228] (D) Standard-accuracy product insertion / removal recognition uses compressed captured images to recognize product insertion, etc., while high-accuracy product insertion / removal recognition uses uncompressed captured images to recognize product insertion, etc. For example, if the captured image size is 1280 x 720 px, it may be compressed to 64 x 64 px, 32 x 32 px, or 8 x 8 px depending on the purpose of the judgment. Note that high-accuracy product insertion / removal recognition only requires that the images used are clearer (more detailed) than standard-accuracy product insertion / removal recognition, and high-accuracy product insertion / removal recognition may also use compressed captured images to recognize product insertion, etc. In the case of method D, standard-accuracy product insertion / removal recognition processes a smaller amount of information (number of pixels), resulting in improved response, while high-accuracy product insertion / removal recognition processes a larger amount of information, resulting in improved accuracy.

[0229] (E) In standard-accuracy product insertion / removal recognition, a partial area of ​​the captured image is used to recognize product insertion, etc., while in high-accuracy product insertion / removal recognition, the entire area of ​​the captured image is used to recognize product insertion, etc. For example, in standard-accuracy product insertion / removal recognition, the captured area within the frame of the shopping basket may be cropped out (i.e., an image with the outside of the shopping basket removed) to recognize product insertion, etc. Note that high-accuracy product insertion / removal recognition only requires a wider captured image range than standard-accuracy product insertion / removal recognition, and high-accuracy product insertion / removal recognition may also use a partial area of ​​the captured image to recognize product insertion, etc. In the case of method E, standard-accuracy product insertion / removal recognition processes a smaller amount of information (number of pixels), resulting in improved response, while high-accuracy product insertion / removal recognition processes a larger amount of information, resulting in improved accuracy.

[0230] We will explain how to achieve standard-accuracy product insertion / removal recognition and high-accuracy product insertion / removal recognition using a trained model.

[0231] Hereinafter, the trained model for determining whether a product has been inserted using standard-accuracy product insertion / removal recognition will be referred to as the "standard-accuracy product insertion determination model," and the trained model for determining whether a product has been inserted / removed using standard-accuracy product insertion / removal recognition will be referred to as the "standard-accuracy product removal determination model." Furthermore, the trained model for determining whether a product has been inserted using high-accuracy product insertion / removal recognition will be referred to as the "high-accuracy product insertion determination model," and the trained model for determining whether a product has been inserted / removed using high-accuracy product insertion / removal recognition will be referred to as the "high-accuracy product removal determination model."

[0232] The learning model generation device generates a standard-accuracy product insertion determination model using the captured image for standard-accuracy product insertion / removal recognition (a captured image with a relatively small amount of information) as described above. The mobile terminal 60 inputs the captured image for standard-accuracy product insertion / removal recognition as described above into the standard-accuracy product insertion determination model, and obtains the standard-accuracy product insertion recognition result (the product insertion recognition result by standard-accuracy product insertion / removal recognition).

[0233] The learning model generation device generates a standard-accuracy product removal judgment model using the captured image for standard-accuracy product insertion / removal recognition (a captured image with a relatively small amount of information) as described above. The mobile terminal 60 inputs the captured image for standard-accuracy product insertion / removal recognition as described above into the standard-accuracy product removal judgment model, and obtains the standard-accuracy product removal recognition result (the product removal recognition result by standard-accuracy product insertion / removal recognition).

[0234] The learning model generation device generates a high-precision product insertion determination model using captured images (captured images with a relatively large amount of information) for high-precision product insertion / removal recognition as described above. The cloud server 20 inputs the captured images for high-precision product insertion / removal recognition as described above into the high-precision product insertion determination model and obtains high-precision product insertion recognition results (recognition results of product insertion by high-precision product insertion / removal recognition). The high-precision product insertion determination model, which is generated with a larger amount of information than the standard-precision product insertion determination model, has higher accuracy in recognizing product insertion than the standard-precision product insertion determination model.

[0235] The learning model generation device generates a high-precision product removal judgment model using captured images (captured images with a relatively large amount of information) for high-precision product insertion / removal recognition as described above. The cloud server 20 inputs the captured images for high-precision product insertion / removal recognition as described above into the high-precision product removal judgment model and obtains high-precision product removal recognition results (recognition results of product removal by high-precision product insertion / removal recognition). The high-precision product removal judgment model, which is generated with a larger amount of information than the standard-precision product removal judgment model, has higher accuracy in recognizing product removal than the standard-precision product removal judgment model.

[0236] In addition, the standard accuracy product input judgment model and standard accuracy product removal judgment model generated by the learning model generation device may be provided to the mobile terminal 60 as part of app A (as a function of app A), or may be provided to the mobile terminal 60 separately from app A (as a separate program) and together with app A.

[0237] Furthermore, the high-precision product input judgment model and the high-precision product removal judgment model generated by the learning model generation device may be part (a function of program B) of a program (referred to as program B) on the cloud server 20 side corresponding to app A, or may be a separate program on the cloud server 20 side that works in conjunction with program B.

[0238] (Restrictions on product input / output recognition) The mobile terminal 60 may perform recognition using the standard-accuracy product insertion / removal recognition method when attached to the holder unit 72, but may not perform recognition using the standard-accuracy product insertion / removal recognition method when not attached to the holder unit 72. In other words, since it is difficult to recognize product insertion, etc. from an image captured when not attached to the holder unit 72, the mobile terminal 60 may not perform standard-accuracy product insertion / removal recognition. For the same reason, the cloud server 20 may not perform product insertion / removal recognition using an image captured by the mobile terminal 60 when not attached to the holder unit 72.

[0239] Note that the mobile terminal 60 may determine that it is attached to the holder unit 72 if there is no change in its posture for a predetermined time (for example, if the acceleration sensor or gyro sensor does not detect any movement). The mobile terminal 60 may also determine whether it is attached to the holder unit 72 based on a captured image (if the frame of the shopping basket, which would be captured if the mobile terminal 60 is attached to the holder unit 72, is captured properly, it may be determined that the mobile terminal 60 is attached to the holder unit 72). The cart 70 may also be equipped with a charging unit that charges the mobile terminal 60 attached to the holder unit 72, and the mobile terminal 60 may determine that it is attached to the holder unit 72 if it recognizes that it is being charged.

[0240] In addition, the mobile terminal 60 may be configured not to transmit captured images to the cloud server 20 when it is not attached to the holder unit 72 so that the cloud server 20 does not recognize the insertion and removal of products using captured images taken by the mobile terminal 60 when it is not attached to the holder unit 72.

[0241] The mobile terminal 60 may be configured not to perform standard-accuracy product insertion / removal recognition when the remaining power is less than a predetermined threshold (e.g., 10%). The mobile terminal 60 may be configured not to transmit captured images to the cloud server 20 when the remaining power is less than a predetermined threshold. In other words, since there is a risk that scan registration will not be possible when the remaining power is low, the burden (transmission of information (captured images) required for standard-accuracy product insertion / removal recognition by the mobile terminal itself and high-accuracy product insertion / removal recognition to the cloud server 20) may be reduced.

[0242] It has been explained that the recognition results of product insertion, etc. by the mobile terminal 60 (recognition results by standard-accuracy product insertion / removal recognition) and the recognition results of product insertion, etc. by the cloud server 20 (recognition results by high-accuracy product insertion / removal recognition) may be stored, but if the mobile terminal 60 does not perform standard-accuracy product insertion / removal recognition or if the cloud server 20 does not perform high-accuracy product insertion / removal recognition, information to the effect that it was not performed may be stored.

[0243] (Product insertion / removal recognition using information other than captured images) The mobile terminal 60 may have a product insertion / removal recognition function based on other information instead of or in addition to the captured image. That is, the mobile terminal 60 may recognize the insertion of a product, etc., using other information instead of or in addition to the captured image. Similarly, the cloud server 20 may recognize the insertion of a product, etc., using other information instead of or in addition to the captured image.

[0244] Vibrations occur when a product is added to or removed from the shopping basket. The mobile terminal 60 (as well as the cloud server 20) may recognize the addition of a product, etc., using vibration information instead of or in addition to a captured image. Vibrations may be detected by an acceleration sensor or a gyro sensor. For example, the mobile terminal 60 may recognize the addition of a product, etc., using a detection value from its own vibration detection unit (acceleration sensor, gyro sensor). Furthermore, by attaching a vibration detection device including a vibration detection unit and a communication unit to the cart 70 (or by preparing a cart 70 including a vibration detection device), the mobile terminal 60 may receive a detection value from the vibration detection device and recognize the addition of a product, etc., using the detection value received from the vibration detection device. The cloud server 20 may receive a detection value from the mobile terminal 60 (a detection value acquired by the mobile terminal 60 from its own vibration detection unit or a detection value received by the mobile terminal 60 from the vibration detection device) and recognize the addition of a product, etc., using the detection value received from the mobile terminal 60.

[0245] When an item is added to or removed from the shopping basket, the number of items in the shopping basket changes, and therefore the weight of the items in the shopping basket also changes. The mobile terminal 60 (as well as the cloud server 20) may recognize the addition of an item, etc., by using the weight of the items in the shopping basket instead of or in addition to the captured image or vibration information. For example, by attaching a weighing device to the cart 70 (or by providing a cart 70 equipped with a weighing device) that includes a weighing unit (e.g., a weighing unit (scale unit) having a platform on which the shopping basket is placed) that weighs the shopping basket and the items in the shopping basket, and a communication unit, the mobile terminal 60 may receive a weight value from the weighing device and recognize the addition of an item, etc., using the weight value received from the weighing device. The cloud server 20 may receive a weight value from the mobile terminal 60 (the weight value received by the mobile terminal 60 from the weighing device) and recognize the addition of an item, etc., using the weight value received from the mobile terminal 60.

[0246] When using only one of the information values ​​of vibration detection and weight measurement, or when using two or more types of information (two or more of captured image, detected vibration, and measured weight), the standard-accuracy product insertion / removal recognition has a faster recognition response than the high-accuracy product insertion / removal recognition, just as when using captured images. When using two or more types of information, it is sufficient that the standard-accuracy product insertion / removal recognition has a faster recognition response overall, and there may be information for which there is no difference in the amount of information used between the standard-accuracy product insertion / removal recognition and the high-accuracy product insertion / removal recognition. As an example, when two types of information, captured image and measurement value, there may be no difference in the amount of information for the measurement value between the standard-accuracy product insertion / removal recognition and the high-accuracy product insertion / removal recognition. In other words, in standard-accuracy product insertion / removal recognition, the amount of information in the captured image and the amount of information in the measurement values ​​can be reduced to recognize product insertion, etc., and in high-accuracy product insertion / removal recognition, the amount of information in the measurement values ​​can be reduced without reducing the amount of information in the captured image to recognize product insertion, etc., or alternatively, in standard-accuracy product insertion / removal recognition, the amount of information in the captured image can be reduced but the amount of information in the measurement values ​​can be reduced to recognize product insertion, etc., and in high-accuracy product insertion / removal recognition, the amount of information in the captured image can be reduced to recognize product insertion, etc. without reducing the amount of information in the measurement values.

[0247] When using only one of the vibration detection value or the weight measurement value, or when using two or more types of information (two or more of the captured image, the vibration detection value, and the weight measurement value), a trained model may be used, just as when using a captured image.

[0248] The learning model generation device generates a standard-accuracy product insertion / removal judgment model using information for standard-accuracy product insertion / removal recognition (one or more of captured images, vibration detection values, and weight measurement values). The mobile terminal 60 inputs the information for standard-accuracy product insertion / removal recognition into the standard-accuracy product insertion judgment model and obtains the standard-accuracy product insertion recognition result (the product insertion recognition result by standard-accuracy product insertion / removal recognition). The same applies to the generation and use of the standard-accuracy product removal judgment model.

[0249] The learning model generation device generates a high-precision product insertion / removal judgment model using information for high-precision product insertion / removal recognition (one or more of captured images, vibration detection values, and weight measurement values). The cloud server 20 inputs the information for high-precision product insertion / removal recognition into the high-precision product insertion judgment model and obtains high-precision product insertion recognition results (recognition results of product insertion by high-precision product insertion / removal recognition). The same applies to the generation and use of the high-precision product removal judgment model.

[0250] When generating a standard-accuracy product insertion determination model using vibration detection values, the learning model generation device learns about vibration using, for example, a large number of vibration detection values ​​(vibration detection values ​​for standard-accuracy product insertion / removal recognition) when a product is inserted into a shopping basket and corresponding training data (trainer data indicating that each is a detection value indicating that a product has been inserted), and a large number of vibration detection values ​​(vibration detection values ​​for standard-accuracy product insertion / removal recognition) when a product has not been inserted into the shopping basket and corresponding training data (trainer data indicating that each is a detection value indicating that a product has not been inserted). The same applies to generating a standard-accuracy product removal determination model using vibration detection values, generating a standard-accuracy product insertion determination model using weight measurement values, and generating a standard-accuracy product removal determination model using weight measurement values.

[0251] When generating a high-precision product insertion determination model using vibration detection values, the learning model generation device may, for example, use a large number of vibration detection values ​​(vibration detection values ​​for high-precision product insertion / removal recognition) when a product is inserted into a shopping basket and corresponding training data (trainer data indicating that each is a detection value indicating that a product has been inserted), and a large number of vibration detection values ​​(vibration detection values ​​for high-precision product insertion / removal recognition) when a product has not been inserted into the shopping basket and corresponding training data (trainer data indicating that each is a detection value indicating that a product has not been inserted). The same applies to generating a high-precision product removal determination model using vibration detection values, generating a high-precision product insertion determination model using weight measurement values, and generating a high-precision product removal determination model using weight measurement values.

[0252] Note that instead of or in addition to the above-mentioned information (captured image, detected vibration value, and measured weight value), sound pressure or temperature may be used to recognize the insertion of a product, etc. When sound pressure or temperature is used, a trained model may be used, just as when a captured image is used.

[0253] (Recognizing product input using temperature) The mobile terminal 60 may be equipped with a temperature sensor (thermometer) that measures (detects) temperature. The temperature sensor measures the temperature around the shopping basket (for example, the space between the bottom of the holder 72 and the bottom of the shopping basket; the space shown in FIG. 9(A)). Because the temperature of a hand (body temperature, around 36°C) is generally higher than other temperatures (items, room temperature), the temperature sensor measures a higher temperature when a hand is present around the shopping basket than when a hand is not present around the shopping basket. In other words, the temperature sensor can determine whether a hand is present around the shopping basket. For example, if the temperature (temperature information) measured by the temperature sensor is below a reference temperature, it is determined that a hand is not present, and if the temperature is above the reference temperature, it is determined that a hand is present. The reference temperature is a temperature set by actually measuring the temperature when no hands are present around the shopping basket and the temperature when hands are present around the shopping basket (for example, a temperature that is below the set temperature in most cases when no hands are present and that is above the set temperature in most cases when hands are present).

[0254] Based on the time-series measurement results of the temperature sensor, it is possible to distinguish whether a hand has been put into or taken out of the shopping basket. In other words, when a state where a hand is not present changes to a state where a hand is present, it is recognized that a hand has been put into the shopping basket, and when a state where a hand is present changes to a state where a hand is not present, it is recognized that a hand has been taken out of the shopping basket. For example, when the temperature measured by the temperature sensor changes from below a reference temperature to above the reference temperature, it is recognized that a hand has been put into the shopping basket, and when it changes from above the reference temperature to below the reference temperature, it is recognized that a hand has been taken out of the shopping basket.

[0255] Furthermore, by combining the time-series measurement results from the temperature sensor with other information, it is possible to recognize whether an empty hand has been placed in the shopping basket, whether a hand holding a product has been placed in the shopping basket, whether an empty hand has been removed from the shopping basket, or whether a hand holding a product has been removed from the shopping basket. In other words, it is possible to recognize the insertion or removal of products.

[0256] For example, if the weighing value measured by the weighing unit (scale unit) increases during the period when the temperature measured by the temperature sensor changes from below the reference temperature to above the reference temperature to below the reference temperature (from the time a hand is placed in the shopping basket to the time the hand is removed from the shopping basket), it can be recognized that a product has been inserted, and if the weighing value measured by the weighing unit decreases during the period when the temperature measured by the temperature sensor changes from below the reference temperature to above the reference temperature to below the reference temperature, it can be recognized that a product has been removed.

[0257] Furthermore, in the explanations of Figures 9 to 11, it is assumed that the accuracy of object recognition is high enough to be able to distinguish between an empty hand and a hand holding a product from a captured image. However, if the accuracy of object recognition is such that the overall size of a moving object can be grasped (for a hand holding a product, the product and the hand cannot be distinguished and the hand is recognized as one object), then the size (size of the circumscribing rectangle) of the moving object at the moment when the hand was put into the shopping basket (at the moment when the temperature measured by the temperature sensor changed from below the reference temperature to above the reference temperature) will be the same as the size of the moving object at the moment when the hand was taken out of the shopping basket during the period when the temperature measured by the temperature sensor changed from below the reference temperature to above the reference temperature again below the reference temperature (from the time the hand was put into the shopping basket to the time the hand was taken out of the shopping basket). If the size of the object moving in the scene (when the temperature measured by the temperature sensor changes from above the reference temperature to below the reference temperature) is smaller (when the size of the object in the scene where a hand is put into the shopping basket is the size of a hand + product, and the size of the object in the scene where a hand is taken out of the shopping basket is assumed to be the size of a hand), it can be recognized as an item being inserted; if the size of the object moving in the scene where a hand is put into the shopping basket is larger than the size of the object moving in the scene where a hand is taken out of the shopping basket (when the size of the object in the scene where a hand is put into the shopping basket is the size of a hand, and the size of the object in the scene where a hand is taken out of the shopping basket is assumed to be the size of a hand + product), it can be recognized as an item being removed.

[0258] Although there are products that are warmer than the temperature of a hand, by combining the time-series measurement results of the temperature sensor with other information (weighing values ​​from the weighing unit, captured images), it is possible to recognize the insertion and removal of products. Because the temperature measured by the temperature sensor is more influenced by heat sources present in the front (top of the shopping basket) than by heat sources present in the back (bottom of the shopping basket) (in other words, the influence of products at the bottom of the shopping basket that are warmer than the temperature of a hand is relatively small), it is possible to recognize the scene where an empty hand is removed from the shopping basket at the end of the product insertion scene.

[0259] Although the mobile terminal 60 has been described as being equipped with a temperature sensor, the cart 70 or the shopping basket may also be equipped with a temperature sensor. Furthermore, the number of temperature sensors may be one or more. For example, one or more of the mobile terminal 60, the cart 70, and the shopping basket may be equipped with a total of two or more temperature sensors (for example, the mobile terminal 60 may be equipped with one temperature sensor and the cart 70 may be equipped with one temperature sensor, or the cart 70 may be equipped with three temperature sensors). The same applies to the infrared camera (described below).

[0260] Furthermore, one or more infrared cameras may be used instead of or in addition to one or more temperature sensors. For example, the mobile terminal 60 may have an infrared camera function (an infrared camera gadget may be connected to the mobile terminal 60, or the mobile terminal 60 may have a built-in infrared camera). If an infrared camera is used, a hand can be recognized as an object that is a heat source, and therefore, the insertion and removal of a product can be recognized with higher accuracy.

[0261] When a temperature sensor is used, it is combined with other information to recognize product insertion and removal, so the amount of information can be the same in standard-accuracy product insertion / removal recognition situations as in high-accuracy product insertion / removal recognition situations. In other words, when a temperature sensor is used, the amount of other information to be combined with it can be made different between high-accuracy product insertion / removal recognition situations and standard-accuracy product insertion / removal recognition situations (the amount of information in the former situation can be made larger).

[0262] However, even when a temperature sensor is used, the amount of information from the temperature sensor may be different between high-precision product insertion / removal recognition situations and standard-precision product insertion / removal recognition situations (the amount of information may be greater in the former situations).

[0263] For example, the mobile terminal 60 side acquires temperature information at a fine pitch (fine pitch in terms of time) and transmits it to the cloud server 20. The mobile terminal 60 may recognize the insertion of a product, etc. (standard-accuracy product insertion / removal recognition) using part of the information it has acquired (temperature information at a coarse pitch after thinning out), and the cloud server 20 may recognize the insertion of a product, etc. (high-accuracy product insertion / removal recognition) using all the temperature information received from the mobile terminal 60 (temperature information at a fine pitch acquired by the mobile terminal 60 side).

[0264] Also, for example, the mobile terminal 60 side acquires temperature information at the limit of resolution (for example, in units of 0.1 degrees) and transmits it to the cloud server 20. The mobile terminal 60 may recognize the insertion of a product, etc. (standard-precision product insertion / removal recognition) using information obtained by preprocessing the information it has acquired (for example, temperature information rounded to the nearest 0.5 degrees), and the cloud server 20 may recognize the insertion of a product, etc. (high-precision product insertion / removal recognition) using the original temperature information received from the mobile terminal 60.

[0265] When an infrared camera is used, it can recognize product insertion and removal either alone or in combination with other information. As with images captured by a regular camera, the amount of information captured by the infrared camera may be different between high-precision product insertion and removal recognition scenes and standard-precision product insertion and removal recognition scenes (the amount of information may be greater for the former).

[0266] The mobile terminal 60 may display temperature information (temperature display) on the display unit. For example, when the mobile terminal 60 detects the temperature of a hand (when it recognizes a hand (an empty hand or a hand holding a product)), it may display the temperature in a display mode different from the normal display mode.

[0267] An image captured by an infrared camera may be displayed (for example, a thermographic image produced by thermography). For example, the portable terminal 60 may display (based on an operation by the operator) an image captured by an infrared camera provided in the terminal itself (or an image captured by an infrared camera provided in the cart 70, etc.). The portable terminal 60 may display an image captured by an infrared camera (for example, a thermographic image) in the display area for the through image (captured image display area HR620) based on the operation. The portable terminal 60 may switch between displaying (for example, displaying in the display area for the through image) a normal captured image (a captured image used for scan registration, etc.) and an image captured by an infrared camera (for example, a thermographic image).

[0268] Alternatively, the mobile terminal 60 may transmit images captured by the infrared camera to the cloud server 20, and the device used by the store clerk may display the images captured by the infrared camera stored in the cloud server 20.

[0269] Furthermore, the mobile terminal 60 may display (for example, display in the display area of ​​the through-the-lens image) an image captured by an infrared camera (for example, a thermographic image) when a hand (an empty hand or a hand holding a product) is not recognized, and may display (for example, display in the display area of ​​the through-the-lens image) a normal image captured by an infrared camera when a hand is recognized. This allows the user to check the positions of hot and cold items and place them apart before adding them to the shopping basket. Furthermore, since the normal image captured is viewed when scanning and registering, there is no sense of incongruity. Note that, different from the above, the mobile terminal 60 may display (for example, display in the display area of ​​the through-the-lens image) an image captured by an infrared camera (for example, a thermographic image) whether a hand is recognized or not (the normal image captured is used for the scan registration itself, which is not displayed). In this case, the user can add the products before adding them to the shopping basket and check the positions of the hot and cold items even while adding the products to the shopping basket.

[0270] Furthermore, instead of or in addition to displaying an image captured by an infrared camera (e.g., a thermographic image), the mobile terminal 60 may also notify (on a small screen or with a voice) that a warm and a cold object have been placed in close proximity to each other (e.g., when the two are placed within a predetermined distance (e.g., 3 cm)).

[0271] (Two types of barcode reading functions with different reading accuracy) The sales system 1 may be equipped with a barcode reading function with standard accuracy for reading barcodes and a high-accuracy barcode reading function with higher accuracy for reading barcodes than the standard accuracy. Hereinafter, barcode reading using the standard-accuracy barcode reading function will be referred to as "standard-accuracy barcode reading," and barcode reading using the high-accuracy barcode reading function will be referred to as "high-accuracy barcode reading."

[0272] Although it has been explained that both the mobile terminal 60 and the cloud server 20 have a barcode reading function, the mobile terminal 60 executes standard-accuracy barcode reading, and the cloud server 20 executes high-accuracy barcode reading.

[0273] The period during which the mobile terminal 60 performs standard-accuracy barcode reading is naturally the same as the period during which each product is registered, i.e., the period during which the mobile terminal 60 performs standard-accuracy product insertion / removal recognition and standard-accuracy fraud monitoring (the period shown in FIGS. 7(A) to 7(C)). The period during which the cloud server 20 performs high-accuracy barcode reading may be the same as the period during which the cloud server 20 performs high-accuracy product insertion / removal recognition and high-accuracy fraud monitoring (any of the periods shown in FIGS. 7(A) to 7(C)).

[0274] High-precision barcode reading has higher barcode reading accuracy than standard-precision barcode reading, but takes longer to read. Standard-precision barcode reading outputs reading results in a shorter time than high-precision barcode reading, but the barcode reading accuracy is lower. In other words, standard-precision barcode reading is inferior to high-precision barcode reading in terms of accuracy (reliability), but is superior to high-precision barcode reading in terms of speed (response). On the other hand, high-precision barcode reading is inferior to standard-precision barcode reading in terms of speed, but is superior to standard-precision barcode reading in terms of accuracy.

[0275] The amount of information in the captured image used for high-precision barcode reading is greater than the amount of information in the captured image used for standard-precision barcode reading. High-precision barcode reading may be performed using an original captured image, similar to high-precision product insertion / removal recognition.

[0276] In standard accuracy barcode reading, a barcode is read using a single captured image (the first captured image in which the entire barcode is captured), whereas in high accuracy barcode reading, a barcode may be read using multiple captured images (two or more captured images in which the entire barcode is captured). For example, in standard accuracy barcode reading, a barcode is read from an individual through image, whereas in high accuracy barcode reading, a barcode may be read using two or more stored captured images (for example, the original captured image).

[0277] Furthermore, in high-precision barcode reading, a captured image (for example, an original captured image) is subjected to preprocessing suitable for reading barcodes, and then the barcode is read using the preprocessed image, whereas in standard-precision barcode reading, a barcode may be read without the above-mentioned preprocessing. Preprocessing is also performed in standard-precision barcode reading, but it may be simpler (simpler) than the preprocessing performed in high-precision barcode reading.

[0278] In standard accuracy barcode reading, no retry is performed if the reading is unsuccessful, but in high accuracy barcode reading, a retry may be performed if the reading is unsuccessful. Standard accuracy barcode reading also retries, but the upper limit of the number of retries may be lower than in high accuracy barcode reading. In addition, the logic (specifications) for recognizing objects (barcodes) may be different between standard accuracy barcode reading and high accuracy barcode reading.

[0279] Standard accuracy barcode reading (standard accuracy barcode reading function) may be realized without using a trained model or may be realized using a trained model, while high accuracy barcode reading (high accuracy barcode reading function) may be realized without using a trained model or may be realized using a trained model.

[0280] When standard-accuracy barcode reading is achieved using a trained model, the trained model for standard-accuracy barcode reading may be generated by a trained model generation device and provided to the mobile terminal 60 as part of app A (as one function of app A). ​​When high-accuracy barcode reading is achieved using a trained model, the trained model for high-accuracy barcode reading may be generated by the trained model generation device as part of program B (as a function of program B) or as a separate program on the cloud server 20 side that cooperates with program B.

[0281] (Storage of barcode reading results) The barcode reading results by mobile terminal 60 (reading results by standard accuracy barcode reading) and the barcode reading results by cloud server 20 (reading results by high accuracy barcode reading) are stored in sales system 1 (for example, cloud server 20). For example, mobile terminal 60 transmits the barcode recognition results by mobile terminal 60 to cloud server 20, and cloud server 20 stores the barcode reading results by mobile terminal 60 and the barcode reading results by cloud server 20. As with the recognition results of product insertion, etc., the barcode reading results by mobile terminal 60 and the barcode reading results by cloud server 20 for a certain object are stored in a comparable linked manner.

[0282] In addition, when the mobile terminal 60 reads a barcode (when a product code is acquired), it may send the product code to the cloud server 20 as the reading result (for example, send the product code to the cloud server 20 together with the captured image), and when the mobile terminal 60 does not read a barcode (when a product code is not acquired), it may send information to the cloud server 20 that the product code was not read as the reading result (for example, send information to the cloud server 20 that the product code was not read together with the captured image).

[0283] (If the barcode reading result is reversed) The barcode reading result by the mobile terminal 60 (reading result by standard accuracy barcode reading) may differ from the barcode reading result by the cloud server 20 (reading result by high accuracy barcode reading). When the barcode reading result by the mobile terminal 60 differs from the barcode reading result by the cloud server 20, the barcode reading result by the cloud server 20 takes precedence. Therefore, for example, there are cases where a product that was not registered because the barcode was not successfully read by the mobile terminal 60 is successfully read by the cloud server 20 and registered; cases where a product that was registered by the mobile terminal 60 due to an incorrect barcode reading is accurately read by the cloud server 20 and corrected (amended) to the original product; cases where a product that became a non-file (a product whose product code is not registered in the product master) by the mobile terminal 60 is registered by the cloud server 20; and cases where a product that was registered by the mobile terminal 60 becomes a non-file product by the cloud server 20.

[0284] For example, if the barcode of a product (loaf of bread) that was not registered because the barcode could not be read successfully on the mobile terminal 60 is successfully read on the cloud server 20 and registered, the settlement device 50 will display a product list screen as shown in Figure 8 (E) after reading the settlement code, and will also display a small screen SG103 in front of the product list screen.

[0285] The small screen SG103 displays a message indicating that an unregistered product has been registered (in the illustrated example, "'Bread' has been registered."). The checkout device 50 may output a sound in addition to displaying the small screen SG103. That is, the checkout device 50 may notify the customer that an unregistered product has been registered by displaying and sounding the message. The small screen SG103 is erased, for example, after a predetermined time (e.g., 3 seconds) has elapsed. The product list screen on the back displays the registration information for bread as the first item.

[0286] In the scene of Fig. 8(E), information indicating that the product code was not read as the reading result of standard-accuracy barcode reading for one product item, and the product code of bread is stored as the reading result of high-accuracy barcode reading, are stored in cloud server 20. Based on the registration information and the above information stored in cloud server 20, settlement device 50 displays the settlement list screen and small screen SG103 of Fig. 8(E).

[0287] As described above, the product list screen in Figure 8 (E) (product list screen on the back) displays the registration information for one item (bread), but in addition to the registration information (i.e., for the product), it may also display information showing the results of reading using standard-precision barcode reading (e.g., "Terminal (unregistered)") or information showing the results of reading using high-precision barcode reading (e.g., "Server (bread)").

[0288] Instead of or in addition to the above information, the checkout device 50 may display information regarding one item of product, such as information indicating that the barcode reading result "unregistered" by the mobile terminal 60 has been overwritten by "bread" on the cloud server 20 (for example, "terminal (unregistered) → server (bread)").

[0289] In addition, the settlement device 50 may display information showing the reading results from standard accuracy barcode reading and the reading results from high accuracy barcode reading (for example, the above-mentioned "Terminal (unregistered)", the above-mentioned "Server (bread)", and the below-mentioned "Terminal (steamed bun)"), or information that the reading results of the barcode from standard accuracy barcode reading have been overturned by high accuracy barcode reading, in a display format different from the display format of "age verification product" and "medicine requiring explanation".

[0290] The settlement device 50 may display information regarding the results of the judgment on fraud, etc. (information indicating the results of the judgment on fraud, etc. based on standard accuracy fraud monitoring, information indicating the results of the judgment on fraud, etc. based on high accuracy fraud monitoring, information that the results of the judgment on fraud, etc. based on standard accuracy fraud monitoring have been maintained in high accuracy fraud monitoring, information that the results of the judgment on fraud, etc. based on standard accuracy fraud monitoring have been overturned in high accuracy fraud monitoring) and information regarding the results of barcode reading (reading results based on standard accuracy barcode reading, information indicating the reading results based on high accuracy barcode reading, information that the results of the barcode reading based on standard accuracy barcode reading have been overturned in high accuracy barcode reading) in different display formats.

[0291] For example, if a product (steamed bread) was registered due to a misreading of the barcode (bread barcode) on the mobile terminal 60, and the barcode is accurately read on the cloud server 20 and corrected to the original product (bread), the settlement device 50 will, after reading the settlement code, display a product list screen as shown in Figure 8 (F), and display a small screen SG104 in front of the product list screen.

[0292] The small screen SG104 displays a message indicating that the product registration has been corrected (in the illustrated example, "steamed bun" has been corrected to "white bread"). The checkout device 50 may output a sound in addition to displaying the small screen SG104. That is, the checkout device 50 may notify the customer that the product registration has been corrected by displaying and sounding the message. The small screen SG104 is erased, for example, after a predetermined time (e.g., 3 seconds) has elapsed. On the product list screen on the back, the registration information for steamed bun, which was registered as the first item due to a misreading, has been cancelled, and the registration information for white bread is displayed.

[0293] In the scene of Figure 8(F), the product code of the steamed bun is stored as the reading result of the standard accuracy barcode reading for one item, and the product code of the loaf of bread is stored as the reading result of the high accuracy barcode reading for the first item in the cloud server 20. The settlement device 50 displays the settlement list screen and small screen SG104 of Figure 8(F) based on the registration information stored in the cloud server 20 and the above information.

[0294] Note that the product list screen in Figure 8 (F) (product list screen on the back) displays information that the steamed bread has been cancelled and the registration information for the product (bread) as the registration information for the first item of product (bread), but in addition to this information, information showing the results of reading the barcode using standard accuracy (for example, "terminal (steamed bread)") or information showing the results of reading the barcode using high accuracy (for example, "server (bread)") may also be displayed. For example, "terminal (steamed bread)" may be displayed in association with the information that the steamed bread has been cancelled, and "server (bread)" may be displayed in association with the registration information for the product (bread).

[0295] Instead of or in addition to the above information, the settlement device 50 may display information regarding one item of product, such as information indicating that the barcode read by the mobile terminal 60, "steamed bread", has been covered by "white bread" on the cloud server 20 (for example, "terminal (steamed bread) → server (white bread)").

[0296] Although not shown in the figure, the same applies to the case where a product that has become a non-file on the mobile terminal 60 is registered on the cloud server 20, or the case where a product registered on the mobile terminal 60 becomes a non-file product on the cloud server 20 at the settlement device 50.

[0297] (Notification of barcode reading results by device used by store clerk) A device used by a store clerk (for example, a transaction status management device 40 or a mobile terminal for the store clerk (not shown)) may access the cloud server 20 and display on a screen the barcode reading results (reading results by standard accuracy barcode reading) by the mobile terminal 60 or the barcode reading results (reading results by high accuracy barcode reading) by the cloud server 20, which are stored in the cloud server 20. The device used by the store clerk may output audio instead of or in addition to the screen display. The same applies to the information below (such as information that the barcode reading results by the mobile terminal 60 have been overwritten by the cloud server 20).

[0298] In addition, the cloud server 20 also stores the captured image (original captured image) used by the cloud server 20 to perform high-precision barcode reading, so the device used by the store clerk displays the reading results of the high-precision barcode reading along with the captured image, allowing the store clerk to check whether the barcode reading results by the cloud server 20 are correct.

[0299] The device used by the store clerk may display information that the barcode reading result by the mobile terminal 60 has been overwritten on the cloud server 20, based on the above information stored in the cloud server 20 (the barcode reading result by the mobile terminal 60, the barcode reading result by the cloud server 20). For example, the device used by the store clerk may display information that the barcode reading result by the mobile terminal 60, "unregistered", has been overwritten on the cloud server 20 as "bread" (for example, "terminal (unregistered) → server (bread)"), as information about the first item of product in FIG. 8(E). Also, for example, the device used by the store clerk may display information that the barcode reading result by the mobile terminal 60, "steamed bread", has been overwritten on the cloud server 20 as "bread" (for example, "terminal (steamed bread) → server (bread)"), as information about the first item of product in FIG. 8(F).

[0300] The device used by the store clerk may display information that the barcode reading result by the mobile terminal 60 has been maintained in the cloud server 20, based on the above information stored in the cloud server 20 (the barcode reading result by the mobile terminal 60, the barcode reading result by the cloud server 20). For example, the device used by the store clerk may display information that the barcode reading result by the mobile terminal 60, "bread", has been maintained in the cloud server 20 (for example, "bread (maintained)"), as information about the first item of merchandise in FIG. 8(A).

[0301] In addition, the device used by the store clerk may display the barcode reading results by the mobile terminal 60, the barcode reading results by the cloud server 20, information that the barcode reading results by the mobile terminal 60 have been overwritten in the cloud server 20, and information that the barcode reading results by the mobile terminal 60 have been maintained in the cloud server 20 in a display format different from the display format for ``age verification products'' and ``medicines requiring explanation.''

[0302] The device used by the store clerk may display information regarding the determination result of fraud, etc. and information regarding the barcode reading result in different display formats.

[0303] (Notification by the sign pole of the settlement device 50) The settlement device 50 may illuminate the pole based on the determination result of fraud, etc. For example, the settlement device 50 may change the illumination mode of the pole depending on the determination result of fraud, etc. For example, the settlement device 50 illuminates the pole (e.g., the first light-emitting unit) in a first color (e.g., yellow) when both the mobile terminal 60 (standard-accuracy fraud monitoring) and the cloud server 20 (high-accuracy fraud monitoring) determine that fraud, etc. has occurred. However, when the determination of the mobile terminal 60 that the action did not constitute fraud, etc. is overturned by the cloud server 20 that the action constitutes fraud, etc., the settlement device 50 illuminates the pole (e.g., the first light-emitting unit) in a second color (e.g., red). In other words, even if both cases are ultimately determined to constitute fraud, etc., the settlement device 50 illuminates the pole in different illumination modes depending on whether the mobile terminal 60 determined that the action constituted fraud, etc. When asked by a customer of the settlement device 50 illuminating in the second color (a customer who has not notified the mobile terminal 60 about the possibility of calling a store clerk, etc.), the store clerk can check the light emission pattern of the pole (second color) and immediately inform the customer of the situation (store clerk confirmation is no longer necessary → it is now necessary). Furthermore, if the determination made by the mobile terminal 60 that the act constituted fraud, etc. is overturned by the cloud server 20 that the act constituted fraud, etc., the settlement device 50 may illuminate the pole (e.g., the first light-emitting unit) in a third color (e.g., blue). When asked by a customer of the settlement device 50 illuminating in the third color (a customer who has not notified the mobile terminal 60 about the possibility of calling a store clerk, etc.), the store clerk can check the light emission pattern of the pole (third color) and immediately inform the customer of the situation (store clerk confirmation was previously necessary → it is no longer necessary).

[0304] The settlement device 50 also illuminates the sign pole when, for example, the change is near end, the receipt is near end, a store clerk is called using the store clerk call button, there are products that require age verification, or there are medicines that require explanation. When the settlement device 50 illuminates the sign pole based on the determination result of fraud or the like, it illuminates the sign pole in a different illumination mode than when the change is near end, the receipt is near end, etc.

[0305] The settlement device 50 may illuminate the sign pole based on the result of reading the barcode. For example, the settlement device 50 may change the illumination mode of the sign pole depending on the result of reading the barcode. For example, the settlement device 50 may illuminate the sign pole (e.g., the second light-emitting unit) in a first color when a product that was not registered on the mobile terminal 60 (standard accuracy barcode reading) is registered on the cloud server 20 (high accuracy barcode reading) (when a product that was not registered because the barcode was not successfully read on the mobile terminal 60 is registered because the barcode is successfully read on the cloud server 20, or when a product that was not filed because the barcode was misread on the mobile terminal 60 is correctly read on the cloud server 20 and registered), illuminate the sign pole (e.g., the second light-emitting unit) in a second color when a product that was incorrectly registered on the mobile terminal 60 (standard accuracy barcode reading) is corrected to the correct product on the cloud server 20 (when a product that was incorrectly registered because the barcode was misread on the mobile terminal 60 is correctly read on the cloud server 20 and registered as the original product), and illuminate the sign pole (e.g., the second light-emitting unit) in a third color when the product is not filed on both the mobile terminal 60 and the cloud server 20. This allows the store staff to check the lighting state of the sign pole and immediately report the respective situation.

[0306] In addition, when the settlement device 50 illuminates the sign pole based on the results of reading the barcode, it illuminates the sign pole in a different illumination mode than when it illuminates the sign pole based on the results of a judgment such as fraud, or when change is near end or receipt is near end, etc.

[0307] (Updating a trained model) When using a trained model for product input / output recognition (product input / output recognition function), the trained model may be updated as appropriate using the results of actual operation. In this case, training data may be created by a human being based on the captured images. The same applies when using a trained model for barcode reading (barcode reading function).

[0308] The embodiment has been described above. According to the embodiment, product registration can be suitably performed using a mobile terminal.

[0309] For example, sales system 1 has standard-accuracy product insertion / removal recognition that is excellent in speed (response) and high-accuracy product insertion / removal recognition that is excellent in accuracy (reliability) for recognizing the insertion and removal of products into a shopping basket, and uses the recognition results from standard-accuracy product insertion / removal recognition to monitor fraud, etc. in real time (standard-accuracy fraud monitoring), and also uses the recognition results from high-accuracy product insertion / removal recognition to accurately monitor fraud, etc. (high-accuracy fraud monitoring).This allows the sales floor to recognize fraud, etc. in real time and respond quickly and easily (notify the customer on a small screen, etc.), and to accurately recognize the occurrence of fraud, etc. and respond appropriately (call a store clerk, etc.) before the checkout is completed.

[0310] Furthermore, in the sales system 1, if the judgment result obtained by standard-accuracy fraud monitoring (the recognition result obtained by standard-accuracy product insertion / removal recognition) differs from the judgment result obtained by high-accuracy fraud monitoring (the recognition result obtained by high-accuracy product insertion / removal recognition), the judgment result obtained by high-accuracy fraud monitoring takes precedence, and the judgment result obtained by standard-accuracy fraud monitoring is overturned by the high-accuracy fraud monitoring. However, if the judgment result is overturned, an appropriate announcement is made by the settlement device 50 (FIGS. 8(B) to 8(D)), which makes it less likely for customers to become confused and easier for customers to understand. Furthermore, the store clerk can confirm that the judgment result has been overturned by the illumination status of the sign pole of the settlement device 50 or on the screen of the device used by the store clerk (such as the transaction status management device 40 or the store clerk's mobile terminal), so the store clerk can easily grasp the situation and, if necessary, can smoothly explain the situation to the customer in question.

[0311] As described above, if the determination results from standard-accuracy fraud monitoring differ from those from high-accuracy fraud monitoring, the determination results from standard-accuracy fraud monitoring are overturned by high-accuracy fraud monitoring. In other words, standard-accuracy fraud monitoring, which can be overturned by high-accuracy fraud monitoring, is customer support monitoring aimed at providing support (guidance display, warnings) for operations (registration operations, cancellation operations, etc.) and actions (item insertion, item removal) to customers (customers using mobile terminals 60 or carts 70) in the sales floor, while high-accuracy fraud monitoring, which excels in accuracy, can also be considered fraud detection monitoring aimed at detecting actual fraud, etc. Furthermore, the standard-accuracy product insertion / removal recognition used in standard-accuracy fraud monitoring (customer support monitoring) can also be considered customer support product insertion / removal recognition, and the high-accuracy product insertion / removal recognition used in high-accuracy fraud monitoring (fraud detection monitoring) can also be considered fraud detection product insertion / removal recognition.

[0312] In other words, sales system 1 has a customer support product insertion / removal recognition system that is excellent in speed and a fraud detection product insertion / removal recognition system that is excellent in accuracy, and uses the recognition results from the customer support product insertion / removal recognition system to support customers in real time (customer support monitoring), and also uses the recognition results from the fraud detection product insertion / removal recognition system to monitor fraud, etc. (fraud detection monitoring). This allows the sales floor to support customers in real time and respond appropriately to fraud, etc. Note that it is preferable to implement both customer support monitoring (standard-accuracy fraud monitoring) and fraud detection monitoring (high-accuracy fraud monitoring), but it is also possible to implement either one.

[0313] Furthermore, sales system 1 has standard-precision barcode reading, which is excellent in speed (response), and high-precision barcode reading, which is excellent in accuracy (reliability), for reading barcodes during scan registration, and performs barcode reading quickly using standard-precision barcode reading, and then performs even higher-precision barcode reading after the rapid barcode reading. This reduces stress at the sales floor when reading, and allows the barcode to be read accurately and appropriate action to be taken (such as correcting the price to the original value) before payment is completed.

[0314] (Variations, etc.) Although each embodiment has been described above, the device configuration, data configuration, processing flow, display and output modes, etc. can be appropriately changed as shown in the following (1) to (17). In addition, the following (1) to (17) can be appropriately combined with each other.

[0315] (1) In the above embodiment, an example was described in which mobile terminal 60 transmits an acquired product code to cloud server 20 each time it acquires a product code ( FIG. 3 ). However, one or more already acquired product codes may be transmitted together to cloud server 20 after operating checkout instruction button BT631. In the case where mobile terminal 60 transmits one or more product codes together to cloud server 20 after operating checkout instruction button BT631, high-precision product insertion / removal recognition (high-precision fraud monitoring as well) may be performed during the period shown in FIG. 7(A) or 7(B). However, captured images and operation information may be transmitted in advance before operating checkout instruction button BT631. In the case where captured images and operation information are transmitted in advance, high-precision product insertion / removal recognition (high-precision fraud monitoring as well) may be performed during the period shown in FIG. 7(C) in addition to the period shown in FIG. 7(A) or 7(B).

[0316] (2) In the above embodiment, an example was described in which the settlement device 50 displayed a product list screen as the first screen after reading the settlement code (FIGS. 3 and 8(A)). However, the first screen is not limited to the product list screen. For example, the settlement device 50 may display a payment method selection screen as the first screen after reading the settlement code. In other words, the settlement device 50 may display each of the small screens in FIGS. 8(C) to 8(F) in front of the payment method selection screen. Note that a list display button for displaying the product list screen may be provided on the payment method selection screen.

[0317] (3) In the above embodiment, an example was described in which a notification was given that the result of the judgment of fraud, etc. made by the mobile terminal 60 has been overturned by the cloud server 20 (that the judgment of fraud, etc. made by standard-accuracy fraud monitoring has been overturned by high-accuracy fraud monitoring) (Figures 8(C) and 8(D)). However, instead of or in addition to the above, a notification may be given that the recognition result of product insertion, etc. made by the mobile terminal 60 has been overturned by the cloud server 20 (that the recognition made by standard-accuracy product insertion / removal recognition has been overturned by high-accuracy product insertion / removal recognition).

[0318] In addition, the fact that the judgment result of the mobile terminal 60 regarding fraud, etc. has been overturned by the cloud server 20, or the fact that the recognition result of the mobile terminal 60 regarding the insertion of goods, etc. has been overturned by the cloud server 20, may be notified by the settlement device 50, or may be notified by a device used by a store clerk instead of the settlement device 50 (transaction status management device 40, mobile terminal for store clerk, etc.).

[0319] (4) In the above embodiment, the settlement device 50 (as well as devices used by store clerks) displays information (e.g., "Terminal OK -> Server NG (Unregistered Insert)") that the determination result of fraud, etc. by standard-accuracy fraud monitoring has been overturned by high-accuracy fraud monitoring, based on the determination result (e.g., "Terminal OK") by standard-accuracy fraud monitoring and the determination result (e.g., Server NG (Unregistered Insert)) stored by the cloud server 20. However, the cloud server 20 may store the information that indicates the overturn, and the settlement device 50 (as well as devices used by store clerks) may display the information that indicates the overturn. This is related to (3) above, but the same applies to information that indicates that the recognition result by standard-accuracy product insertion / removal recognition has been overturned by high-accuracy product insertion / removal recognition.

[0320] (5) In the above embodiment, an example was described in which the settlement device 50 (as well as the device used by the store clerk) displays information ("Terminal (Steamed Bread) → Server (Bread)") that the reading result from standard accuracy barcode reading has been overturned by high accuracy barcode reading, based on the reading result from standard accuracy barcode reading (e.g., "Terminal (Steamed Bread)") and the reading result from high accuracy barcode reading (e.g., "Server (Bread)") stored in the cloud server 20. However, the cloud server 20 may store information that the reading result from standard accuracy barcode reading has been overturned by high accuracy barcode reading, and the settlement device 50 (as well as the device used by the store clerk) may display information that the reading result from standard accuracy barcode reading stored in the cloud server 20 has been overturned by high accuracy barcode reading.

[0321] (6) In the above embodiment, it was explained that the settlement device 50 (as well as the device used by the store clerk) may display information regarding the judgment results regarding fraud, etc., information regarding the barcode reading results, and ``age verification product'' or ``medicine requiring explanation'' in different display formats, but it may also display that each applies (display each applicable item in a distinguishable manner).

[0322] For example, if the mobile terminal 60 recognizes the product input of a registered beer but the cloud server 20 does not recognize the product input, the message "age verification product" and "terminal OK → server NG (unregistered input)" may be displayed for that beer. If the mobile terminal 60 does not recognize the product input of a registered beer but the cloud server 20 recognizes the product input, the message "age verification product" and "terminal NG (unregistered input) → server OK" may be displayed for that beer. If the product input of a registered beer is not recognized by either the mobile terminal 60 or the cloud server 20, the message "age verification product" and "terminal NG (unregistered input) → server NG (unregistered input)" may be displayed for that beer.

[0323] If the mobile terminal 60 recognizes that a canceled beer has been taken out but the cloud server 20 does not, the message "Age verification product" and "Terminal OK → Server NG (cancelled but not taken out)" may be displayed for the beer.

[0324] If the barcode of a beer is misread as orange juice by mobile terminal 60 but is read correctly by cloud server 20, the message "age verification product" and "terminal (orange juice) → server (beer)" may be displayed for the beer. If mobile terminal 60 recognizes the product insertion for the beer that was misread as orange juice, and cloud server 20 correctly reads it as beer but does not recognize the product insertion, the message "age verification product," "terminal (orange juice) → server (beer)," and "terminal OK → server NG (unregistered insertion)" may be displayed for the beer.

[0325] In addition, each display such as "Age verification product", "Terminal OK → Server NG (unregistered input)", "Terminal NG (unregistered input) → Server NG (unregistered input)", "Terminal NG (unregistered input) → Server NG (unregistered input)", "Terminal OK → Server NG (cancelled, not removed)", etc. may be displayed using a mark, symbol, icon, etc. that indicates each item.

[0326] (7) In the above embodiment, an example has been described in which the settlement device 50 acquires (reads) information (transaction identification information, etc.) for acquiring transaction information for the mobile terminal 60 from the settlement code displayed by the mobile terminal 60, and then acquires (receives) the transaction information for the mobile terminal 60 from the cloud server 20 using the acquired information. However, the method by which the settlement device 50 acquires transaction information is not limited to this. For example, the settlement device 50 may acquire transaction information using short-range wireless communication. As one example, the settlement device 50 may acquire (receive) information for acquiring transaction information for the mobile terminal 60 from the mobile terminal 60 via short-range wireless communication, and then acquire (receive) the transaction information for the mobile terminal 60 from the cloud server 20 using the acquired information. As another example, the settlement device 50 may acquire (receive) transaction information for the mobile terminal 60 (transaction information stored in the mobile terminal 60) from the mobile terminal 60 via short-range wireless communication.

[0327] In a mode in which the settlement device 50 acquires transaction information of the mobile terminal 60 from the mobile terminal 60 via short-range wireless communication, the settlement device 50 separately acquires the recognition results of high-precision product insertion / removal recognition and the monitoring results of high-precision fraud monitoring for the mobile terminal 60 from the cloud server 20 (except when the settlement device 50 itself performs high-precision product insertion / removal recognition or high-precision fraud monitoring; see (8) below). Note that the settlement device 50 may first acquire information for acquiring transaction information of the mobile terminal 60 from the mobile terminal 60 via short-range wireless communication, acquire the recognition results of high-precision product insertion / removal recognition and the judgment results (monitoring results) of high-precision fraud monitoring for the mobile terminal 60 from the cloud server 20, and then acquire the transaction information from the mobile terminal 60 via short-range wireless communication.

[0328] (8) In the above embodiment, an example was described in which the cloud server 20 performed high-precision product insertion / removal recognition and high-precision fraud monitoring, but other devices may also perform high-precision product insertion / removal recognition and high-precision fraud monitoring.

[0329] For example, the sales system 1 may include a recognition server (not shown) separate from the cloud server 20, which performs high-precision product insertion / removal recognition and high-precision fraud monitoring. The period during which the recognition server performs high-precision product insertion / removal recognition and high-precision fraud monitoring may be the same as the period described above in which the cloud server 20 performs high-precision product insertion / removal recognition and high-precision fraud monitoring (any of the periods shown in FIGS. 7(A) to 7(C)). The recognition server may acquire information (captured images, operation information) necessary for high-precision product insertion / removal recognition and high-precision fraud monitoring from the cloud server 20 or from the mobile terminal 60.

[0330] In addition, the cloud server 20 may perform high-precision product insertion / removal recognition, and the recognition server may perform high-precision fraud monitoring using the recognition results from the high-precision product insertion / removal recognition by the cloud server 20, or the recognition server may perform high-precision product insertion / removal recognition, and the cloud server 20 may perform high-precision fraud monitoring using the recognition results from the high-precision product insertion / removal recognition by the recognition server.

[0331] Furthermore, the mobile terminal 60 may perform high-accuracy product insertion / removal recognition and high-accuracy fraud monitoring in addition to standard-accuracy product insertion / removal recognition and standard-accuracy fraud monitoring. The period during which the mobile terminal 60 performs high-accuracy product insertion / removal recognition and high-accuracy fraud monitoring may be the same as the period described as the period during which the cloud server 20 performs high-accuracy product insertion / removal recognition and high-accuracy fraud monitoring (any of the periods shown in FIGS. 7(A) to 7(C)).

[0332] As described above, in standard-accuracy product insertion / removal recognition, product insertion, etc. is recognized using a preprocessed captured image obtained by performing preprocessing (processing to reduce the amount of information) on a captured image (original captured image) acquired from the imaging unit, whereas in high-accuracy product insertion / removal recognition, product insertion, etc. is recognized using a captured image acquired from the imaging unit (original captured image that has not been preprocessed). Also, in the above-described mode in which the mobile terminal 60 executes standard-accuracy product insertion / removal recognition and high-accuracy product insertion / removal recognition (for example, a mode in which the mobile terminal 60 executes standard-accuracy product insertion / removal recognition and high-accuracy product insertion / removal recognition during the period shown in FIG. 7(C)), the mobile terminal 60 can start high-accuracy product insertion / removal recognition using the N frames of captured images (original captured images) acquired from the imaging unit at the same time (substantially simultaneously) as it starts preprocessing for standard-accuracy product insertion / removal recognition using the N frames of captured images. In addition, in the case where the mobile terminal 60 performs standard accuracy product insertion / removal recognition and the cloud server 20 performs high accuracy product insertion / removal recognition, even if the mobile terminal 60 performs standard accuracy product insertion / removal recognition and the cloud server 20 performs high accuracy product insertion / removal recognition during the period shown in Figure 7 (C), it is difficult to start at the same time (approximately simultaneously) because the captured image (original captured image) must be sent from the mobile terminal 60 to the cloud server 20.

[0333] If the mobile terminal 60 starts high-precision product insertion / removal recognition using the N frames of captured images at the same time (almost simultaneously) as it starts pre-processing for standard-precision product insertion / removal recognition using the N frames of captured images acquired from the imaging unit, the high-precision product insertion / removal recognition, which does not require pre-processing, will start before the standard-precision product insertion / removal recognition, which does require pre-processing. However, because the standard-precision product insertion / removal recognition is faster than the high-precision product insertion / removal recognition, the processing load of the pre-processing is smaller than the difference in processing load between the standard-precision product insertion / removal recognition and the high-precision product insertion / removal recognition. In other words, in at least an aspect in which standard-accuracy product insertion / removal recognition and high-accuracy product insertion / removal recognition are performed on the same device, the total time required for the preprocessing and the standard-accuracy product insertion / removal recognition is set to be shorter than the time required for high-accuracy product insertion / removal recognition, so that standard-accuracy product insertion / removal recognition is superior in terms of speed to high-accuracy product insertion / removal recognition (Note that in an aspect in which standard-accuracy product insertion / removal recognition and high-accuracy product insertion / removal recognition are performed on different devices as shown in the above embodiment, the total time required for the preprocessing and the standard-accuracy product insertion / removal recognition may be set to be shorter than the total time required for sending and receiving captured images between the two devices and the high-accuracy product insertion / removal recognition.) From the above, even if high-accuracy product insertion / removal recognition, which does not require preprocessing, is started before standard-accuracy product insertion / removal recognition, which does require preprocessing, the timing at which the recognition result from the high-accuracy product insertion / removal recognition is obtained will be later than the timing at which the recognition result from the standard-accuracy product insertion / removal recognition is obtained.

[0334] Note that the pre-processing performed before the standard-accuracy product insertion / removal recognition may be considered as part of the standard-accuracy product insertion / removal recognition. In this case, the standard-accuracy product insertion / removal recognition (including the pre-processing) and the high-accuracy product insertion / removal recognition may be started simultaneously (or approximately simultaneously), but in either case, the timing at which the recognition result from the high-accuracy product insertion / removal recognition is obtained will be later than the timing at which the recognition result from the standard-accuracy product insertion / removal recognition is obtained.

[0335] Furthermore, the settlement device 50 may perform high-precision product insertion / removal recognition and high-precision fraud monitoring. After reading the settlement code, the settlement device 50 may obtain, in addition to transaction information, information required for high-precision product insertion / removal recognition and high-precision fraud monitoring (captured images, operation information) from the cloud server 20, and perform high-precision product insertion / removal recognition before displaying the first screen (product list screen, payment method selection screen) after receiving the transaction information. In other words, the settlement device 50 may display the first screen after performing high-precision product insertion / removal recognition and high-precision fraud monitoring.

[0336] (9) In the above embodiment, an example was described in which the cloud server 20 performed high-precision barcode reading. However, similar to the high-precision product insertion / removal recognition and high-precision fraud monitoring described in (8) above, other devices (e.g., a recognition server, a mobile terminal 60, or a settlement device 50) may also perform high-precision barcode reading.

[0337] As an example, pre-processing is required for high-precision barcode reading rather than standard-precision barcode reading, so in a configuration in which the portable terminal 60 performs both standard-precision barcode reading and high-precision barcode reading, the timing at which the reading results from high-precision barcode reading are obtained will naturally be later than the timing at which the reading results from standard-precision barcode reading are obtained.

[0338] Note that the pre-processing performed before high-accuracy barcode reading may be considered as part of high-accuracy barcode reading. In this case, high-accuracy barcode reading (including pre-processing) and standard-accuracy barcode reading may start simultaneously (or approximately simultaneously), but in either case, the timing at which the reading result from high-accuracy barcode reading is obtained will be later than the timing at which the reading result from standard-accuracy barcode reading is obtained.

[0339] (10) In the above embodiment, an example was described in which the mobile terminal 60 performs standard-accuracy product insertion / removal recognition and standard-accuracy fraud monitoring, but other devices may also perform standard-accuracy product insertion / removal recognition and standard-accuracy fraud monitoring.

[0340] For example, the cloud server 20 (or the recognition server) may perform standard-accuracy product insertion / removal recognition and standard-accuracy fraud monitoring in addition to high-accuracy product insertion / removal recognition and high-accuracy fraud monitoring. The period during which the cloud server 20 (or the recognition server) performs standard-accuracy product insertion / removal recognition and standard-accuracy fraud monitoring is the same as the period described above in which the mobile terminal 60 performs standard-accuracy product insertion / removal recognition and standard-accuracy fraud monitoring (the period shown in FIGS. 7(A) to 7(C)).

[0341] In addition, similar to the mode in which the mobile terminal 60 performs standard-accuracy product insertion / removal recognition and high-accuracy product insertion / removal recognition (above (8)), even in the mode in which the cloud server 20 (or the recognition server) performs standard-accuracy product insertion / removal recognition and high-accuracy product insertion / removal recognition, the timing at which the recognition result from high-accuracy product insertion / removal recognition is obtained is later than the timing at which the recognition result from standard-accuracy product insertion / removal recognition is obtained.

[0342] (11) In the above embodiment, an example was described in which the mobile terminal 60 performed standard-accuracy barcode reading. However, similar to the standard-accuracy product insertion / removal recognition and standard-accuracy fraud monitoring described in (10) above, other devices (e.g., cloud server 20, recognition server) may also perform standard-accuracy barcode reading.

[0343] In addition, similar to the mode in which the mobile terminal 60 performs standard accuracy barcode reading and high accuracy barcode reading (above (9)), in the mode in which the cloud server 20 (or the recognition server) performs standard accuracy barcode reading and high accuracy barcode reading, the timing at which the reading result from high accuracy barcode reading is obtained will be later than the timing at which the reading result from standard accuracy barcode reading is obtained.

[0344] (12) In the above embodiment, an example was described in which a product insertion determination model (standard-accuracy product insertion determination model, standard-accuracy product removal determination model) and a product removal determination model (standard-accuracy product removal determination model) were individually generated and used, but a single determination model that combines both functions may be generated and used. In other words, in the above embodiment, an example was described in which a trained model for determining product insertion and a trained model for determining product removal were separately prepared, but a single trained model that can determine product insertion and product removal may be provided.

[0345] (13) In the above embodiment, the amount of information (such as captured images) used for high-precision product insertion / removal recognition is greater than the amount of information (such as captured images) used for standard-precision product insertion / removal recognition. However, the types of information used for high-precision product insertion / removal recognition may be greater than the types of information used for standard-precision product insertion / removal recognition. For example, standard-precision product insertion / removal recognition may use one type of information (such as captured images), while high-precision product insertion / removal recognition may use two types of information (such as captured images and weighing values ​​from a weighing unit). High-precision product insertion / removal recognition may use captured images with the same amount of information as the captured images used for standard-precision product insertion / removal recognition. If captured images with the same amount of information as the captured images used for standard-precision product insertion / removal recognition are used, the recognition results from standard-precision product insertion / removal recognition may be used as high-precision product insertion / removal recognition using captured images. High-precision product insertion / removal recognition determines whether its own recognition results match those of standard-precision product insertion / removal recognition.

[0346] (14) In the above embodiment, an example was described in which the settlement device 50 notifies the user of fraudulent items such as unregistered items (FIGS. 8(B) and 8(D)), but this notification may also be made on the mobile terminal 60. In other words, in the above embodiment, an example was described in which a store clerk intervenes at the settlement device 50 for transactions involving "unregistered items" (a called store clerk deals with fraudulent items), but a store clerk may also intervene on the mobile terminal 60. Since fraudulent items are resolved before the settlement device 50 is operated, the user can concentrate on settlement at the settlement device 50. For example, the following control may be used.

[0347] When a checkout instruction is given (checkout instruction button BT631 is operated) and there is no fraud, the mobile terminal 60 displays a settlement code; if there is fraud, the mobile terminal 60 displays a notice of fraud for the product on the registration screen (or a screen equivalent to the product list screen of the checkout device 50) and executes a store clerk call process (for example, sends a store clerk call notice to the terminal used by the store clerk (for example, sends via cloud server 20)). The mobile terminal 60 that displayed the presence of fraud displays the settlement code when the fraud is resolved with the intervention of a store clerk (for example, based on a specified operation after resolution). In other words, if there is no fraud, the mobile terminal 60 displays the settlement code based on the checkout instruction, and if there is fraud, the mobile terminal 60 displays a screen for store clerk intervention based on the checkout instruction, and displays the settlement code after the fraud is resolved with the intervention of a store clerk.

[0348] The determination of whether or not fraud has occurred is made after high-precision fraud monitoring has been performed. In other words, high-precision fraud monitoring should be performed before the settlement code is displayed (FIGS. 7(B) and 7(C)). Note that, in relation to (8) above, high-precision fraud monitoring may be performed by the cloud server 20 or the mobile terminal 60.

[0349] The case where fraud or the like is present may be a case where fraud or the like is found in at least one of the standard-accuracy fraud monitoring or the high-accuracy fraud monitoring. In other words, the three cases (a case where the standard-accuracy fraud monitoring was found to be OK but the high-accuracy fraud monitoring was found to be NG, a case where the standard-accuracy fraud monitoring was found to be NG but the high-accuracy fraud monitoring was found to be OK, and a case where the standard-accuracy fraud monitoring and the high-accuracy fraud monitoring were found to be NG) excluding the case where the standard-accuracy fraud monitoring was found to be NG but the high-accuracy fraud monitoring was found to be OK, and a case where the standard-accuracy fraud monitoring and the high-accuracy fraud monitoring were found to be NG) may be cases where fraud or the like is present (the case may not be subject to store clerk intervention). The same applies to (15) below.

[0350] In relation to (7) above, the same applies when short-range wireless communication is used instead of a settlement code. In other words, processing at the settlement device 50 is restricted until the fraud, etc. is resolved at the mobile terminal 60, and processing at the settlement device 50 is enabled after the fraud, etc. is resolved at the mobile terminal 60.

[0351] In the above explanation, an example was given in which a store clerk intervenes when there is fraud, etc., but a store clerk also intervenes at the mobile terminal 60 in the case of a barcode reading result or an "age verification product" or "medicine requiring explanation." That is, when a payment instruction is received, for example, in a case where a non-file was found in at least one of the standard accuracy barcode reading or the high accuracy barcode reading, a case where a misreading in the standard accuracy barcode reading is discovered by the high accuracy barcode reading, or a product requiring age verification, the mobile terminal 60 displays the relevant details of each of the relevant products on the relevant registration screen, etc., and executes a store clerk call process. Note that a case in which a non-file in the standard accuracy barcode reading is resolved by the high accuracy barcode reading may not require store clerk intervention. The same applies to (15) below.

[0352] (15) In the above embodiment, an example was described in which a product was registered on the mobile terminal 60 and settled at the settlement device 50, but settlement may also be performed at a device other than the settlement device 50. For example, the mobile terminal 60 may communicate with a settlement server to perform settlement (so-called over-the-air settlement).

[0353] Related to (14) above, before the payment server executes the payment, a store clerk may intervene regarding fraud, the barcode reading results, or "age verification products," and the payment may be executed by the payment server after the fraud or the like is resolved. For example, when a checkout instruction is given, if there is no fraud or age verification products, the mobile terminal 60 may proceed with the in-air payment based on the checkout instruction (for example, by displaying an operable execute button), and if there is fraud or age verification products, the mobile terminal 60 may display a screen for the store clerk to intervene based on the checkout instruction, and proceed with the in-air payment after the fraud or the age verification products are resolved through the store clerk's intervention (in-air payment may be restricted until the fraud or the age verification products are resolved).

[0354] (16) In the above embodiment, the mobile terminal 60 is described as being owned by the customer, but it may also be a terminal (rental terminal) prepared by the store. The mobile terminal 60 as a rental terminal may not have some (e.g., an audio input unit) of the components (e.g., a CPU, a memory unit, a display unit, an operation unit, an audio output unit, an audio input unit, an imaging unit, a detection unit, and a communication unit) that the mobile terminal 60 is described as having. The mobile terminal 60 as a rental terminal may also be prepared as a dedicated terminal in which only the app A can be executed. Furthermore, the mobile terminal 60 as a rental terminal may not be separate from the cart 70, but may be part of the cart 70 (the cart 70 may include the mobile terminal 60). In other words, the cart 70 may be a dedicated cart that also has the functions of the mobile terminal 60.

[0355] In the dedicated cart, some components of the mobile terminal 60 may be provided as separate units. In other words, in the dedicated cart, the mobile terminal 60 may be composed of multiple devices. The dedicated cart may be provided with a scanning unit (barcode reader) separate from the imaging unit (corresponding to the imaging unit of the mobile terminal 60). The imaging unit may be separate from the operation display unit (corresponding to the display unit of the mobile terminal 60). The scanning unit may be separate from the operation display unit.

[0356] In other words, the mobile terminal 60 may be any of those called information communication terminals, mobile communication terminals, mobile registration terminals, portable registration terminals, etc., and the main components may be recognizable in other devices as constituting one part of another device (e.g., a shopping cart, etc.) (may be configured in a unified form on other devices), or may be recognizable in other devices as constituting two or more parts of other devices (may be configured in a scattered form on other devices).

[0357] (17) Some or all of the functions (input / output, storage, processing (including judgment)) of each device (mobile terminal 60, cloud server 20, settlement device 50, cart 70, etc.) may be realized by a device other than the device described as the entity that executes the function. For example, each device may function as a so-called thin client specialized for input / output interfaces, etc. As an example, a cloud server (which may be the same as cloud server 20 or a different cloud server) may execute a program for controlling the operation of each device, and each device may display a screen using a browsing function. For example, the behavior of mobile terminal 60 obtained by executing app A may be realized by a program (referred to as program A) on the cloud server 20 side.

[0358] <Summary of the embodiment> [Technical field] The present invention relates to a product registration system. [Background technology] BACKGROUND ART A system is known in which customers themselves register products using mobile terminals (see, for example, Patent Document 1). [Prior art document] [Patent documents] [Patent Document 1] JP 2018-055700 A [Summary of the Invention] [Problem to be solved by the invention] However, the above-described system still has room for improvement in terms of measures against malicious or inexperienced customers. The present invention has been made in consideration of these circumstances, and aims to provide a technology for suitably registering products using a mobile terminal. [Means for solving the problem]

[0359] (1) One aspect for solving the above-mentioned problem is a product registration system (e.g., sales system 1) that registers products purchased by customers at a sales floor (e.g., scan registration, button registration), the system including: a registration unit that registers products to be purchased; an imaging unit that can capture images of products secured by customers as products to be purchased (e.g., products added to a shopping basket); a recognition unit (product insertion / removal recognition function) that recognizes specific actions by customers regarding securing products (e.g., adding products to a shopping basket, removing products from a shopping basket) based on the captured images acquired from the imaging unit; and a registration status (product registration status) by the registration unit and a previous and an alarm unit that notifies (see, for example, Figures 5(A) to 5(D), Figures 6(A) to 6(F), and Figures 8(B) to 8(D)) information based on the recognition status (status of product insertion, etc.) by the recognition unit (information based on the results of fraud, etc., determined by fraud monitoring based on the product registration status and the product insertion, etc.), wherein the recognition unit has a first recognition unit (for example, standard-accuracy product insertion / removal recognition using a standard-accuracy product insertion / removal recognition function) and a second recognition unit (for example, high-precision product insertion / removal recognition using a high-precision product insertion / removal recognition function) having higher recognition accuracy than the first recognition unit. According to the product registration system (1), product registration can be suitably performed using a mobile terminal. For example, at the sales floor, the results of a determination of fraud, etc., by standard-accuracy fraud monitoring based on the presence or absence of product registration and the recognition results of product insertion, etc., by standard-accuracy product insertion / removal recognition can be easily notified, and the results of a determination of fraud, etc., by high-accuracy fraud monitoring based on the presence or absence of product registration and the recognition results of product insertion, etc., by high-accuracy product insertion / removal recognition can be accurately notified before the checkout is completed. Therefore, at the sales floor, a quick response (notifying the customer on a small screen, etc.) can be made, and an accurate response (calling a store clerk, etc.) can be made before the checkout is completed.

[0360] In addition, the product registration system is equipped with a fraud monitoring unit (standard accuracy fraud monitoring unit, high accuracy fraud monitoring unit) that monitors fraud, etc. based on the registration status by the registration unit and the recognition status by the recognition unit, and the notification unit may notify the judgment results (monitoring results) by the fraud monitoring unit.

[0361] (2) One aspect for solving the above-mentioned problem is a product registration system for registering products purchased by customers at a sales floor, the system including: a registration unit for registering the products to be purchased; a quantity change / cancellation operation receiving unit for receiving an operation to change the quantity of the products to be purchased (quantity increase operation, quantity decrease operation) or an operation to cancel the purchase (product cancellation operation); an imaging unit for capturing an image of the product reserved by the customer as the product to be purchased; a recognition unit for recognizing specific behaviors related to the customer's reservation of the product based on the captured image acquired from the imaging unit; This product registration system is equipped with an alarm unit that notifies (see, for example, Figures 5(A) to 5(D), Figures 6(A) to 6(F), and Figures 8(B) to 8(D)) information based on the operation status (status of quantity increase, quantity decrease, product cancellation) by the cancellation operation receiving unit and the recognition status by the recognition unit (information based on the results of fraud, etc. determined by fraud monitoring based on the status of quantity increase, quantity decrease, product cancellation and the status of product input, etc.), and the recognition unit is characterized by having a first recognition unit and a second recognition unit with higher recognition accuracy than the first recognition unit. According to the product registration system of (2), similar to the above (1), product registration can be suitably performed using a mobile terminal.

[0362] In addition, the product registration system is equipped with a fraud monitoring unit (standard accuracy fraud monitoring unit, high accuracy fraud monitoring unit) that monitors fraud, etc. based on the operation status by the quantity change cancellation operation receiving unit and the recognition status by the recognition unit, and the notification unit may notify the judgment result (monitoring result) by the fraud monitoring unit.

[0363] (3) The product registration system described in (1) or (2) is characterized in that the second recognition unit recognizes the specific behavior after the first recognition unit recognizes the specific behavior (see Figures 7(A) to 7(C)). According to the product registration system (3), for example, fraudulent acts can be dealt with in real time at the sales floor, and accurate action can be taken until the payment is completed.

[0364] (4) The product registration system described in (1) is characterized in that, when the information based on the registration status by the registration unit and the recognition status by the first recognition unit differs from the information based on the registration status by the registration unit and the recognition status by the second recognition unit (when the judgment result by standard-accuracy fraud monitoring is overturned by high-accuracy fraud monitoring), the notification unit notifies (Figure 8) that the information differs (is overturned). According to the product registration system of (4), necessary information (information that the judgment result obtained by standard-precision fraud monitoring has been overturned by high-precision fraud monitoring) is reported as appropriate, so that, for example, confusion does not occur.

[0365] (5) The product registration system described in (2) is characterized in that, if the information based on the operation status by the quantity change cancellation operation receiving unit and the recognition status by the first recognition unit differs from the information based on the operation status by the quantity change cancellation operation receiving unit and the recognition status by the second recognition unit, the notification unit notifies that there is a difference. According to the product registration system of (5), necessary information (information that the judgment result based on standard-precision fraud monitoring has been overturned by high-precision fraud monitoring) is reported appropriately, so confusion does not occur.

[0366] Regarding (4) and (5), the notification unit is a product registration system that notifies the fact that the judgment result by the standard-accuracy fraud monitoring unit and the judgment result by the high-accuracy fraud monitoring unit are different when they differ.

[0367] (6) A product registration system as described in (1) or (2) is characterized in that it includes a settlement transition unit that executes a settlement transition process (a process that causes the settlement device 50 to receive transaction information, such as a process that causes the settlement device 50 to receive transaction information using a settlement code or short-range wireless communication) that transitions to a settlement process for settling the purchased products, and the settlement transition unit executes the settlement transition process after the second recognition unit has completed recognition of all purchased products (see Figures 7(A) to 7(C)). According to the product registration system (6), by the time the transaction proceeds to settlement, all products have already been identified as being fraudulent through high-precision fraud monitoring, so that the process after the transaction proceeds to settlement can be carried out quickly.

[0368] (7) A product registration system as described in (1) or (2) that includes a settlement unit (e.g., a settlement device 50, or a non-cash settlement unit of the settlement process 50, or a cash settlement unit of the settlement device 50) that executes a settlement process to settle the purchase of products, and the settlement unit executes the settlement process after the second recognition unit has completed recognition of all the products to be purchased. According to the product registration system (7), at the time of executing the settlement process, fraud etc. has already been determined for all products through high-precision fraud monitoring, so the settlement process can be carried out in an appropriate state (for example, a state in which there is no or almost no undetected fraud etc.).

[0369] (8) A product registration system as described in (1) or (2) that includes an information communication terminal (e.g., a mobile terminal 60, a dedicated cart, etc.) that has at least the registration unit, a server (e.g., a cloud server 20, etc.) that stores at least the registration information by the registration unit, and a settlement device (e.g., a settlement device 50, etc.) that executes a settlement process to settle the purchase of products, wherein the information communication terminal has the first recognition unit, and the server or the settlement device has the second recognition unit. The product registration system (8) is preferably realized according to the characteristics of each device (for example, the location where it is used (installed), the situation, the function, the processing capacity, etc.).

[0370] (9) A product registration system as described in (1) or (2) that includes an information communication terminal (e.g., a mobile terminal 60, a dedicated cart, etc.) that has at least the registration unit, and a settlement device (e.g., a settlement device 50, etc.) that executes a settlement process to settle the purchase of the product, and the notification unit notifies information based on the recognition status by the first recognition unit at the information communication terminal, and notifies information based on the recognition status by the second recognition unit at the settlement device. The product registration system (9) is preferably realized according to the characteristics of each device (for example, the location where it is used (installed), the situation, the function, the processing capacity, etc.).

[0371] Note that programs for implementing the functions of each of the above-described devices (e.g., the headquarters server 10, the store controller 30, the transaction status management device 40, the checkout device 50, the mobile terminal 60, and the cart 70) may be recorded on a computer-readable recording medium, and the programs may be loaded into a computer system and executed to perform the processing of each device. Here, "loading a program recorded on a recording medium into a computer system and executing it" includes installing the program on a computer system. The term "computer system" here includes hardware such as an operating system and peripheral devices. The term "computer system" may also include multiple computer devices connected via a network, including the Internet, a WAN, a LAN, a dedicated line, or other communication lines. The term "computer-readable recording medium" refers to portable media such as a floppy disk, a magneto-optical disk, a ROM, or a CD-ROM, as well as storage devices such as a hard disk built into a computer system. Thus, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM. The term "recording medium" also includes internal or external recording media accessible from a distribution server for distributing the program. The program code stored on the recording medium of the distribution server may be different from the program code in a format executable by the terminal device. In other words, the format in which the program is stored on the distribution server is not important as long as it can be downloaded from the distribution server and installed in a format executable by the terminal device. The program may be divided into multiple parts, each downloaded at a different time and then combined on the terminal device, or each part may be distributed by a different distribution server. Furthermore, the term "computer-readable recording medium" also includes a storage medium that stores a program for a certain period of time, such as volatile memory (RAM) within a computer system that serves as a server or client when a program is transmitted over a network. The program may also be for implementing part of the functions described above.Furthermore, the above-mentioned functions may be realized in combination with a program already recorded in the computer system, that is, a so-called differential file (differential program). [Explanation of symbols]

[0372] 1...Sales system 10...Head office server 20...Cloud server 30...Store controller 40...Transaction status management device 50...Payment device 60...Mobile terminal 70...Shopping cart 90...Special bag

Claims

1. A product registration system for registering products purchased by customers at a sales floor, a registration section for registering products to be purchased; an imaging unit capable of imaging an item reserved by a customer as an item to be purchased; a recognition unit that recognizes specific behaviors of customers regarding the acquisition of merchandise based on the captured images acquired from the imaging unit; a notification unit that notifies information based on a registration status by the registration unit and a recognition status by the recognition unit; Equipped with The recognition unit A first recognition unit and a second recognition unit having higher recognition accuracy than the first recognition unit. A product registration system characterized by:

2. A product registration system for registering products purchased by customers at a sales floor, a registration section for registering products to be purchased; a quantity change / cancellation operation receiving unit that receives an operation to change the quantity of a product to be purchased or an operation to cancel a purchase; an imaging unit capable of imaging an item reserved by a customer as an item to be purchased; a recognition unit that recognizes specific behaviors of customers regarding the acquisition of merchandise based on the captured images acquired from the imaging unit; a notification unit that notifies information based on the operation status by the quantity change cancellation operation receiving unit and the recognition status by the recognition unit; Equipped with The recognition unit A first recognition unit and a second recognition unit having higher recognition accuracy than the first recognition unit. A product registration system characterized by:

3. The second recognition unit Recognizing the specific behavior after the first recognition unit recognizes the specific behavior 3. The product registration system according to claim 1 or 2.

4. The notification unit When information based on the registration status by the registration unit and the recognition status by the first recognition unit differs from information based on the registration status by the registration unit and the recognition status by the second recognition unit, a notification is given to that effect.

2. The product registration system according to claim 1.

5. The notification unit If information based on the operation status by the quantity change cancellation operation receiving unit and the recognition status by the first recognition unit differs from information based on the operation status by the quantity change cancellation operation receiving unit and the recognition status by the second recognition unit, a notification is given to that effect.

3. The product registration system according to claim 2.

6. A payment transition unit that executes a payment transition process to transition to a payment process for settling the purchased product. Equipped with The settlement transition unit After the second recognition unit has completed the recognition of all the products to be purchased, the settlement transition process is executed.

3. The product registration system according to claim 1 or 2.

7. A checkout section that executes checkout processing to settle the purchase price of the product. Equipped with The settlement unit The settlement process is executed after the second recognition unit has completed recognition of all the products to be purchased.

3. The product registration system according to claim 1 or 2.

8. an information communication terminal including at least the registration unit; a server that stores at least the registration information by the registration unit; a payment device that executes a payment process for paying for the purchased products; Including, the information communication terminal includes the first recognition unit, The server or the settlement device is equipped with the second recognition unit.

3. The product registration system according to claim 1 or 2.

9. an information communication terminal including at least the registration unit; a payment device that executes a payment process for paying for the purchased products; Including, The notification unit notifying, at the information communication terminal, information based on the recognition status by the first recognition unit; Information based on the recognition status by the second recognition unit is notified at the settlement device.

3. The product registration system according to claim 1 or 2.

Images