Programs, information terminals, systems

Abstract

To appropriately register a commodity by using a portable terminal.SOLUTION: A program causes a computer to function as an information terminal including a first imaging unit and a second imaging unit. The program causes the computer to function as: output means; first detection means which detects that a commodity is held over, on the basis of a first captured image captured by the first imaging unit; and second detection means which detects that operator's behavior is a prescribed behavior, on the basis of a second captured image captured by the second imaging unit. The output means outputs a detection result of the second detection means while the first detection means detects that the commodity is held over.SELECTED DRAWING: Figure 1

Description

[[Technical Field]]

[0001] The present invention relates to a program, an information terminal, and a system. [[Background Art]]

[0002] A system is known in which customers themselves register products using a mobile terminal (see, for example, Patent Document 1). [[Prior Art Documents]] [[Patent Documents]]

[0003] [[Patent Document 1]] Japanese Unexamined Patent Application Publication No. 2018-147252 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] However, the systems as described above still have room for improvement, for example in countermeasures against malicious customers or inexperienced customers.

[0005] The present invention has been made in view of such circumstances, and an object thereof is to provide a technology for suitably implementing product registration using a mobile terminal. [[Means for Solving the Problem]]

[0006] A program according to one aspect for solving the above-described problem is a program that causes a computer to function as an information terminal having a first imaging unit and a second imaging unit, the program causing the computer to function as: output means; first detection means that detects that a product is being held toward the device on the basis of a first captured image captured by the first imaging unit; and second detection means that detects that an operator's action is a predetermined action on the basis of a second captured image captured by the second imaging unit, wherein the output means outputs the detection result of the second detection means when the first detection means detects that a product is being held toward the device. [[Brief Description of the Drawings]]

[0007] [Figure 1] This is a conceptual diagram of a network used to explain the sales system. [Figure 2] This is an explanatory diagram describing a cart that can be fitted with a mobile device. [Figure 3] This is a sequence diagram illustrating the general operation of the mobile terminal, cloud server, and payment device. [Figure 4] This is an example of how it looks on a mobile device. [Figure 5] This is an example of how it looks on a mobile device. [Figure 6] This is an example of how it looks on a mobile device. [Figure 7] This is a flowchart illustrating an example of how a mobile device works. [Figure 8] This is a flowchart illustrating an example of how a mobile device works. [Figure 9] This is an example of a display on a payment device. [Figure 10] This is an example of a display on a payment device. [Figure 11] This is an example of a display on a payment device. [Figure 12] This is an example of a display on a payment device. [Figure 13] This is an example of a display on a payment device. [Figure 14] This is an example of a display on a payment device. [Modes for carrying out the invention]

[0008] Figure 1 is a conceptual diagram of a network illustrating the sales system 1 of this embodiment. The sales system 1 shown in Figure 1 includes a headquarters server 10, a cloud server 20, a store controller 30, a transaction status management device 40 (e.g., a personal computer), a settlement device 50, and a mobile terminal 60.

[0009] Sales system 1 can be implemented in various types of businesses (supermarkets, convenience stores, drugstores, home centers, consumer electronics stores, etc.). In Figure 1, the dashed line T represents the inside of the store (product display area, checkout area, back room, etc.). As shown in Figure 1, the store controller 30, transaction status management device 40, and settlement device 50 are installed inside the store. These are connected via LAN 19 (which can be wired or wireless) for communication. A mobile terminal 60 is present inside the store. The mobile terminal 60 is basically used attached to (set up) a cart (shopping cart) 70. The headquarters server 10 and cloud server 20 are installed outside the store (for example, at the head office or data center). The devices installed inside the store (store controller 30, transaction status management device 40, settlement device 50) and the devices installed outside the store (head office server 10, cloud server 20) can communicate (either directly or via the store controller 30).

[0010] Note that the number of each device (terminal) in Figure 1 is illustrative. For example, although "..." is shown to the right of the payment device 50 in Figure 1, the number of payment devices 50 installed in the store only needs to be one or more. Also, although two mobile terminals 60 are shown in Figure 1, the number of mobile terminals 60 used in the store will vary depending on the number of customers. In addition, other devices may be installed in the store. For example, a group of semi-self-service devices (registration device, payment device) that register products based on the operation of a store clerk and settle the payment based on the operation of the customer may be installed in the store, or a fully self-service device (registration and payment device) that registers products based on the operation of the customer and settles the payment based on the operation of the customer may be installed.

[0011] (Mobile device 60) The mobile device 60 is used by the customer. Specifically, the mobile device 60 is used by the customer when registering products themselves within the store. The mobile device 60 is, for example, a smartphone, but it may also be a tablet device. The mobile device 60 may be owned by the customer, or it may be lent to the store.

[0012] Although not shown in the drawings, the mobile terminal 60 includes, for example, a CPU, a storage unit, a display unit, an operation unit, an audio output unit (speaker), an audio input unit (microphone), an imaging unit (camera), and a communication unit. The mobile terminal 60 may further include a light emitting unit (light) and a sensor (e.g., an acceleration sensor, a gyro sensor) capable of detecting the posture and vibration (displacement, velocity, or acceleration) of the mobile terminal 60 (housing).

[0013] The CPU is a central processing unit, and controls the entire mobile terminal 60 by reading and executing programs stored in the storage unit. The storage unit stores various types of information (e.g., information referenced or generated by the CPU, information acquired from the outside, etc.). The storage unit is a general term for one or more various storage members (e.g., members that serve as memory or storage; for example, RAM, ROM, hard disk, SSD (Solid State Drive), etc.). The display unit is, for example, a liquid crystal display, and displays various types of information including image information. The display unit is configured as a touch panel capable of accepting operations on a display screen. The operation unit accepts operations from an operator (customer). The operation unit is a general term for one or more various operation members (various operators (hard buttons) provided on the housing. Since the display unit is configured as a touch panel, it may also be regarded as a part of the operation unit. The audio output unit outputs audio. The audio input unit inputs audio. The imaging unit will be described later. The communication unit transmits and receives various types of information. The communication unit is a general term for one or more various communication members (e.g., communication members for mobile wireless communication such as 5G, wireless LAN, short-range wireless communication, non-contact communication, etc.).

[0014] (First imaging unit, second imaging unit, first captured image, second captured image) The mobile terminal 60 includes two imaging units. Specifically, the mobile terminal 60 includes a first imaging unit (first camera) that captures an image of the back side (the side on which the display unit is not provided), and a second imaging unit (second camera) that captures an image of the front side (the side on which the display unit is provided). The first imaging unit is what is commonly referred to as an out camera. The second imaging unit is what is commonly referred to as an in camera.

[0015] The first imaging unit images the rear side and acquires (generates) captured images (still images, moving images) of the rear side. The second imaging unit images the near side and acquires captured images (still images, moving images) of the near side. In the following description, a captured image captured by the first imaging unit may be referred to as a first captured image (or an out-camera image), and a captured image captured by the second imaging unit may be referred to as a second captured image (or an in-camera image).

[0016] (Output of First Captured Image and Second Captured Image) The mobile terminal 60 can display the first captured image (out-camera image) on the display unit as a so-called through image (live view image). The same applies to the second captured image (in-camera image). The mobile terminal 60 may display the other captured image (e.g., the second captured image) as a through image while displaying one captured image (e.g., the first captured image) as a through image. The mobile terminal 60 may display a part of the captured first captured image as a through image. The part of the captured first captured image refers to a part (temporal part) of the plurality of first captured images, or a partial region (partial in terms of imaging region) of one captured image. The same applies to the second captured image.

[0017] Instead of or in addition to displaying the first captured image as a through image on the display unit, the mobile terminal 60 may store the first captured image in a storage unit. That is, the mobile terminal 60 may store the first captured image in the storage unit without displaying it as a through image on the display unit, may display the first captured image as a through image on the display unit without storing it in the storage unit, or may display the first captured image as a through image on the display unit and store it in the storage unit at the same time. The mobile terminal 60 may store the first captured image in association with time information (e.g., current time, elapsed time from the start of imaging). The same applies to the second captured image. The mobile terminal 60 may store the first captured image and the second captured image captured at the same timing in association with each other, either with or without association with time information.

[0018] The mobile terminal 60 may display the first captured image stored in the memory unit. The same applies to the second captured image. When the mobile terminal 60 is displaying one of the captured images stored in the memory unit (e.g., the first captured image), it may display the other captured image stored in the memory unit (e.g., the second captured image). The mobile terminal 60 may display a part of the first captured image stored in the memory unit. A part of the first captured image stored in the memory unit is a part (temporal part) of a plurality of first captured images stored in the memory unit, or a part of an area (imaging area part) of one captured image stored in the memory unit. The same applies to the second captured image.

[0019] The mobile terminal 60 may transmit the first captured image to an external device (for example, a cloud server 20, a transaction status management device 40, or a settlement device 50) instead of displaying the first captured image on the display unit or storing it in the storage unit. The mobile terminal 60 may transmit the first captured image currently being acquired (which may be a portion of the captured image) to an external device, or it may transmit the first captured image already acquired (the first captured image stored in the storage unit; which may be a portion of the captured image) to an external device. The same applies to the second captured image.

[0020] The mobile terminal 60 (CPU) can control two imaging units (first imaging unit, second imaging unit) in parallel (simultaneously) and process (output, image processing) each of the captured images (first captured image, second captured image) in parallel.

[0021] (Image processing function) The mobile terminal 60 (CPU) has image processing functions (image recognition functions) that use technologies such as object extraction and pattern recognition within captured images.

[0022] (Shaking motion detection function) For example, the mobile terminal 60 has a hold-over motion detection function that detects a hold-over motion (the action of the operator holding a product in front of the first imaging unit; also called a scan action) from the first captured image. Note that "hold-over" means to hold something in your hand and position it somewhere, similar to "holding up a smartphone," and is the action of scanning a product (the action of positioning the mobile terminal 60 in the space between the holder unit 72 and the shopping basket 80). In other words, it does not have to be an action of lifting, covering, or blocking light. The hold-over motion detection function may also detect a hold-over motion based on whether a hand, product, etc. has appeared at the edge of the first captured image, or whether something resembling a barcode has been newly recognized in the first captured image. In other words, the mobile terminal 60 may detect a hold-over motion using the hold-over motion detection function based on whether a hand, product, etc. has appeared at the edge of the first captured image, or whether something resembling a barcode has been newly recognized in the first captured image. Furthermore, the mobile terminal 60 may enable (activate) the motion detection function while outputting (displaying, storing, and transmitting) the first captured image.

[0023] A barcode-like object is, for example, an object with a roughly rectangular outline and a striped pattern overall. Alternatively, an object with a roughly rectangular outline, a striped pattern overall, and numbers at the bottom of its longer side may also be considered a barcode-like object. The method by which the mobile terminal 60 determines whether or not it has newly recognized a barcode-like object in the first captured image is not particularly limited, but for example, a sample (pattern) of barcode-like objects may be prepared in advance (stored in the memory unit), and the mobile terminal 60 may compare the newly recognized object with the above sample to determine whether or not it has newly recognized a barcode-like object in the first captured image. The above sample may include, for example, at least one of the following: a barcode object that is photographed at an angle (the distance to the camera is different on one end and the other end); a barcode object that is partially or completely out of focus; a barcode object that is partially or completely reflective, overexposed, or underexposed; a barcode object that is partially or completely wrinkled or dirty; or a barcode object that is partially or completely obscured by a customer's finger or the like.

[0024] When the mobile terminal 60 detects a shadowing motion using its shadowing motion detection function, it outputs information indicating that a shadowing motion has been detected (information indicating that a shadowing motion has been detected). For example, when the mobile terminal 60 detects a shadowing motion, it displays information indicating that a shadowing motion has been detected (e.g., an image, a message, etc.) on its display unit. The mobile terminal 60 may store information indicating that a shadowing motion has been detected (e.g., a flag) in its storage unit instead of, or in addition to, displaying on the display unit or storing in the storage unit. The mobile terminal 60 may also transmit information indicating that a shadowing motion has been detected to an external source instead of, or in addition to, displaying on the display unit or storing in the storage unit. The mobile terminal 60 may also output (e.g., output for 0.2 seconds) audio (audio for when a shadowing motion is detected).

[0025] (Confirmation action detection function) Furthermore, the mobile terminal 60 has a confirmation action detection function that detects a hold-and-confirm action from the second captured image. A hold-and-confirm action is an action in which the operator adjusts and confirms their hold-and-confirm action (how they hold the barcode) in order to allow the barcode to be read. The confirmation action detection function may, for example, detect the hold-and-confirm action based on the operator's facial expression, the direction of the operator's face, and the direction of their gaze in the second captured image. In other words, the mobile terminal 60 may detect the hold-and-confirm action using the confirmation action detection function, for example, based on the operator's facial expression, the direction of the operator's face, and the direction of their gaze in the second captured image. The mobile terminal 60 may recognize each part of a person's face, such as the outline of the face, the eyes (entire), the pupils, the nose (entire), the nostrils, and the mouth, as objects from the second captured image, and determine the direction of the face and the direction of the gaze based on whether each of these is captured, and on their respective positions, sizes, and positional relationships. Furthermore, the mobile terminal 60 may enable the confirmation action detection function while outputting (displaying, storing, and transmitting) the second captured image.

[0026] When the mobile terminal 60 detects a shadow confirmation action using its confirmation action detection function, it outputs information indicating that a shadow confirmation action has been detected (that a shadow confirmation action has been detected). For example, when the mobile terminal 60 detects a shadow confirmation action, it displays information indicating that a shadow confirmation action has been detected (e.g., an image, a message, etc.) on its display unit. The mobile terminal 60 may store information indicating that a shadow confirmation action has been detected (e.g., a flag) in its storage unit instead of or in addition to displaying it on its display unit or storing it in its storage unit. The mobile terminal 60 may also transmit information indicating that a shadow confirmation action has been detected to an external source instead of or in addition to displaying it on its display unit or storing it in its storage unit. The mobile terminal 60 may also output (e.g., output for 0.2 seconds) audio (audio for when a shadow confirmation action is detected) instead of or in addition to the above.

[0027] (Code acquisition function) Furthermore, the mobile terminal 60 has a code acquisition function (code recognition function, scanner function) that reads a code (for example, a barcode such as a JAN code) from the first captured image (acquires a product code). The code acquisition function may, for example, recognize the barcode as an object and acquire the product code. In other words, the mobile terminal 60 may, for example, recognize the barcode as an object and acquire the product code using the code acquisition function. The mobile terminal 60 may also activate the code acquisition function while outputting the first captured image.

[0028] When the mobile terminal 60 obtains a product code using the code acquisition function, it sends the acquired product code and other information to the cloud server 20. If the mobile terminal 60 is displaying information indicating that a scanning action has been detected or information indicating that a scanning confirmation action has been detected when it obtains a product code, it clears the displayed information (information indicating that a scanning action has been detected, information indicating that a scanning confirmation action has been detected). The mobile terminal 60 may disable the confirmation action detection function when it obtains a product code.

[0029] The mobile terminal 60 (CPU) can control two or three of the following functions in parallel: the shadowing motion detection function, the confirmation motion detection function, and the code acquisition function. For example, if the mobile terminal 60 has enabled the shadowing motion detection function and the code acquisition function, and the shadowing motion detection function detects a shadowing motion, it may also enable the confirmation motion detection function in addition to the shadowing motion detection function and the code acquisition function. The mobile terminal 60 may start imaging by the second imaging unit at the timing of enabling the confirmation motion detection function (i.e., the timing of detecting a shadowing motion by the shadowing motion detection function), or it may start imaging by the second imaging unit at a timing earlier than the timing of enabling the confirmation motion detection function (for example, the timing of starting imaging by the first imaging unit).

[0030] The mobile terminal 60 may also measure (count) the time during which the motion detection function is enabled, or the time during which motion detection is performed by the motion detection function (the time during which a hand or product appearing from the edge of the first image is recognized, or the time during which something resembling a barcode of a newly appearing product is recognized). The mobile terminal 60 may also measure the time during which something resembling a product's barcode is recognized within the time during which motion detection is performed by the motion detection function.

[0031] The mobile terminal 60 may measure the time during which the confirmation action detection function is enabled, or the time during which the confirmation action detection function detects a hold-and-confirm action. The mobile terminal 60 may also measure the time during which the hold-and-confirm action is detected by the hold-and-confirm action detection function and the hold-and-confirm action is detected by the confirmation action detection function (the time during which the mobile terminal 60 determines that the customer is holding the product up and adjusting and confirming how they are holding it). The mobile terminal 60 may also measure the time during which the hold-and-confirm action is detected by the hold-and-confirm action detection function, during which the device recognizes something resembling the product's barcode, and the hold-and-confirm action is detected by the confirmation action detection function (the time during which the customer is holding the barcode portion of the product up and adjusting and confirming how they are holding it).

[0032] (Function to determine suitability of the gesture) The mobile terminal 60 has other functions in addition to image processing functions (e.g., a shadowing motion detection function, a shadowing confirmation function, and a code acquisition function). For example, the mobile terminal 60 has a shadowing motion suitability judgment function (also called a shadowing motion suitability judgment function or a shadowing motion quality evaluation function) that determines (judges, evaluates) whether the shadowing motion performed by the operator was sufficient for the mobile terminal 60 to acquire the product code.

[0033] The flagging action suitability judgment function determines the suitability of a flagging action performed by the operator using the detection results of the flagging confirmation action by the confirmation action detection function. The flagging action suitability judgment function may also determine the suitability of a flagging action performed by the operator using the detection results of the flagging confirmation action by the confirmation action detection function and the detection results of the flagging action by the flagging action detection function. In other words, the mobile terminal 60 uses the flagging action suitability judgment function to determine the suitability of a flagging action performed by the operator using the detection results of the flagging confirmation action by the confirmation action detection function, etc.

[0034] The content of the function for determining the appropriateness of a gesture, that is, the method by which the mobile terminal 60 determines whether the gesture performed by the operator is appropriate, may be, for example, any of the following methods A to G. (Method A) The mobile terminal 60 determines that the shadowing action performed by the operator was appropriate if the time during which the shadowing action is detected from the second captured image by the confirmation action detection function (the time during which the operator adjusts and confirms their own shadowing action) is equal to or greater than a predetermined standard time (for example, 3 seconds). The mobile terminal 60 determines that the shadowing action performed by the operator was not appropriate if the above time is less than the predetermined standard time. (Method B) The mobile terminal 60 determines that the operator's lifting action was appropriate if the time during which the lifting action is detected from the first captured image by the lifting action detection function (the time during which the customer lifts the product) is equal to or greater than a predetermined first reference time (e.g., 5 seconds), and the time during which the lifting confirmation action is detected from the second captured image by the confirmation action detection function is equal to or greater than a predetermined second reference time (e.g., 3 seconds). The mobile terminal 60 determines that the operator's lifting action was inappropriate if the time during which the lifting action is detected from the first captured image by the lifting action detection function is less than the first reference time, or if the time during which the lifting confirmation action is detected from the second captured image by the confirmation action detection function is less than the second reference time. (Method C) The mobile terminal 60 determines that the shadowing action performed by the operator was appropriate if the time during which the shadowing action is detected from the first captured image by the shadowing action detection function and the shadowing confirmation action is detected from the second captured image by the confirmation action detection function is equal to or greater than a predetermined reference time (e.g., 3 seconds). The mobile terminal 60 determines that the shadowing action performed by the operator was inappropriate if the above time is less than the predetermined reference time. (Method D) The mobile terminal 60 determines that the operator's gesture was appropriate if the time during which the gesture detection function newly recognizes something resembling a barcode in the first captured image is 5 seconds or longer, and the time during which the confirmation action detection function detects a gesture confirmation action from the second captured image is 3 seconds or longer. The mobile terminal 60 determines that the operator's gesture was inappropriate if the time during which the gesture detection function newly recognizes something resembling a barcode in the first captured image is less than the first reference time, or if the time during which the confirmation action detection function detects a gesture confirmation action from the second captured image is less than the second reference time. (Method E) If the mobile terminal 60 recognizes something resembling a barcode in the first captured image using the shadowing motion detection function, and the time during which the shadowing and confirmation motion is detected from the second captured image using the confirmation motion detection function is equal to or greater than a predetermined reference time (e.g., 3 seconds), then the mobile terminal 60 determines that the shadowing motion performed by the operator was appropriate. If the above time is less than the predetermined reference time, then the mobile terminal 60 determines that the shadowing motion performed by the operator was not appropriate. (Method F) If the ratio (percentage) of the time spent detecting a shadowing action from the second captured image to the time spent detecting a shadowing action from the first captured image is equal to or greater than a predetermined standard value (e.g., 60%), the system determines that the shadowing action performed by the operator was appropriate. If the above ratio is less than the predetermined standard value, the mobile terminal 60 determines that the shadowing action performed by the operator was not appropriate. (Method G) If the ratio of the time spent detecting the hold-and-confirm action from the second captured image to the time spent newly recognizing something resembling a barcode from the first captured image is equal to or greater than a predetermined standard value (e.g., 50%), the hold-and-confirm action performed by the operator is judged to be appropriate. If the above ratio is less than the predetermined standard value, the mobile terminal 60 judges that the hold-and-confirm action performed by the operator was not appropriate.

[0035] Figure 2 is an explanatory diagram illustrating a cart 70 to which a mobile terminal 60 can be attached. As shown in Figure 2, the cart 70 has a shopping basket holder section 71 on which a shopping basket 80 (not shown) can be placed, and a holder section 72 to which a mobile terminal 60 can be attached. Although it has been explained that the mobile terminal 60 is basically used attached to the cart 70, it is specifically used attached to the holder section 72.

[0036] In the example shown in Figure 2, the shopping basket 80 is not placed on the shopping basket holder 71, and the mobile terminal 60 is not attached to the holder 72. However, when a customer visits the store, they place the shopping basket 80 on the shopping basket holder 71 and attach the mobile terminal 60 to the holder 72. In addition, the cart 70 is equipped with a special bag 90 for temporarily storing items that need to be checked by a store employee at checkout.

[0037] When the mobile terminal 60 is attached to the holder 72 and imaging is performed by the first imaging unit, a first image is acquired in a direction looking down towards the shopping basket 80 from the position of the holder 72. When the mobile terminal 60 is attached to the holder 72 and imaging is performed by the second imaging unit, a second image is acquired in a direction looking up towards the ceiling from the position of the holder 72.

[0038] (Overview of the process to obtain the product code for item 1) The basic process for obtaining the product code for item number 1 is as follows: (1) After attaching the mobile terminal 60 to the holder unit 72 (or before attaching the mobile terminal 60 to the holder unit 72), the customer performs an operation to display the service start screen (menu screen; not shown). For example, the customer operates an icon that is generated when they register as a member. (2) The mobile terminal 60 displays the service start screen. (3) The customer shall perform an operation on the service start screen to authorize imaging by the second imaging unit (in-camera). This operation may be omitted based on prior settings or previous operations. (4) The customer operates the "Start Shopping" button on the service start screen. (5) The mobile terminal 60 displays the initial registration screen (the registration screen before registering the first product item; the registration screen shown in Figure 4(A)). The mobile terminal 60 also enables the gesture detection function and the code acquisition function. The mobile terminal 60 may disable the gesture detection function and the code acquisition function after the payment instruction button (described later) is pressed. (6) When the customer finds the desired product (the product to be purchased) in the product display area (display shelf), the customer moves the product below the holder unit 72 (the space between the holder unit 72 and the shopping basket 80) in order to have the barcode attached to the product read. In other words, the customer holds the product to be purchased below the holder unit 72 so that it is photographed by the first imaging unit (outside camera). (7) The mobile terminal 60 detects the customer's action of holding the product over it (holding action detection function). The mobile terminal 60 also activates the confirmation action detection function. (8) The mobile terminal 60 obtains the product code of the product (code acquisition function). The mobile terminal 60 also disables the confirmation action detection function.

[0039] (Payment device 50) The payment device 50 is used by the customer. Specifically, the payment device 50 settles the payment for products registered by the mobile terminal 60 based on the customer's operation. In other words, the sales system 1 provides a service in which the customer registers products themselves using the mobile terminal 60 and settles the payment themselves using the payment device 50.

[0040] Although not shown in the diagram, the payment device 50 includes a CPU, a memory unit, a display unit, a scanner unit, a card payment unit (non-cash payment unit), a change dispenser (cash payment unit), a printing unit, an audio output unit, an imaging unit, a communication unit, and a signpost. The payment device 50 may further include a key operation unit consisting of one or more various operating members, and sensors (such as motion sensors) that detect (detect and recognize) the presence of people.

[0041] The CPU is a central processing unit that controls the entire payment device 50 by reading and executing programs stored in the memory unit. The memory unit stores various information. The memory unit is a general term for one or more different storage components. The display unit is, for example, a liquid crystal display that displays various information. The display unit is configured as a touch panel that can accept operations on the display screen. The scanner unit scans various codes (for example, payment codes (described later), product codes, employee codes attached to vouchers, point cards, employee name tags, etc.) and reads each code.

[0042] The card payment unit is a payment mechanism that accepts various cards (credit cards, prepaid cards such as transportation cards, cash cards for debit payments, point cards for point payments, etc.). The change dispenser is a payment mechanism that accepts cash. The printing unit prints and issues various media (receipts, etc.). The audio output unit outputs audio (e.g., confirmation sounds, warning sounds, voice guidance, etc.). The imaging unit (camera) acquires captured images (still images, moving images). The communication unit is a general term for one or more various communication components. The sign pole has a light-emitting unit at its tip, and various information (call for staff, operating status, etc.) is communicated by the pattern of light emission from the light-emitting unit.

[0043] (Headquarters Server 10) The headquarters server 10 is a server that manages the entire sales system 1. The headquarters server 10 stores various information (e.g., product master file (product master), etc.) and various performance information (e.g., sales performance information, product sales performance information, etc.). The headquarters server 10 also supplies various information (e.g., product master, etc.) to the cloud server 20 and the store controller 30. The product master is a file that stores product names, prices, etc., associated with product codes. Although not shown in the diagram, the headquarters server 10 includes at least a CPU, a storage unit, and a communication unit. The headquarters server 10 may be composed of multiple servers.

[0044] (Cloud Server 20) The cloud server 20 is a server that primarily supports product registration by the mobile terminal 60. The cloud server 20 stores various types of information (e.g., product master data, transaction information (cart information), etc.). The cloud server 20 also supplies various types of information (e.g., screen information, etc.) to the mobile terminal 60 and the payment device 50. Although not shown in the diagram, the cloud server 20 comprises at least a CPU, a memory unit, and a communication unit. The cloud server 20 may be composed of multiple servers.

[0045] (Store controller 30) The store controller 30 is a server primarily responsible for communicating information between devices installed inside the store (transaction status management device 40, settlement device 50) and devices installed outside the store (head office server 10, cloud server 20). Although not shown in the diagram, the store controller 30 comprises at least a CPU, a memory unit, and a communication unit.

[0046] (Transaction status management device 40) The transaction status management device 40 is a device (for example, a personal computer) that manages (monitors and controls processing status, operating status, etc.) devices installed in the store (store controller 30, settlement device 50). Although not shown in the diagram, the transaction status management device 40 comprises at least a CPU, a memory unit, a display unit (which may be configured as a touch panel), an operation unit (keyboard, mouse, etc.), and a communication unit. The transaction status management device 40 can communicate with a printing device (not shown) and may output various information from the printing device.

[0047] In addition to managing devices installed within stores, the transaction status management device 40 may also access the headquarters server 10, etc., to maintain the product master stored in the headquarters server 10, etc. (such as setting up products), or it may access the cloud server 20 to display or modify transaction information, etc., stored in the cloud server 20.

[0048] (Overview of the operation of the mobile terminal 60, cloud server 20, and payment device 50) Figure 3 is a sequence diagram illustrating the general operation of the mobile terminal 60, the cloud server 20, and the payment device 50. The sequence diagram in Figure 3 shows the operation of each device (mobile terminal 60, cloud server 20, and payment device 50) from the time the mobile terminal 60 displays the initial registration screen (see, for example, Figure 4(A)) until the payment device 50 displays the product list screen (see, for example, Figure 9). Note that since the sequence diagram in Figure 3 mainly explains the coordination of each device, the explanation of parts related to the image processing functions of the mobile terminal 60 (shake detection function, confirmation detection function, code acquisition function, etc.) described above is omitted in the sequence diagram in Figure 3.

[0049] Furthermore, when the mobile terminal 60 displays the initial registration screen (based on the operation of the "Start Shopping" instruction button), it is assumed that it obtains (receives) transaction identification information from the cloud server 20 to identify the transaction (or transaction identification information for said transaction) made by the mobile terminal 60.

[0050] Step S1: The mobile terminal 60 obtains the product code based on the customer's gesture. Step S2: The mobile terminal 60 sends the transaction identification information for the transaction and the product code obtained in Step S1 to the cloud server 20.

[0051] Step S3: The cloud server 20 receives transaction identification information and product code from the mobile terminal 60. Step S4: The cloud server 20 updates the transaction information for the transaction (the transaction corresponding to the transaction identification information received in Step S3) based on the product code. In other words, the cloud server 20 retrieves (extracts) the product name and price corresponding to the product code received in Step S3 from the product master and stores them in the transaction information for that transaction. Step S5: The cloud server 20 sends the updated transaction information to the mobile terminal 60. For example, the cloud server 20 generates a registration screen (see, for example, Figure 5(A)) that reflects the content of the updated transaction information, and sends the screen information of the registration screen to the mobile terminal 60.

[0052] Step S6: The mobile terminal 60 receives screen information (transaction information) from the cloud server 20. Step S7: The mobile terminal 60 displays a registration screen based on the screen information received in Step S6. Specifically, in the case of a single product item, the mobile terminal 60 displays a registration screen reflecting that single product item instead of the initial registration screen, and in the case of a second or subsequent product item, the mobile terminal 60 displays a registration screen reflecting that second or subsequent product item.

[0053] After step S7, there are two possibilities: either the product code is obtained by the customer's gesture (returning to step S1), or a settlement code (described later) is generated and displayed by the customer's operation of the payment instruction button (e.g., payment instruction button BT50; see Figure 5(A)) (proceeding to step S8). In other words, steps S1 to S7 are executed repeatedly each time a product code is obtained.

[0054] Step S8: The mobile terminal 60 generates and displays a settlement code (e.g., settlement code GA60; see Figure 5(B)) based on the customer's operation of the payment instruction button. The settlement code is a coded (e.g., two-dimensional code) representation of the settlement information. The settlement information is the information necessary to obtain transaction information for the transaction from the cloud server 20 (e.g., transaction identification information to identify the transaction).

[0055] The customer has the payment device 50 read the payment code displayed on the mobile terminal 60. The payment device 50, in standby mode, displays a guidance screen that directs the customer to read the payment code (reading operation).

[0056] Step S9: The settlement device 50 reads the settlement code displayed on the mobile terminal 60 based on the customer's reading action. In other words, the settlement device 50 obtains transaction identification information that identifies the transaction. Step S10: The settlement device 50 requests transaction information for the transaction from the cloud server 20. For example, the settlement device 50 sends request information (transaction information acquisition request information) including transaction identification information that identifies the transaction to the cloud server 20.

[0057] Step S11: The cloud server 20 receives a request (transaction information acquisition request information). Step S12: The cloud server 20 transmits the transaction information for the transaction (the transaction corresponding to the transaction identification information included in the transaction information acquisition request information received in step S11) to the settlement device 50. For example, the cloud server 20 generates a product list screen (see Figure 9) that reflects the contents of the transaction information for the transaction, and transmits the screen information of the product list screen to the settlement device 50.

[0058] Step S13: The settlement device 50 and the mobile terminal 60 receive screen information (transaction information) from the cloud server 20. Step S14: The payment device 50 displays the product list screen based on the screen information received in step S13.

[0059] (Example display on mobile device 60) Figures 4 to 6 show examples of the display of the mobile terminal 60. The mobile terminal 60 is assumed to be already attached to the cart 70 (holder part 72). Note that the image captured from the position of the holder part 72 looking down at the shopping basket 80 captures the shopping basket 80 and items already placed in the shopping basket 80 when no waving action is being performed, and when a waving action is being performed, the target product of the waving action and the shopping basket 80 and items already placed in the shopping basket 80 are captured behind the target product. However, in Figures 4 to 6, for simplification, items other than the target product of the waving action (such as the shopping basket 80 and items already placed in the shopping basket 80) are omitted from the illustration.

[0060] Figure 4(A) shows the registration screen (initial registration screen) when the bridging operation has not been performed. The registration screen in Figure 4(A) (and other registration screens are similar) is provided with an image capture display area HR610 and a registration information display area HR620. The image capture display area HR610 is mainly for displaying the first captured image (outside camera image). The registration information display area HR620 is mainly for displaying the product registration information.

[0061] Figure 4(B) is an example of a display of a scene following the scene in Figure 4(A). Specifically, Figure 4(B) is an example of a display of a scene in which the customer begins to hold up the product (○○ bread). When the mobile terminal 60 detects the customer holding up the product, it displays information indicating that the holding action has been detected. In the example shown in Figure 4(B), the mobile terminal 60 displays the image GA01 of the product being held up as information indicating that the holding action has been detected.

[0062] Figure 4(C) is an example of a display of a scene following the scene in Figure 4(B). Specifically, Figure 4(C) is an example of a display of a scene in which the customer has begun to adjust and confirm their own holding motion. When the mobile terminal 60 detects the customer's holding confirmation motion (the motion of adjusting and confirming the holding motion), it displays information indicating that the holding confirmation motion has been detected. In the example shown in Figure 4(C), the mobile terminal 60 displays the confirmation image GA02 as information indicating that the holding confirmation motion has been detected.

[0063] Figure 4(D) is an example of a display of a scene following the scene in Figure 4(C). Specifically, Figure 4(D) is an example of a display of a scene in which the position of the barcode of the product in the captured image has moved to a position that is easier to read than the position in Figure 4(C) due to the scanning and verification action.

[0064] Figure 5(A) is an example of a display of a scene following the scene in Figure 4(D). Specifically, Figure 4(A) is an example of a display of a scene in which, as a result of the scanning operation, the position of the barcode of the product in the captured image moves to a position that is easier to read than the position in Figure 4(D), the barcode of the product is read, and the product is registered. In other words, the mobile terminal 60 acquires the product code of the product (○○ bread) (step S1 in Figure 3), sends the acquired product code etc. to the cloud server 20 (step S2 in Figure 3), and displays the registration information of the product in the registration information display area HR620 (step S7 in Figure 3).

[0065] As shown in Figure 5(A), when the mobile terminal 60 reads a barcode, it displays the product registration information in the registration information display area HR620. The mobile terminal 60 also clears the scanning image GA01 and the confirmation image GA02 after reading a barcode. Furthermore, as shown in Figure 5(A), after registering a product (one item), the mobile terminal 60 displays an operational payment instruction button BT50 (in this example, a button labeled "Proceed to Checkout") which accepts payment instructions. The customer operates the payment instruction button BT50 when they wish to proceed to checkout (i.e., when they have finished registering the product).

[0066] Figure 5(B) shows an example of a scene following the scene in Figure 5(A). In other words, when the payment instruction button BT50 is pressed, the mobile terminal 60 generates a settlement code GA60 and displays it on a screen like the one shown in Figure 5(B) (settlement code display screen).

[0067] Figures 5(C) and 5(D) are examples of displays showing situations where the product code could not be obtained after the customer held the product up to the device, although the circumstances are different. The mobile terminal 60 determines that the product code could not be obtained after the product was held up to the device if the holding action stopped (the device stopped detecting the holding action) without obtaining the product code after the holding action occurred (after the holding action was detected). The mobile terminal 60 may also determine that the product code could not be obtained after the product was held up to the device if the holding action stopped without obtaining the product code and there was no further holding action within a predetermined time (the device did not re-detect the holding action within a predetermined time after the device stopped detecting the holding action).

[0068] Figure 5(C) shows an example of the display when the product code could not be obtained after the product scanning action, but the scanning action was performed correctly. If the mobile terminal 60 determines that the product code could not be obtained after the product scanning action, and that the scanning action was performed correctly, it displays the small screen SG10 as shown in Figure 5(C). For example, if, from the scene in Figure 4(C), it determines that the product code could not be obtained after the product scanning action, and that the scanning action was performed correctly, it displays the small screen SG10 in front of the initial registration screen.

[0069] Figure 5(D) shows an example of a display when the product code could not be obtained after the product scanning action, and the scanning action was deemed improper. If the mobile terminal 60 determines that it could not obtain the product code after the product scanning action, and that the scanning action was deemed improper, it displays a small screen SG20 as shown in Figure 5(D). For example, if, from the scene in Figure 4(B), it determines that it could not obtain the product code after the product scanning action, and that the scanning action was deemed improper, it displays a small screen SG20 in front of the initial registration screen.

[0070] In other words, if the mobile terminal 60 fails to register a product despite the customer properly holding up the product, it displays a small screen SG10 as shown in Figure 5(C), and if the customer fails to register a product because they did not properly hold up the product, it displays a small screen SG20 as shown in Figure 5(D).

[0071] The small screen SG10 in Figure 5(C) displays a message MS11 indicating that the product (barcode) scan failed, and a facial image GA13 of the customer during the scanning operation. The OK button BT15 is also displayed as operable on the small screen SG10. The facial image GA13 is a second image captured by the second imaging unit during the scanning operation (or an image cropped from the second image to show the face). The customer's face in the facial image GA13 is facing approximately forward (towards the lens of the second imaging unit), and their gaze is also approximately directed towards the camera.

[0072] When the OK button BT15 on the small screen SG10 shown in Figure 5(C) is pressed, the mobile terminal 60 displays the small screen SG30 (described later) as shown in Figure 6(A) instead of the small screen SG10.

[0073] The small screen SG20 in Figure 5(D) displays a message MS21 (similar to message MS11 on small screen SG10) indicating that the product scan failed, a message MS22 indicating that the way the product was held up may not have been appropriate, and a facial image GA23 of the customer during the holding action. The OK button BT25 is also displayed as operable on the small screen SG20. The facial image GA23 is a second image captured by the second imaging unit during the holding action (or an image cropped from the second image to show the face). The customer's face in the facial image GA23 is not facing forward, and their gaze is not directed at the camera.

[0074] When the OK button BT25 on the small screen SG20 in Figure 5(D) is pressed, the mobile terminal 60 displays the small screen SG40 (described later) as shown in Figure 6(B) instead of the small screen SG20. Furthermore, when the button BT36 registered on the small screen SG30 (Figure 6(A)) is pressed, the mobile terminal 60 also displays the small screen SG40 (Figure 6(B)) instead of the small screen SG30.

[0075] Figure 6(A) shows an example of a display for a scene following the scene in Figure 5(C). As described above, when the OK button BT15 on the small screen SG10 (Figure 5(C)) is pressed, the mobile terminal 60 displays the small screen SG30 shown in Figure 6(A) instead of the small screen SG10.

[0076] The small screen SG30 displays a message MS31 indicating that it is a confirmation and a message MS34 asking whether or not to register the product. The small screen SG30 also displays two operable buttons: a "Do not register" button BT35 to indicate that the product will not be registered, and a "Register" button BT36 to indicate that the product will be registered.

[0077] On the small screen SG30, if the customer does not wish to register the product, they return the product to the shelf and operate the "Do not register" button BT35 (they may also return the product to the shelf after operating the "Do not register" button BT35). On the small screen SG30, if the customer wishes to register the product, they operate the "Register" button BT36.

[0078] Figure 6(B) is an example of a display of a scene following the scene in Figure 5(D) or the scene in Figure 6(A). As described above, when the OK button BT25 on the small screen SG20 (Figure 5(D)) is pressed, the mobile terminal 60 displays the small screen SG40 as shown in Figure 6(B) instead of the small screen SG20, and when the registration button BT36 on the small screen SG30 (Figure 6(A)) is pressed, the small screen SG40 as shown in Figure 6(B) is displayed instead of the small screen SG30.

[0079] The small screen SG40 displays a message MS41 instructing the customer to place the items in the special bag 90, an image GA42 instructing the customer to place the items in the special bag 90, and a message MS44 informing the customer that a store employee will assist them at the checkout machine (checkout device 50). The OK button BT45 is also displayed as operable on the small screen SG40.

[0080] On the small screen SG40, the customer places the product into the dedicated bag 90 and operates the OK button BT45 (they may also place the product into the dedicated bag 90 after operating the OK button BT45).

[0081] If the mobile terminal 60 detects an operation of the button BT35 that is not to be registered on the small screen SG30, it clears the small screen SG30 and displays information on the registration screen indicating that the customer's holding action was correct but the product scan failed. For example, as shown in Figure 6(C), the mobile terminal 60 displays "Scan Failed (Holding OK)" in the registration information display area HR620 to indicate that the customer's holding action was correct but the product scan failed.

[0082] If the OK button BT45 is pressed on the small screen SG40, the mobile terminal 60 clears the small screen SG40 and displays information on the registration screen indicating that the customer's holding motion was not appropriate and the product scan failed. For example, as shown in Figure 6(D), the mobile terminal 60 displays "Scan Failed (Holding Motion NG)" in the registration information display area HR620 as information indicating that the customer's holding motion was not appropriate and the product scan failed.

[0083] In other words, if the scan fails but the customer's gesture is correct, the transition is from Figure 4(C) to Figure 6(A). If the button BT35 (not to be registered) is pressed in Figure 6(A), the transition is from Figure 6(A) to Figure 4(A). If the button BT36 (to be registered) is pressed in Figure 6(A), the transition is from Figure 6(A) to Figure 6(B) to Figure 6(C). On the other hand, if the scan fails but the customer's gesture is incorrect, the transition is from Figure 4(D) to Figure 6(B) to Figure 6(C).

[0084] Figures 7 and 8 are flowcharts illustrating an example of the operation of the mobile terminal 60. The flowcharts in Figure 8(A) and Figure 8(B) transition from the flowchart in Figure 7, respectively. Note that the flowcharts in Figures 7 and 8 are related to steps S1, S2, S7, S8, etc., of the sequence diagram in Figure 3 (mainly related to step S1).

[0085] (Flowchart in Figure 7) The flowchart in Figure 7 starts based on the operation of the "Start Shopping" instruction button on the service start screen. It is assumed that imaging by the second imaging unit is permitted. Although not shown in the diagram, the mobile terminal 60 activates the "Hold" motion detection function and the "Code Acquisition" function, and initializes the "Store Clerk Confirmation Flag" (described later), based on the operation of the "Start Shopping" instruction button. Although not shown in the diagram, the mobile terminal 60 also measures various times used to determine the appropriateness of the "Hold" motion (step S130), such as the time it takes for the confirmation motion detection function to detect the "Hold" confirmation motion from the second captured image.

[0086] Step S100: The mobile terminal 60 determines whether or not it has detected the customer's wave-to-the-device motion. In other words, the mobile terminal 60 determines whether or not it has started to detect the customer's wave-to-the-device motion using its wave-to-the-device motion detection function. If a wave-to-the-device motion is detected (Step S100: YES), proceed to Step S101. If a wave-to-the-device motion is not detected (Step S100: NO), return to Step S100 (repeat Step S100).

[0087] Step S101: The mobile terminal 60 displays information indicating that a holding motion has been detected (for example, the holding motion image GA01 in Figure 4(B)). The process then proceeds to step S102. Step S102: The mobile terminal 60 enables the confirmation action detection function. Then proceed to step S110.

[0088] While the confirmation action detection function is enabled (after activation but before deactivation), if the customer adjusts or confirms a shadowing action, the mobile terminal 60 will detect this as a shadowing confirmation action, although this is not shown in the diagram. When the mobile terminal 60 detects a shadowing confirmation action, it displays information indicating that a shadowing confirmation action has been detected (for example, the confirmation image GA02 in Figure 4(C)). When the mobile terminal 60 no longer detects shadowing confirmation actions, it deletes the information indicating that a shadowing confirmation action has been detected. Furthermore, while the mobile terminal 60 is displaying information indicating that a shadowing confirmation action has been detected, if it no longer detects a shadowing action (step S110(NO)) or if it obtains a product code (step S200(YES)), it deletes the displayed information indicating that a shadowing confirmation action has been detected.

[0089] Step S110: The mobile terminal 60 determines whether or not it is detecting the customer's wave-to-the-device action. In other words, the mobile terminal 60 determines whether or not it is continuing to detect the wave-to-the-device action that it began detecting in step S100. If it is detecting the wave-to-the-device action (step S110: YES), proceed to step S200. If it is not detecting the wave-to-the-device action (step S110: NO), proceed to step S111. Step S200: The mobile terminal 60 determines whether or not it has obtained a product code. If it has obtained a product code (Step S200: YES), proceed to Step S211. If it has not obtained a product code (Step S200: NO), return to Step S110.

[0090] Step S111: The mobile terminal 60 erases the information indicating that it has detected a gesture. Then proceed to step S112. Step S112: The mobile terminal 60 disables the confirmation action detection function. Then proceed to step S120.

[0091] Step S120: The mobile terminal 60 determines whether or not it has detected the customer's raising action. In other words, the mobile terminal 60 determines whether or not the customer has resumed the raising action. If the raising action is detected (Step S120: YES), the process returns to Step S101. If the raising action is not detected (Step S120: NO), the process proceeds to Step S121. Step S121: The mobile terminal 60 determines whether or not it is within the recovery grace period. The recovery grace period is a predetermined time (for example, 3 seconds) from the time when no gesture is detected (step S110 (NO)). If it is within the recovery grace period (step S121: YES), the process returns to step S120. If it is not within the recovery grace period (step S121: NO), the process proceeds to step S130.

[0092] In other words, if the mobile terminal 60 detects a scanning action again within a predetermined time (recovery grace period) (step S120 (YES)), it considers that a series of scanning actions (scanning actions for one product) are continuing and prevents the process from proceeding to the scan failure sequence (step S130 onwards) (it returns to step S101).

[0093] Step S130: The mobile terminal 60 determines whether the customer's gesture was appropriate. For example, the mobile terminal 60 evaluates (determines) the appropriateness of the gesture performed by the customer using one of the methods A to G described above. If it was appropriate (Step S130: YES), proceed to step S132 in Figure 8(A). If it was inappropriate (Step S130: NO), proceed to step S172 in Figure 8(B).

[0094] Step S211: The mobile terminal 60 erases the information indicating that it has detected a gesture. Then proceed to step S212. Step S212: The mobile terminal 60 disables the confirmation action detection function. Then proceed to step S220. Step S220: The mobile terminal 60 registers the product. That is, the mobile terminal 60 sends the product code obtained in step S200, along with the transaction identification information, to the cloud server 20, and displays the product information of the product on the display unit (steps S2 to S7 in Figure 3). The process then proceeds to step S230.

[0095] Step S230: The mobile terminal 60 determines whether it has detected the customer's gesture of raising the device. In other words, the mobile terminal 60 determines whether it has started detecting the customer's gesture of raising the device again. If a gesture of raising the device is detected (Step S230: YES), the process returns to Step S101. If a gesture of raising the device is not detected (Step S230: NO), the process proceeds to Step S240. Step S240: The mobile terminal 60 determines whether or not the payment instruction button BT50 has been operated. If it has been operated (Step S240: YES), proceed to Step S250. If it has not been operated (Step S240: YES), return to Step S230. Step S250: The mobile terminal 60 generates and displays a settlement code (Step S8 in Figure 3). If the "Clerk Confirmation Required" flag is set, the mobile terminal 60 generates and displays a settlement code that includes information indicating that the transaction has the "Clerk Confirmation Required" flag set. The flowchart in Figure 7 then ends.

[0096] (Flowchart in Figure 8(A)) The flowchart in Figure 8(A) shows the processing flow in the case where scanning fails but the customer's holding action was correct.

[0097] Step S132: The mobile terminal 60 outputs a notification sound (confirmation sound) (for example, for 1 second). The notification sound is a softer tone than the warning sound output in step S172 of Figure 8(B) (for example, a sound that can be described as "poloron"). The process then proceeds to step S140.

[0098] Step S140: The mobile terminal 60 displays a message indicating that the reading (scanning) failed, along with the face image being scanned. For example, the mobile terminal 60 displays the small screen SG10 shown in Figure 5(C). The process then proceeds to step S141. Step S141: The mobile terminal 60 determines whether or not a confirmation operation was performed. For example, the mobile terminal 60 determines whether or not the OK button BT15 on the small screen SG10 in Figure 5(C) was pressed. If a confirmation operation was performed (Step S141: YES), proceed to step S150. If no confirmation operation was performed (Step S141: NO), return to step S141.

[0099] Step S150: The mobile terminal 60 confirms with the customer whether or not to register the product. For example, the mobile terminal 60 displays the small screen SG30 shown in Figure 6(A). The process then proceeds to step S151. Step S151: The mobile terminal 60 determines whether or not an operation to not register was performed. For example, the mobile terminal 60 determines whether or not the "do not register" button BT35 on the small screen SG30 in Figure 6(A) was operated. If an operation to not register was performed (Step S151: YES), the process returns to Step S100 in Figure 7. If no operation to not register was performed (Step S151: NO), the process proceeds to Step S152.

[0100] Step S152: The mobile terminal 60 determines whether or not an operation to register was performed. For example, the mobile terminal 60 determines whether or not the registration button BT36 on the small screen SG30 in Figure 6(A) was operated. If an operation to register was performed (Step S152: YES), proceed to step S160. If an operation to register was not performed (Step S152: NO), return to step S151.

[0101] Step S160: The mobile terminal 60 instructs the customer to place the items into the dedicated bag 90 and notifies the customer via the checkout device 50 that a store employee will assist them. For example, the mobile terminal 60 displays the small screen SG40 shown in Figure 6(B). The process then proceeds to step S161. Step S161: The mobile terminal 60 determines whether or not a confirmation operation was performed. For example, the mobile terminal 60 determines whether or not the OK button BT45 on the small screen SG40 in Figure 6(B) was pressed. If a confirmation operation was performed (Step S161: YES), proceed to step S162. If no confirmation operation was performed (Step S161: NO), return to step S161.

[0102] Step S162: The mobile terminal 60 sets the flag indicating that the transaction requires confirmation from a store employee. If the flag indicating that the transaction requires confirmation from a store employee has already been set, the process in step S160 may be omitted. The process then proceeds to step S163.

[0103] The "Staff Requires Confirmation" flag indicates whether or not staff intervention is required (whether or not a staff call is necessary) during the processing by the payment device 50. When the "Staff Requires Confirmation" flag is set, it indicates that staff intervention is required, and when the "Staff Requires Confirmation" flag is cleared, it indicates that staff intervention is not required. As mentioned above, the "Staff Requires Confirmation" flag is cleared at the start of the transaction (when the "Start Shopping" instruction button is pressed).

[0104] Step S163: The mobile terminal 60 displays a message indicating that the scanning operation was performed correctly, but registration could not be completed. For example, as shown in Figure 6(C), the mobile terminal 60 displays "Scan failed (Scanning OK)" in the registration information display area HR620. The process then proceeds to step S164.

[0105] Step S164: The mobile terminal 60 sends the captured image and code information (for example, code information "01") indicating that the scanning operation was correct but registration could not be completed, along with the transaction identification information, to the cloud server 20. Then, the process returns to step S100 in Figure 7.

[0106] In step S164 (and similarly in step S194 in Figure 8(B)), the mobile terminal 60 transmits a first image captured by the first imaging unit and a second image captured by the second imaging unit. For example, the mobile terminal 60 transmits to the cloud server 20 a first image captured while a shadowing motion is being detected by the shadowing motion detection function (the image displayed as confirmation image GA52 on the small screen SG50 shown in Figures 13 and 14), a first image captured before a shadowing motion is detected by the shadowing motion detection function (the image displayed as confirmation image GA51 on the small screen SG50 shown in Figures 13 and 14), a first image captured after the shadowing motion is no longer detected by the shadowing motion detection function (the image displayed as confirmation image GA53 on the small screen SG50 shown in Figures 13 and 14), and multiple second images captured while a shadowing motion is being detected by the shadowing motion detection function (the images displayed as face images GA55 on the small screen SG50 shown in Figures 13 and 14). The cloud server 20 stores the captured images received from the mobile terminal 60, associating them with the transaction. The captured images (first captured image, second captured image) transmitted by the mobile terminal 60 to the cloud server 20 may be images with necessary parts cropped or otherwise edited.

[0107] (Flowchart in Figure 8(B)) The flowchart in Figure 8(B) shows the processing flow when scanning fails and the customer's holding action was not performed correctly.

[0108] Step S172: The mobile terminal 60 outputs a warning sound (for example, for 1 second). The warning sound is louder than the notification sound output in step S132 of Figure 8(A) (for example, a sound that can be described as a "beep"). The process then proceeds to step S180.

[0109] Step S180: The mobile terminal 60 displays a message indicating that the reading (scanning) failed, along with the face image being scanned. For example, the mobile terminal 60 displays the small screen SG20 shown in Figure 5(D). The process then proceeds to step S181. Step S181: The mobile terminal 60 determines whether or not a confirmation operation was performed. For example, the mobile terminal 60 determines whether or not the OK button BT25 on the small screen SG20 in Figure 5(D) was pressed. If a confirmation operation was performed (Step S181: YES), proceed to step S190. If no confirmation operation was performed (Step S181: NO), return to step S181.

[0110] The processes in steps S190, S191, and S192 in Figure 8(B) are identical to the processes in steps S160, S161, and S162 in Figure 8(A), respectively, so their explanation is omitted. The process proceeds from step S192 to step S193.

[0111] Step S193: The mobile terminal 60 displays a message indicating that registration failed because the scanning operation was not performed correctly. For example, as shown in Figure 6(D), the mobile terminal 60 displays "Scan Failed (Scanning Failed)" in the registration information display area HR620. The process then proceeds to step S194.

[0112] Step S194: The mobile terminal 60 sends the transaction identification information, the captured image, and code information indicating that registration could not be completed due to improper scanning (for example, code information "02") to the cloud server 20. Then, the process returns to step S100 in Figure 7.

[0113] The operation of the mobile terminal 60 has been explained above using the flowcharts in Figures 8 and 9, but some additional information regarding the operation of the mobile terminal 60 will be provided below.

[0114] (1) The acquisition of product codes using the code acquisition function is basically related to the detection of a holding motion using the holding motion detection function, but is basically not related to the detection of a holding confirmation motion using the confirmation motion detection function. The product code of a product is acquired when a customer holds the product over it, so the flow is "Holding motion detection in progress" → "Acquisition of product code". However, the product code may also be acquired when the customer has not confirmed their holding motion (for example, the product code may be acquired after the holding motion has been confirmed (=not confirmed), or in cases where the product is registered without looking, so to speak). Therefore, in addition to the flow "Holding confirmation motion detection in progress" → "Acquisition of product code", there is also the flow "Holding confirmation motion not detected" → "Acquisition of product code". In other words, the product code can be acquired not only when a holding confirmation motion is being detected (when the confirmation image GA02 is displayed), but also when a holding confirmation motion is not being detected (when the confirmation image GA02 is not displayed). Here, we are focusing on the detection results of the confirmation action performed by the confirmation action detection function when the product code is not obtained, rather than when it is obtained.

[0115] (2) Although it was explained that the system focuses on the detection result of the confirmation action detection function when the product code is not acquired, rather than when it is acquired, it is also acceptable to focus on the detection result of the confirmation action detection function when the product code is acquired. For example, the detection result of the confirmation action detection function when the product code is acquired may be used to set (change) the reference time, etc., used to determine whether the confirmation action is appropriate (step S130).

[0116] As an example, the reference time in Method A (the reference time compared to the detection time of the shadow confirmation action) may be set (changed) based on the detection time of the shadow confirmation action in the situation in which the product code was acquired. In other words, the detection times of the shadow confirmation action in the situation in which the product code was acquired during experiments or actual operation may be accumulated and aggregated, and the reference time in Method A may be set accordingly. The median or average of the N detection times accumulated in each of the N situations may also be set as the reference time in Method A. Furthermore, in order to improve the accuracy (certainty) of the evaluation of inappropriateness (to reduce cases in which appropriateness is judged as inappropriate), the shortest time among the N detection times (times below which there is no record of product code acquisition) or the shortest time in the top 10% may be set as the reference time in Method A.

[0117] Figures 9 to 14 show examples of the display of the settlement device 50. Figure 9 (and similarly Figures 11 and 12) is the product list screen, which displays a list of products within a transaction. Figure 10 is the settlement type selection screen, where the settlement type (settlement method, settlement method, payment method) can be selected. Figure 13 (and similarly Figure 14) is the small screen (status confirmation screen) SG50, which is displayed in front of the product list screen. For the sake of explanation, the small screen SG50 shown in Figure 13, etc., is shown enlarged relative to the product list screen shown in Figure 9, etc.

[0118] Figure 9 shows the contents of the product list screen when staff assistance is not required. For example, when the payment device 50 reads the payment code GA60 for a transaction in which five items (XX bread, XX coffee, XX snacks, XX canned goods, XX bento box) are sequentially scanned on the mobile terminal 60 and each item is successfully registered (a transaction in which the staff confirmation flag was not set), it displays the product list screen as shown in Figure 9.

[0119] The product list screen in Figure 9 displays the five items mentioned above (XX bread, XX coffee, XX snacks, XX canned goods, XX bento boxes) in the order they were registered. The product list screen in Figure 9 also displays an operable checkout button (a button labeled "Checkout" in the lower right corner of the screen) to proceed with the payment process of the payment device 50 (to select the payment type). The customer confirms the items and prices, and then operates the checkout button.

[0120] When the payment button is pressed on the product list screen, the payment device 50 displays the payment type selection screen. For example, when the payment button is pressed on the product list screen shown in Figure 9, the payment device 50 displays the payment type selection screen as shown in Figure 10. The customer selects their desired payment type on the payment type selection screen. The payment device 50 then processes the payment using the selected payment type. If the "Product List" button on the payment type selection screen is pressed, the payment device 50 displays the product list screen (returns to the product list screen).

[0121] Figures 11 and 12 show the display content of the product list screen when staff assistance is required. For example, when the payment device 50 reads the payment code GA60 for a transaction (a transaction with the staff confirmation flag set) in which the customer scanned the five items listed above in order using the mobile terminal 60, and the scanning operation for the XX loaf of bread was correct but registration could not be completed, displaying "Scan failed (Scan OK)", and the scanning operation for the XX canned goods was incorrect, so registration could not be completed, displaying "Scan failed (Scan NG)", the payment device 50 displays the product list screen as shown in Figure 11.

[0122] In the product list screen in Figure 11, products that resulted in a "Scan Failure (Scanning OK)" (e.g., product XX) are displayed as "Scan NG (Appropriate)," and products that resulted in a "Scan Failure (Scanning NG)" (e.g., canned goods XX) are displayed as "Scan NG (Inappropriate)." For example, the payment device 50 displays "Scan NG (Appropriate))" based on code information indicating that the scanning operation was appropriate but registration could not be completed (e.g., code information "01"), and displays "Scan NG (Inappropriate)" based on code information indicating that registration could not be completed because the scanning operation was not appropriate (e.g., code information "02").

[0123] For example, if the payment device 50 reads the payment code GA60 for a transaction where the customer scanned the five items in order using the mobile terminal 60, but the scanning operation for the canned goods was not performed correctly and therefore the transaction could not be registered, resulting in a "Scan Failed (Scanning Not Allowed)" message (a transaction with a flag indicating that staff confirmation is required), the device will display a product list screen as shown in Figure 12. The product list screen in Figure 12 displays "Scan Failed (Scanning Not Allowed)" for the product that resulted in a "Scan Failed (Scanning Not Allowed)" message (the canned goods).

[0124] The product list screen in Figure 11 (and similarly in Figure 12) does not display a checkout button. Additionally, the product list screen in Figure 11 (and similarly in Figure 12) shows the number of items that have been put on hold because they require assistance from a store employee (held items; in this example, items that fall under "Scan Failed (Appropriate)" or "Scan Failed (Inappropriate)"). This number is shown to the right of "Holding Items" in the lower right corner of the screen.

[0125] In the payment device 50, if a product list screen indicating that staff assistance is required (for example, the product list screens in Figures 11 and 12) is displayed, a staff member is called (staff call processing is executed). The staff call processing includes, for example, the illumination of the sign pole (light-emitting part) of the payment device 50, and the transmission of staff call information to devices used by staff (for example, the transaction status management device 40, a tablet terminal or smartphone carried by the staff, a semi-self-service registration device, signage installed in the store, etc.). Note that the product list screen in Figure 11 (and similarly the product list screen in Figure 12) displays the message, "Calling a staff member. Please wait a moment," indicating that staff assistance is being requested.

[0126] The payment device 50 displays a small screen SG50 in front of the product list screen when an operation (touch) is performed on the "Scan Failed (Appropriate)" or "Scan Failed (Inappropriate)" column on the product list screen. For example, when an operation is performed on the "Scan Failed (Appropriate)" column on the product list screen in Figure 11, the payment device 50 displays a small screen SG50 as shown in Figure 13 in front of the product list screen in Figure 11. Also, for example, when an operation is performed on the "Scan Failed (Inappropriate)" column on the product list screen in Figure 12, the payment device 50 displays a small screen SG50 as shown in Figure 14 in front of the product list screen in Figure 12.

[0127] The display of the small screen SG50 described above is performed by a store employee (for example, a called employee). The payment device 50 may accept operations in the "Scan Failed (Appropriate)" or "Scan Failed (Inappropriate)" column when a store employee logs in. The payment device 50 may display the small screen SG50 when a store employee logs in after an operation in the "Scan Failed (Appropriate)" or "Scan Failed (Inappropriate)" column (or when there is a secret operation known to the store employee). The payment device 50 may display a small screen (not shown) in front of the product list screen (product list screen in Figures 11 and 12) that accepts the store employee's login or secret operation when a store employee's response is required, and may not accept operations on the product list screen (operations in the "Scan Failed (Appropriate)" or "Scan Failed (Inappropriate)" column) before accepting the store employee's login or secret operation on the small screen (not shown).

[0128] As shown in Figures 13 and 14, the central part of the small screen SG50 displays confirmation images GA51, GA52, and GA53 from left to right. The central confirmation image GA52 is the first captured image during the detection of the shadowing motion. The left confirmation image GA51 is the first captured image before the shadowing motion was detected. The right confirmation image GA53 is the first captured image after the shadowing motion was no longer detected. The right side of the small screen SG50 displays the face image GA55. The face image GA55 is the second captured image during the detection of the shadowing motion. In other words, the face image GA55 is the second captured image taken at the same time as confirmation image GA52.

[0129] On the small screen SG50, above and below the face image GA55, the previous image button BT56 and the subsequent image button BT57 are displayed and operable. The previous image button BT56 is a button that displays the second image taken slightly before the second image currently displayed as face image GA55 from among multiple second images taken during the detection of the shadowing motion. The subsequent image button BT57 is a button that displays the second image taken slightly after the second image currently displayed as face image GA55 from among multiple second images taken during the detection of the shadowing motion.

[0130] At the bottom of the small screen SG50, the input screen button BT54 is displayed and operable. The input screen button BT54 is a button that displays the registration screen (not shown) for registering unregistered items, such as items that failed to scan.

[0131] The above images (confirmation images GA51-GA53, face image GA55) are sent from the mobile terminal 60 to the cloud server 20, but the payment device 50 only needs to receive the above images from the cloud server 20 at least (no later) before displaying them. For example, the payment device 50 may receive the above images from the cloud server 20 as a response after reading the payment code GA60 (i.e., at the start of processing), or it may receive the above images from the cloud server 20 when there is an operation in the "Scan Failed (Appropriate)" or "Scan Failed (Inappropriate)" column on the product list screen (i.e., when displaying the small image screen SG50).

[0132] The confirmation image GA52 of the small screen SG50 in Figure 13 captures the motion of holding up the "XX slice of bread" which resulted in a "Scan NG (Appropriate)" (= "Scan Failed (Holding Up OK)"). Also, similar to the face image GA13 in Figure 5(C), the customer's face in the face image GA55 of the small screen SG50 in Figure 13 is facing approximately forward, and their gaze is also approximately directed towards the camera.

[0133] The confirmation image GA52 of the small screen SG50 in Figure 14 captures the motion of holding up the "○○ canned food" which resulted in a "scan NG (inappropriate)" (= "scan failure (holding up NG)"). Also, similar to the face image GA23 in Figure 5(D), the customer's face in the face image GA55 of the small screen SG50 in Figure 14 is not facing forward, and their gaze is not directed at the camera.

[0134] The store clerk checks the situation when a scan fails using the confirmation images GA51-GA53 on the small screen SG50. Specifically, they check the pallet gesture when the scan fails using the central confirmation image GA52, and confirm the state of the shopping basket 80 (whether there are any changes) before and after the gesture. In addition, the store clerk checks the customer's facial expression and gaze when the scan fails using the face image GA55 on the small screen SG50.

[0135] For example, in the case of the small screen SG50 in Figure 13, the store clerk can confirm from the central confirmation image GA52 and the face image GA55 that the customer was attempting to register the product (XX bread), as it is evaluated as scan NG (appropriate). The store clerk can also confirm from the left and right confirmation images GA51 and GA53 (as there is no change in the displayed content), or from the right confirmation image GA53 (as the product is not captured), that the customer has not placed the product in the shopping basket 80. Note that when purchasing a product that has failed to scan (scan OK), the customer is instructed to place the product in the special bag 90 instead of the shopping basket 80 (step S160 in Figure 8(A)). However, whether or not the product is in the special bag 90 does not allow confirmation of whether the customer followed the instructions and placed the product in the special bag 90 while simultaneously placing the same product in the shopping basket 80.

[0136] In the case of the small screen SG50 in Figure 14, the store clerk can confirm from the central confirmation image GA52 and the face image GA55 that the customer is not attempting to register the product (○○ canned goods) as it is evaluated as scan NG (inappropriate) (a pseudo-scan, a fraudulent act). The store clerk can also confirm from the left and right confirmation images GA51 and GA53, or from the right confirmation image GA53, that the customer has not placed the product in the shopping basket 80. In the case of a scan failure (scan NG), the customer is instructed to place the product in the special bag 90 (step S190 in Figure 8(B)), but whether or not the product is in the special bag 90 does not allow confirmation of whether the customer followed the instructions and placed the product in the special bag 90 while also placing the same product in the shopping basket 80.

[0137] Having grasped the registration status as described above and confirmed the contents of the special bag 90, the store clerk operates the input screen button BT54. When the input screen button BT54 is operated, the payment device 50 displays the registration screen (not shown). On the registration screen, the store clerk enters the product code (inputs the quantity) of the product removed from the special bag 90 (products that failed to scan), or scans the barcode of the product removed from the special bag 90. The payment device 50 obtains the product code based on the input of the product code or the scan of the barcode and registers the product. When the payment device 50 has registered all the products that failed to scan, it displays the payment button as operable, as shown in the product list screen in Figure 9.

[0138] As described above, each embodiment allows for the effective implementation of product registration using a mobile terminal.

[0139] (Variations, etc.) Although each embodiment has been described above, the configuration of the equipment, the data configuration, the processing flow, the display and output modes, etc., can be changed as appropriate, for example, as shown in (1) to (40) below. Furthermore, (1) to (40) below can be combined as appropriate.

[0140] (1) Figure 4 illustrates an example in which the mobile terminal 60 displays a message indicating that a wave-over motion has been detected (displays the wave-over image GA01). However, as mentioned when describing the wave-over motion detection function, instead of or in addition to the display, information indicating that a wave-over motion has been detected may be stored in the memory unit, or information indicating that a wave-over motion has been detected may be transmitted to an external source, or sound may be output.

[0141] (2) Figure 4 illustrates an example in which the mobile terminal 60 notifies the user that it has detected a holding motion (by displaying the holding motion image GA01) when it detects a holding motion, but this notification is not required. In other words, as also related to (1) above, the mobile terminal 60 does not need to display different information depending on the detection result of the holding motion detection function.

[0142] (3) In Figure 4, an example was shown in which, when the mobile terminal 60 detects a wave-over motion, a notification is made that a wave-over motion has been detected (a wave-over image GA01 showing "Waving Over" is displayed), and nothing is notified when no wave-over motion is detected. However, when no wave-over motion is detected, a notification may be made that no wave-over motion has been detected (for example, an image showing "No Wave-over" or "Not Wave-over" is displayed). The text on the image may also be switched according to the detection result.

[0143] (4) In Figure 4, an example was shown in which the mobile terminal 60 notifies that a shadow confirmation action has been detected (by displaying the confirmation image GA02) when it detects a shadow confirmation action. However, as mentioned when describing the confirmation action detection function, instead of or in addition to the display, information indicating that a shadow confirmation action has been detected may be stored in the memory unit, or information indicating that a shadow confirmation action has been detected may be transmitted to an external source, or sound may be output.

[0144] (5) Figure 4 illustrates an example in which the mobile terminal 60 notifies the user that a hold-over action has been detected (by displaying the hold-over image GA01) when the hold-over action is detected, but this notification is not required. In other words, as also related to (4) above, the mobile terminal 60 does not need to display different information depending on the detection result of the hold-over action detection function.

[0145] (6) In Figure 4, an example was shown in which, when the mobile terminal 60 detects a hold-and-confirm action, a notification is made that a hold-and-confirm action has been detected (by displaying a confirmation image GA02 that says "Confirming"), and no notification is made when a hold-and-confirm action has not been detected. However, when a hold-and-confirm action has not been detected, a notification may be made that a hold-and-confirm action has not been detected (for example, by displaying an image that says "No Confirmation" or "Unconfirmed"). The text on the image may also be switched according to the detection result.

[0146] (7) In Figure 4, an example was shown in which the mobile terminal 60 explicitly notifies the user that it has detected a hold-and-confirm action (by displaying the "Checking" image GA02). However, it is not necessary to explicitly notify the user that it has detected a hold-and-confirm action when it has detected one. For example, the mobile terminal 60 may simply display different information depending on whether it has detected a hold-and-confirm action or not. As an example, when the mobile terminal 60 has not detected a hold-and-confirm action, it may display "Please hold the product over the image" in the image display area HR610, and when it has detected a hold-and-confirm action, it may display "Please scan the barcode" or "Please align the barcode" in the image display area HR610, as shown in Figure 4(C).

[0147] (8) Related to (7) above, the mobile terminal 60 may display a message to support the hold-and-confirm action when it has detected the hold-and-confirm action. For example, when the mobile terminal 60 has not detected the hold-and-confirm action, it may display "Please hold the product over the device" in the captured image display area HR610, and when it has detected the hold-and-confirm action, it may display "Please scan the barcode" or "Please align the barcode" in the captured image display area HR610.

[0148] (9) In the above embodiment, an example was described in which the confirmation action detection function is activated when the mobile terminal 60 detects a gesture, but similar to the gesture detection function and the code acquisition function, the gesture suitability judgment function may be activated at the start of a transaction (based on the operation of the start shopping instruction button).

[0149] (10) Related to (9) above, the mobile terminal 60 may also notify that it has detected a hold-over action (display the hold-over image GA01) when it has not detected a hold-over action (display the hold-over confirmation image GA02). In other words, the mobile terminal 60 may notify that it has detected a hold-over confirmation action when it has detected a hold-over confirmation action, regardless of whether it has detected a hold-over action or not.

[0150] (11) In the above embodiment, an example was described in which the mobile terminal 60 displays the face image GA13 on the small screen SG10 (Figure 5(C)) when it determines that the customer's gesture is appropriate (step S130 (YES)). However, the face image GA13 may not be displayed on the small screen SG10, but rather on the small screen SG40 which is displayed when an operation to register a product (operation of the registration button BT36) is performed on the small screen SG30 that follows the small screen SG10.

[0151] (12) In the above embodiment, the mobile terminal 60 has been described as displaying small screen SG30 from small screen SG10 when it determines that the customer's gesture is appropriate, and displaying small screen SG40 from small screen SG30 in response to the operation of small screen SG30. However, when it determines that the customer's gesture is appropriate, it may also display a small screen that combines the contents of small screen SG10 and small screen SG30, and display small screen SG40 from that small screen in response to the operation of that small screen.

[0152] (13) In the above embodiment, an example was described in which the mobile terminal 60 displays the face image GA23 on the small screen SG20 (Figure 5(D)) when it determines that the customer's gesture is inappropriate (step S130(NO)). However, the face image GA23 may not be displayed on the small screen SG20, but rather on the small screen SG40 that follows the small screen SG20.

[0153] (14) In the above embodiment, an example was described in which the mobile terminal 60 displays small screen SG40 from small screen SG20 when it determines that the customer's gesture is inappropriate. However, if it determines that the customer's gesture is inappropriate, it may display a small screen that combines the contents of small screen SG20 and small screen SG40.

[0154] (15) In the above embodiment, the mobile terminal 60 is shown to display a face image (face image GA13 in the former case and face image GA23 in the latter case) whether it is determined that the customer's gesture is appropriate (i.e., the customer has no malicious intent and the possibility of a false scan is relatively low) or inappropriate (i.e., the customer has malicious intent and the possibility of a false scan is relatively high). However, the display / hide of the face image may be controlled according to the determination result of whether the customer's gesture is appropriate or not.

[0155] For example, the mobile terminal 60 may not display the face image (face image GA13) if it determines that the customer's gesture was appropriate, but may display the face image (face image GA23) if it determines that the customer's gesture was inappropriate. In other words, the mobile terminal 60 may omit displaying the face image (face image GA13) if the customer does not have malicious intent and the possibility of a false scan is considered relatively low. The payment device 50 may also omit displaying the face image GA55, or it may display the face image GA55.

[0156] For example, the mobile terminal 60 may not display the face image (face image GA23) if it determines that the customer's gesture is inappropriate, but may display the face image (face image GA13) if it determines that the customer's gesture is appropriate. In other words, the mobile terminal 60 may omit displaying the face image (face image GA23) if it is considered that the customer has malicious intent and there is a relatively high possibility of a false scan, and may only display the face image GA55 at the payment device 50.

[0157] (16) In the above embodiment, an example was described in which the mobile terminal 60 displays a face image (face image GA13 in the former case, face image GA23 in the latter case) if it is determined that the customer's gesture was appropriate or inappropriate when it is not possible to obtain a product code. However, it is not necessary to display a face image. In other words, the face image GA55 may be displayed for the first time at the payment device 50. Furthermore, if it is determined that the gesture was appropriate, the display of the face image GA55 at the payment device 50 may be omitted.

[0158] (17) In the above embodiment, an example was described in which a series of raising actions is considered to be continuing if a raising action is detected again within a predetermined time (recovery grace period) (steps S120 and S121 in Figure 7). However, for example, if a raising action is detected again before the customer turns to face forward (i.e., before it can be considered that the customer has lost interest in the product), a series of raising actions may be considered to be continuing. Whether or not the customer is facing forward can be determined from the orientation of the face in the second captured image.

[0159] (18) In the above embodiment, an example was described in which the mobile terminal 60 transmits captured images (first captured image, second captured image) to the cloud server 20 when the product is placed in the dedicated bag 90 (step S164 in Figure 8(A), step S194 in Figure 8(B)). However, the captured images may also be transmitted to the cloud server 20 when the product is returned to the display shelf. In other words, in addition to when the product is placed in the dedicated bag 90, the checkout device 50 may also be configured to confirm that the shopping basket 80 does not contain any products when the product is returned to the display shelf.

[0160] (19) In the above embodiment, an example was described in which the mobile terminal 60 determines that the customer's gesture is appropriate and, if the customer does not purchase the product, returns the product to the display shelf and continues processing. However, the mobile terminal 60 may also instruct the customer to place the product in the dedicated bag 90. In other words, in the above embodiment, an example was described in which the control proceeds from step S151(YES) to step S100 in Figure 8(A). However, the control may also proceed from step S151(YES) to step S160 in Figure 8(A).

[0161] (20) In the above embodiment, an example was described in which the mobile terminal 60 determines that the customer's scanning action is appropriate and instructs the customer to place the product in the dedicated bag 90 if the customer purchases the product. However, the customer may be instructed to scan the product again. In other words, in the above embodiment, an example was described in which the control proceeds from step S152 (YES) to step S160 in Figure 8(A). However, the control may proceed from step S152 (YES) to step S100 in Figure 8(A).

[0162] (21) In the above embodiment, an example was described in which a store clerk responds at the payment device 50 when at least one of the conditions a1 (see below) and b1 (see below) is met. However, a store clerk may respond at the payment device 50 only when condition a1 is met. In the case of condition b1, the store clerk may not have to respond, so the product may be scanned again as shown in (20) above. (Condition a1) Within a transaction, there was one or more instances where the swipe operation was not performed correctly and the product code could not be obtained. In other words, the mobile terminal 60 determined NO at step S130 in Figure 7 at least once within a single transaction. (Condition b1) Within a transaction, there was one or more instances in which an operation to purchase a product occurred where the swipe action was correct but the product code could not be obtained. In other words, the mobile terminal 60 determined YES at step S152 in Figure 8(A) at least once within a single transaction.

[0163] (22) In the above embodiment, an example was described in which a store employee responds at the payment device 50 when at least one of the conditions a1 (above) and b1 (above) is met. However, a store employee may respond at the payment device 50 when at least one of the conditions a1, b1, and c1 (below) is met. (Condition c1) Within a transaction, there was one or more instances where the user decided not to purchase an item for which the swipe action was correct but the product code could not be obtained. In other words, the mobile terminal 60 made a YES judgment at step S151 in Figure 8(A) at least once within a single transaction.

[0164] (23) In the above embodiment, an example was described in which a store employee responds at the payment device 50 when at least one of the conditions a1 (above) and b1 (above) is met. However, a store employee may respond at the payment device 50 only when condition a2 (below) is met. (Condition a2) During the transaction, there were L or more instances where the product code could not be obtained due to an improper scanning action. L is 2 or greater. Until L instances occur, the product may be returned to the shelf so that a store employee does not have to intervene.

[0165] (24) In the above embodiment, an example was described in which a store employee responds at the payment device 50 when at least one of the conditions a1 (above) and b1 (above) is met. However, a store employee may also respond at the payment device 50 when at least one of the conditions a2 (above), b2 (below), and c2 (below) is met. (Condition b2) Within a transaction, there were M or more instances where the user attempted to purchase an item for which the product code could not be obtained despite the correct scanning action. M in condition b2 may be the same number as L in condition a2, but it is preferable that M is greater than L. Until M instances are reached, the product may be returned to the shelf so that a store employee does not have to intervene. (Condition c2) Within a transaction, there were N or more instances where the user chose not to purchase an item for which the swipe action was correct but the product code could not be obtained. N in condition c2 may be the same as L in condition a2, but it is preferable that N be greater than L. M in condition b2 and N in condition c2 may be the same or different.

[0166] (25) As per (22) to (24) above, the message displayed to the customer operating the mobile terminal 60 may vary depending on the reason why the product code could not be obtained and the number of times the product code could not be obtained. As an example based on the reason, if the scanning action was correct but the product code could not be obtained, the mobile terminal 60 may simply display "Failed to obtain product code," and if the scanning action was incorrect and therefore the product code could not be obtained, it may display "Failed to obtain product code because the scan was incorrect." As an example based on the number of attempts, the mobile terminal 60 may display "Please scan while looking at the barcode" the first time, "Please scan the product carefully" the second time, and "Please do not put the item in the cart without scanning" from the third time onward. The mobile terminal 60 may vary the message depending on the combination of reason and number of attempts.

[0167] In place of or in addition to the above message, the voice (volume, frequency, wording of the voice guidance, presence or absence of voice, etc.) may be varied based on the reason why the product code could not be obtained or the number of times the product code could not be obtained. For example, the first time, it could be "Please scan while looking at the barcode" + a gentle sound that simply informs the customer of the situation (or silence); the second time, "Please scan the product carefully" + a sound to draw attention; and from the third time onward, "Please do not put the item in your basket without scanning" + a warning sound that other customers around you will also notice.

[0168] (26) In the above embodiment, an example was described in which the appropriateness of a shaking motion is determined mainly based on a shaking confirmation motion from a second captured image by a confirmation motion detection function when a shaking motion is detected from a first captured image by a shaking motion detection function. In other words, in the above embodiment, an example was described in which the mobile terminal 60 determines the appropriateness of a shaking motion based on the customer's facial expression, etc., when the product is being shaked. However, as is also related to (9) above, the mobile terminal 60 may determine the appropriateness of a shaking motion based on the customer's facial expression, etc., when the product is not being shaked, in addition to or instead of the customer's facial expression, etc., when the product is being shaked. Since the customer checks the through image when shaking the product (before the product is actually shaked) (because they see the through image before the product itself is captured in the through image), the customer's facial expression, etc., before the detection of the shaking motion is also important. For example, the appropriateness of a shadowing action may be determined by weighting the shadowing confirmation action before shadowing (a predetermined time before the shadowing action is detected, e.g., 3 seconds) to be given the same importance as the shadowing confirmation action during shadowing (while the shadowing action is being detected), and not giving importance to the shadowing confirmation action after shadowing (a predetermined time after the shadowing action is no longer detected, e.g., 3 seconds). Alternatively, the appropriateness of a shadowing action may be determined by weighting the pre-shadowing action the most, the shadowing action the next most important, and not giving importance to the post-shadowing action. Furthermore, if the mobile terminal 60 does not acquire a product code, it will display a message indicating that the scan failed (small screen SG10 in Figure 5(C) or small screen SG20 in Figure 5(D)). However, since the customer's reaction to this message is also considered to be related to the customer's behavior, the time after the above-mentioned completion of the scanning should include the time the message is displayed. The facial image of the user while viewing the message (whether they appear calm, surprised, flustered, etc.) may be analyzed to determine whether the scanning action is appropriate.

[0169] (27) In the above embodiment, an example was described in which the mobile terminal 60 and the payment device 50 display the face image being held up. However, the mobile terminal 60 and the payment device 50 may display the face image before or after being held up, instead of or in addition to the face image being held up. For example, the payment device 50 may display the face image GA55 before being held up when the front image button BT56 is operated on the small screen SG50 in Figure 13 (and similarly in Figure 14), and display the face image GA55 after being held up when the rear image button BT57 is operated. In addition, as explained in (26) above, the appropriateness of a waving motion may be determined based on a waving confirmation action before or after detection of the waving motion. However, regardless of whether or not the appropriateness of a waving motion is determined based on a waving confirmation action before or after detection of the waving motion, the mobile terminal 60 and the payment device 50 may display the face image before or after the waving motion.

[0170] (28) In the above embodiment, an example was described in which the settlement device 50 that reads the settlement code GA60 always displays the product list screen. However, the settlement device 50 does not have to display the product list screen if the store clerk's assistance is not required. In other words, the settlement device 50 that reads the settlement code GA60 may display a payment type selection screen as shown in Figure 10 if the store clerk's assistance is not required, and may display a product list screen as shown in Figures 11 and 12 if the store clerk's assistance is required. In other words, the cloud server 20 may send a payment type selection screen as shown in Figure 10 to the settlement device 50 for transactions that do not require the assistance of a store clerk, and may send a product list screen as shown in Figures 11 and 12 to the settlement device 50 for transactions that require the assistance of a store clerk.

[0171] (29) In the above embodiment, an example was described in which a second captured image (face image) captured by the mobile terminal 60 is displayed on the settlement device 50 (Figures 13 and 14). However, it may also be displayed on a device other than the settlement device 50 (for example, the transaction status management device 40). For example, the transaction status management device 40 may access the cloud server 20, display a product list screen for a single transaction, and then display the small screen SG50 from the product list screen.

[0172] (30) In the above embodiment, an example was described in which a second image captured by the mobile terminal 60 is transmitted to an external device (e.g., a cloud server 20), stored therein, and displayed on an external device (e.g., a settlement device 50). However, the second image may be deleted as appropriate to prevent it from being used for purposes other than its original purpose (determining and confirming whether the waving operation is appropriate or not). For example, when the settlement device 50 completes the settlement of a transaction, it transmits settlement information for that transaction to the cloud server 20. In addition to transmitting the settlement information, it may also delete the second image related to that transaction that it has stored in its own memory. Furthermore, when the cloud server 20 receives settlement information for a transaction, it may delete the second image related to that transaction that it has stored in its own memory. Also, the settlement device 50 (and the cloud server 20) may, for example, delete all second images related to all transactions stored in its memory at once at a predetermined time outside of business hours.

[0173] (31) In the above embodiment, it was explained that permission is sought from the customer for imaging by the second imaging unit. In relation to (30) above, when seeking permission from the customer, one or more of the following may be informed to the customer: that the second image will not be used for purposes other than those intended; that the second image may be transmitted to an external party; and that the second image transmitted to an external party will be properly deleted. For example, the mobile terminal 60 may display a screen (which may also be a service start screen) that requests permission from the operator for imaging by the second imaging unit, and may display information regarding the deletion of the second image on that screen.

[0174] (32) In the above embodiment, permission is sought from the customer for imaging by the second imaging unit, but customers who grant permission may be given benefits (for example, discounts on products or total amounts, points or coupons).

[0175] (33) In the above embodiment, the confirmation action detection function was described as detecting the shadow confirmation action based on the operator's facial expression, the direction of the operator's face and gaze in the second captured image, but the shadow confirmation action may also be detected using a trained model generated by machine learning. The trained model for detecting the shadow confirmation action (hereinafter referred to as trained model 1200) may be a model represented using a neural network, for example. Specifically, the trained model 1200 may output the probability (likelihood) that the shadow confirmation action is being performed and the probability (likelihood) that the shadow confirmation action is not being performed when the second captured image is input (i.e., based on the second captured image).

[0176] The trained model 1200 is generated by a trained model generation device (e.g., a personal computer). The trained model generation device generates a model based on, for example, a second image taken in a scene where the shadow-confirmation operation is not performed, and training data indicating that the shadow-confirmation operation is not performed in each of those scenes (i.e., training data corresponding to each of the second image taken, indicating that the shadow-confirmation operation is not performed), and a second image taken in a scene where the shadow-confirmation operation is performed, and training data indicating that the shadow-confirmation operation is performed in each of those scenes (i.e., training data corresponding to each of the second image taken, indicating that the shadow-confirmation operation is performed). The trained model 1200 generated by the trained model generation device is stored in a location accessible by the mobile terminal 60 (e.g., the memory of the mobile terminal 60).

[0177] The mobile terminal 60 inputs the second image acquired from the second imaging unit into the input layer of the trained model 1200 and obtains output values ​​(the probability that the shadow confirmation operation is being performed, and the probability that the shadow confirmation operation is not being performed) from the output layer. For example, the mobile terminal 60 determines that the shadow confirmation operation is not being performed if the probability obtained from the output layer is less than a predetermined probability (for example, 50%), and determines that the shadow confirmation operation is not being performed if it is equal to or greater than the predetermined probability.

[0178] (34) In the above embodiment, an example was described in which the mobile terminal 60 determines whether the gesture is appropriate when it does not acquire a product code. However, as is also related to the setting of the reference time and (33) above, the gesture may also determine whether the gesture is appropriate when it acquires a product code. In addition, the result of the determination of whether the gesture is appropriate when it does not acquire a product code may be displayed, but the result of the determination of whether the gesture is appropriate when it acquires a product code may not be displayed.

[0179] (35) As explained in the above embodiment, the first captured image is a rear-facing image (still image, moving image), and the second captured image is a front-facing image (still image, moving image). The first captured image (and the second captured image as well) may be a still image or a moving image. For example, the face image GA13 on the small screen SG10 in Figure 5(C) may be a video that can be played / stopped in response to operation. The same applies to the face image GA23 on the small screen SG20 in Figure 5(D), and the face image GA55 on the small screen SG50 in Figures 13 and 14. Also, the confirmation image GA52 on the small screen SG50 in Figures 13 and 14 may be a video that can be played / stopped in response to operation. The same applies to the confirmation images GA51 and GA53.

[0180] (36) In the above embodiment, an example was described in which captured images, etc., are displayed on a small screen (small screens SG10, SG20 in the mobile terminal 60, and small screen SG50 in the payment device 50), but the small screen is just an example, and the screen on which captured images, etc., are displayed does not have to be a small screen.

[0181] (37) In the above embodiment, it was explained that a customer who comes to the store places the shopping basket 80 on the shopping basket placement section 71 and attaches the mobile terminal 60 to the holder section 72 (Figure 2). However, the angle (elevation angle) and height of the holder section 72 may be adjustable. The mobile terminal 60 has a holder confirmation and adjustment function for checking and adjusting the angle and height of the holder section 72. The mobile terminal 60 may also check the angle and height of the holder section 72 based on the rear camera image and the front camera image when attached to the holder section 72, and announce that the angle and height should be adjusted as necessary. For example, the mobile terminal 60 may check the image capture range of the rear camera image (for example, it is OK if a part of the edge of the shopping basket 80 or the basket support of the cart 70 is captured), or check the objects captured in the front camera image (for example, it is OK if the outline of a face is captured), and if adjustment is necessary, it may display how to adjust the angle and height of the holder part 72 (for example, make it slightly horizontal without changing the height) using images (animations) or text. The mobile terminal 60 may also check the angle and height before displaying the service start screen, or after registering the first product.

[0182] (38) In the above embodiment, an example was described in which the mobile terminal 60 is attached to the cart 70 (holder portion 72), but the mobile terminal 60 may not be the customer's property but be part of the cart 70. The cart 70 and the mobile terminal 60 may be configured as an integral part.

[0183] (39) In the above embodiment, the lenses of the first imaging unit (first camera) and the second imaging unit (second camera) have not been described, but the first imaging unit may have one lens or multiple lenses. For example, the first imaging unit may be equipped with three fixed focal length lenses (standard lens, wide-angle lens, telephoto lens) with different angles of view (focal lengths), and may be switched according to the distance to the object (barcode). Alternatively, the first imaging unit may be equipped with one zoom lens with a wide range of angles of view (focal lengths), and the angle of view may be controlled according to the distance to the object. Digital zoom may be used instead of optical zoom.

[0184] (40) Some or all of the functions (input / output, storage, processing (including judgment)) in each device (mobile terminal 60, settlement device 50, transaction status management device 40, etc.) may be implemented in a device other than the device described as the entity that executes the function. For example, each device may function as a so-called thin client specializing in input / output interfaces, etc. As an example, a cloud server (which may be the same as cloud server 20 or a different cloud server) may execute a program to control the operation of each device, and each device may display a screen using a browsing function, etc. As a specific example, cloud server 20 may determine whether or not a gesture is appropriate.

[0185] <Summary of Embodiments> [Technical field] This invention relates to a program, an information terminal, and a system. [Background technology] A system is known in which customers register products themselves using a mobile device (see, for example, Patent Document 1). [Prior art document] [Patent] [Patent Document 1] Japanese Unexamined Patent Publication No. 2018-147252 [Overview of the prefecture] [Problems the invention aims to solve] However, the systems described above have room for improvement, for example, in dealing with malicious or unfamiliar customers. The present invention has been made in view of these circumstances and aims to provide a technology for suitably implementing product registration using a mobile terminal. [Means for solving the problem]

[0186] (1) One embodiment for solving the above-mentioned problems is a program that causes a computer to function as an information terminal (e.g., a mobile terminal 60) having a first imaging unit and a second imaging unit, wherein the computer functions as an output means, a first detection means (e.g., a holding motion detection function) that detects that an item is being held up based on a first image captured by the first imaging unit, and a second detection means (e.g., a confirmation motion detection function) that detects that the operator's action is a predetermined action based on a second image captured by the second imaging unit, and the output means outputs the detection result of the second detection means (e.g., a confirmation image GA02) when the first detection means detects that an item is being held up. According to the program in (1), product registration using a mobile device can be suitably performed. For example, the detection result can be used to confirm whether the customer is holding the product over the device with the intention of having the barcode scanned. It can also be confirmed that the customer was holding the product over the device without the intention of having the barcode scanned, thus allowing for suitable product registration using a mobile device.

[0187] (2) The program described in (1) is characterized in that the code acquisition means (e.g., code acquisition function) for acquiring a product code from the first captured image and the suitability determination means (e.g., suitability determination function for holding up the product) for determining whether the operator's way of holding up the product is appropriate based on the detection result of the second detection means, and the output means outputs information (e.g., small screen SG10, small screen SG20) based on the determination result of the suitability determination means when the first detection means detects that the product is being held up and the code acquisition means has not acquired a product code. According to the program in (2), for example, it is possible to determine whether the scan failed despite being correct, or because it was an invalid scan (a pseudo-scan).

[0188] (3) The program described in (2) is characterized in that it functions as a means for determining whether or not a store employee needs to be involved, and the means for determining whether or not a store employee needs to be involved (for example, the part that sets a flag indicating that a store employee needs to be confirmed) determines whether or not a store employee needs to be involved based on the number of times the means for determining whether or not the product is being held up has determined that it is not being held up properly. According to program (3), store staff are involved depending on the severity of the incident, such as the frequency of occurrences, so product registration using mobile terminals can be suitably implemented while balancing the effect (amount of improvement) and the burden (burden of store staff intervention).

[0189] (4) The program is as described in any one of (1) to (3), characterized in that the second captured image is an image of the operator's face, the predetermined action is the action of adjusting how the product is held, and the second detection means detects that the operator is adjusting how the product is held from the operator's facial expression or gaze in the second captured image. According to the program in (4), for example, it is possible to suitably detect that a customer was holding up a product without intending to have the barcode scanned.

[0190] (5) The output means is a program according to any one of (1) to (4), characterized in that it outputs a screen requesting permission from the operator for imaging by the second imaging unit, and outputs information regarding the deletion of the second image on the screen. According to program (5), for example, it becomes easier to gain customer understanding.

[0191] (6) One embodiment for solving the above-mentioned problems is an information terminal having a first imaging unit and a second imaging unit, comprising an output means, a first detection means for detecting that an item is being held up based on a first image captured by the first imaging unit, and a second detection means for detecting that the operator's action is a predetermined action based on a second image captured by the second imaging unit, wherein the output means outputs the detection result of the second detection means when the first detection means detects that an item is being held up. According to the information terminal in (6), the same effect as the program in (1) can be obtained.

[0192] (7) One embodiment for solving the above-mentioned problems is a system including an information terminal having a first imaging unit and a second imaging unit, comprising an output means, a first detection means for detecting that a product is being held up based on a first image captured by the first imaging unit, and a second detection means for detecting that the operator's action is a predetermined action based on a second image captured by the second imaging unit, wherein the output means outputs the detection result of the second detection means when the first detection means detects that a product is being held up. According to the information terminal in (7), the same effect as the program in (1) can be obtained.

[0193] Furthermore, the functions of each of the devices described above (headquarters server 10, store controller 30, transaction status management device 40, settlement device 50, mobile terminal 60, etc.) may be realized by recording the program to realize the functions of each device on a computer-readable recording medium, loading the program recorded on this recording medium into a computer system, and executing it. Here, "loading the program recorded on the recording medium into a computer system and executing it" includes installing the program into the computer system. Here, "computer system" includes hardware such as the OS and peripheral devices. Also, "computer system" may include multiple computer devices connected via a network including communication lines such as the Internet, WAN, LAN, and dedicated lines. Also, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into the computer system. Thus, the recording medium storing the program may also be a non-transient recording medium such as a CD-ROM. Also, the recording medium includes internal or external recording media that can be accessed from the distribution server for distributing the program. The program code stored on the distribution server's recording medium may be different from the program code in a format executable by the terminal device. In other words, the format in which the program is stored on the distribution server is irrelevant, as long as it can be downloaded from the distribution server and installed in a form that can be executed on the terminal device. Furthermore, the program may be divided into multiple parts, each downloaded at a different time and then combined on the terminal device, and each divided program may be distributed by a different distribution server. In addition, "computer-readable recording medium" includes volatile memory (RAM) within computer systems that act as servers or clients when a program is transmitted over a network, which retains the program for a certain period of time. Moreover, the above program may be intended to implement only a part of the functions described above.Furthermore, the above-mentioned functions may be realized in combination with programs already recorded in the computer system, such as so-called differential files (differential programs). [Explanation of Symbols]

[0194] 1…Sales system 10…Headquarters server 20…Cloud server 30…Store controller 40…Transaction status management device 50…Settlement device 60…Mobile terminal 70…Shopping cart 90…Dedicated bag

Claims

1. A program that causes a computer to function as an information terminal having a first imaging unit and a second imaging unit, The aforementioned computer, Output means, A first detection means that detects whether an item is being held up based on a first image captured by the first imaging unit, A second detection means detects whether the operator is adjusting how they are holding the product based on the operator's facial expression or gaze in the second captured image, which is an image of the operator's face captured by the second imaging unit. To make it function as, The output means is The detection result of the second detection means is output when the first detection means detects that an item is being held over it. A program characterized by the following features.

2. An information terminal having a first imaging unit and a second imaging unit, Output means and A first detection means for detecting that a product is being held up based on a first image captured by the first imaging unit, A second detection means detects whether the operator is adjusting how they are holding the product based on the operator's facial expression or gaze in the second captured image, which is an image of the operator's face captured by the second imaging unit. Equipped with, The output means is The detection result of the second detection means is output when the first detection means detects that an item is being held over it. An information terminal characterized by the following features.

3. A system including an information terminal having a first imaging unit and a second imaging unit, Output means and A first detection means for detecting that a product is being held up based on a first image captured by the first imaging unit, A second detection means detects whether the operator is adjusting how they are holding the product based on the operator's facial expression or gaze in the second captured image, which is an image of the operator's face captured by the second imaging unit. Equipped with, The output means is The detection result of the second detection means is output when the first detection means detects that an item is being held over it. A system characterized by the following features.

Images