Information processing device and program

Abstract

To provide an information processing unit capable of suppressing product registration unintended by the customer, and a program.SOLUTION: The information processing unit of the embodiment is an information processing unit operated by a customer moving in a store, and is equipped with a first acquisition unit to acquire product identification information that identifies products, a second acquisition unit to acquire a customer image, a registration unit to register product information of products identified by the product identification information acquired by the first acquisition unit as transaction targets, and a determination unit to determine whether to permit registration by the registration unit based on the customer image acquired by the second acquisition unit.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an information processing device and a program. [Background technology]

[0002] BACKGROUND ART Conventionally, a payment system has been known in stores such as supermarkets in which customers register products to be purchased by themselves using mobile terminals such as smartphones and tablet terminals (for example, Patent Document 1).

[0003] In this payment system, a customer picks up an item they wish to purchase at a sales counter and scans the barcode attached to the item with a scanner. The mobile terminal registers the item based on the scanned product code.

[0004] Once a customer has completed product registration for all items they wish to purchase, they proceed to the checkout area where a checkout device is installed. Based on the customer's operation, the mobile device sends checkout information, including the registered product information, to the checkout device. Based on the received checkout information, the checkout device performs checkout processing, including payment of the customer's purchase price. In the conventional checkout system described above, product registration is not performed at the checkout device where payment is made, so processing at the checkout device can be completed in a short time, thereby reducing the wait time at the register.

[0005] However, with the above-mentioned conventional technology, since customers who are unfamiliar with product registration register products while moving around the sales floor, there is a possibility that products will be registered unintentionally by the customer. For example, when a customer moves around the sales floor in a state where scanning by a scanner is possible, they may accidentally scan the barcodes attached to products displayed on the product shelves. Summary of the Invention [Problem to be solved by the invention]

[0006] The problem to be solved by the present invention is to provide an information processing device and a program that can prevent product registration that is not intended by a customer. [Means for solving the problem]

[0007] The information processing device of the embodiment is an information processing device operated by a customer moving around a store, and includes a first acquisition unit that acquires product identification information that identifies a product, a second acquisition unit that acquires a customer image that is an image of the customer, a registration unit that registers product information of a product identified by the product identification information acquired by the first acquisition unit as a trading object, and a judgment unit that judges whether or not to allow registration by the registration unit based on the customer image acquired by the second acquisition unit. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an overview of a transaction system including a mobile terminal according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing the main hardware configuration of the mobile terminal of the first embodiment. [Figure 3] FIG. 3 is a diagram showing the data structure of a product information section stored in the mobile terminal of the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a display on the display unit of the mobile terminal according to the first embodiment. [Figure 5] FIG. 5 is a block diagram showing the main functional configuration of the control unit of the mobile terminal of the first embodiment. [Figure 6] FIG. 6 is a flowchart showing the flow of the registration process by the control unit of the mobile terminal according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing the data structure of a product information section stored in a mobile terminal according to the second embodiment. [Figure 8] FIG. 8 is a flowchart showing the flow of registration processing by the control unit of the mobile terminal according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an information processing device and a program according to an embodiment will be described with reference to the drawings. Note that the present invention is not limited to the embodiments described below. For example, in the embodiments described below, an example will be described in which the information processing device is a mobile terminal such as a smartphone owned by a customer, but the information processing device may also be a mobile terminal such as a tablet terminal provided by a store.

[0010] In addition, in each of the embodiments described below, an example will be described in which the mobile terminal includes a first imaging unit that captures a product image for recognizing the product code and a second imaging unit that captures a customer image, which is an image of the customer capturing the product image, but this is not limiting. The mobile terminal may not include the first imaging unit and the second imaging unit, and may instead acquire the product image and customer image from an external device, such as a camera provided in a shopping cart.

[0011] (First embodiment) The following describes the information processing device and program of the first embodiment. Figure 1 is a diagram showing an overview of a checkout system 100. The checkout system 100 of this embodiment is applied to, for example, a store that sells products, such as a supermarket.

[0012] The accounting system 100 includes a mobile terminal 1, an accounting device 2, and a server device 3. The accounting device 2 and the server device 3 are connected to each other so that they can communicate with each other via a network N such as a local area network (LAN). The mobile terminal 1 is connected to the accounting device 2 and the server device 3 so that they can communicate with each other via a wireless LAN or the like.

[0013] The mobile terminal 1 is configured, for example, as a smartphone owned by customer C. A shopping application program (hereinafter also referred to as a "shopping app") is installed on the mobile terminal 1. By running the shopping app inside a store, the mobile terminal 1 becomes able to connect to the store's wireless LAN and can perform processes for shopping at the store. The mobile terminal 1 is an example of an information processing device.

[0014] For example, the mobile terminal 1 captures an image of a code symbol such as a barcode (hereinafter also referred to as a "product barcode") attached to a product by an operation of the customer C, and reads the product code indicated by the product barcode. The product code is information that identifies the product and is an example of product identification information. The image of the product barcode is an example of a product image for recognizing product identification information. The mobile terminal 1 may also read the product code using general object recognition (object recognition), in which case an image of the product's exterior becomes the product image for recognizing product identification information.

[0015] When the mobile terminal 1 determines that the read product code is a product to be traded, it registers the product based on the product code. Product registration here means acquiring product information (product name, price, etc.) based on the product code determined to be a product to be traded and registering (storing) it in a storage unit. The mobile terminal 1 functions as a product registration device.

[0016] The mobile terminal 1 captures an image of the customer C capturing the product barcode (hereinafter also referred to as "customer image") and determines whether or not the read product code is a transaction target based on the customer image. In other words, when the mobile terminal 1 reads a product code, it determines whether or not to permit product registration of the product identified by the product code based on the customer image.

[0017] For example, if customer C, who is taking an image of a product barcode, is looking at the screen displaying the product barcode being imaged, it is determined that customer C is intentionally reading the product code, and product registration based on that product code is permitted. Conversely, if customer C, who is taking an image of a product barcode, is not looking at the screen displaying the product barcode being imaged, it is determined that customer C is unintentionally reading the product code, and product registration based on that product code is not permitted.

[0018] This prevents the mobile terminal 1, which moves around the sales floor together with the customer C, from inadvertently registering products that the customer C does not intend to purchase. For example, it is possible to prevent the product barcode of a product displayed on a product shelf from being read and unintentionally registering the product. Whether the customer C, who is capturing an image of the product barcode, is looking at the screen displaying the captured product barcode is determined by analyzing the customer image using known image processing technology.

[0019] Multiple accounting devices 2 are installed in the checkout area of ​​the store. These accounting devices 2 are so-called self-service accounting devices where, for example, customer C pays for their purchases themselves. The accounting device 2 performs accounting processing related to payments by cash or cashless payment such as credit card payment based on the accounting information (including product registration information) received from the mobile terminal 1. The accounting processing is the process performed by the accounting device 2 to pay for the products purchased by customer C. The accounting device 2 may also be a point-of-sale (POS) terminal operated by a store clerk. The mobile terminal 1 may also be equipped with accounting processing functions such as credit card payment, in which case the accounting device 2 may not be installed.

[0020] The server device 3 functions as a store server that manages store information. The server device 3 receives payment information, such as product information for products paid for in the checkout process and payment method information, from each checkout device 2, and manages store sales information. The server device 3 also stores a product master that associates product codes with product information. The server device 3 transmits the product master to the mobile terminal 1 as appropriate.

[0021] Next, the configuration and functions of the mobile terminal 1 will be described in detail. Fig. 2 is a block diagram showing the main hardware configuration of the mobile terminal 1. The mobile terminal 1 includes a control unit 10, a display unit 11, an operation unit 12, an outer camera 13, an inner camera 14, a communication unit 15, and a storage unit 16. The control unit 10, the display unit 11, the operation unit 12, the outer camera 13, the inner camera 14, the communication unit 15, and the storage unit 16 are connected to one another via a bus 17 or the like.

[0022] The control unit 10 is configured as a computer including a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103. The CPU 101, the ROM 102, and the RAM 103 are connected to each other via a bus 17.

[0023] The CPU 101 controls the overall operation of the mobile terminal 1. The ROM 102 stores various programs, such as a program used to drive the CPU 101, and various data. The RAM 103 is used as a work area for the CPU 101, and loads various programs and data stored in the ROM 102 and the storage unit 16. The control unit 10 executes various control processes for the mobile terminal 1 by the CPU 101 operating in accordance with the control programs stored in the ROM 102 and the storage unit 16 and loaded in the RAM 103.

[0024] The RAM 103 also includes a product information section 1031. The product information section 1031 stores product information and the like relating to product codes determined to be the subject of a transaction among the product codes read by the outer camera 13. Product registration is executed by storing product information of the product to be traded in the product information section 1031. FIG. 3 is a diagram showing the data configuration of the product information section 1031. The product information section 1031 stores product information in association with a product code that identifies the product. The product information includes the product name, price, and quantity. Note that the product information stored in the product information section 1031 is not limited to the example shown in the figure, and may include, for example, an illustration showing the appearance of the product.

[0025] The display unit 11 is configured, for example, with a liquid crystal panel, and displays various information. For example, the display unit 11 displays product information of registered products. The display unit 11 also displays a registration screen for registering products that do not have product barcodes, and controls for inputting various information.

[0026] Furthermore, the display unit 11 displays a product image captured by the outer camera 13. Specifically, the display unit 11 displays a captured image including a product barcode captured by the outer camera 13 in order to read the product code. Fig. 4 is a diagram showing an example of a display screen displaying a captured image captured by the outer camera 13.

[0027] The display unit 11 displays an image captured by the outer camera 13 provided facing the rear side of the display unit 11. In other words, the display unit 11 displays the captured image of the rear side of the display unit 11 captured by the outer camera 13. At this time, a message 111 such as "Please align the barcode within the frame" and a frame 112 are displayed on the display unit 11.

[0028] When the product barcode is positioned within the frame 112, the outer camera 13 can recognize the product barcode and read the product code. While visually checking the display unit 11, the customer C moves the mobile terminal 1 so that the product barcode of the product to be purchased is aligned with the frame 112. When the product barcode is positioned within the frame 112 and the outer camera 13 reads the product code, the display unit 11 displays a screen showing product information and various controls related to the read product code.

[0029] The operation unit 12 is used to input various information into the control unit 10 and is composed of a touch panel and other components mounted on the surface of the display unit 11. When customer C operates an operator displayed on the display unit 11, the operation unit 12 inputs information corresponding to the operator into the control unit 10. For example, the operation unit 12 inputs the product code of a product that does not have a product barcode into the control unit 10. The operation unit 12 also inputs a transaction information transmission command to the control unit 10 to instruct the control unit 10 to send transaction information, including registered product information, to the accounting device 2.

[0030] The outer camera 13 is provided facing the rear side of the display unit 11. The outer camera 13 is equipped with an optical imaging element such as a CCD (Charge Coupled Device) and is an example of a first imaging unit that captures product images. The outer camera 13 functions as a scanner that reads product codes from product barcodes. Specifically, the outer camera 13 captures and recognizes the product barcode of the product purchased by customer C, and decodes the product barcode to read the product code. Note that the recognition of the product barcode based on the product image captured by the outer camera 13 and the decoding of the product barcode may be performed by the control unit 10. The outer camera 13 may also capture an image of the exterior of the product, in which case the product code may be read (recognized) using general object recognition (object recognition).

[0031] The in-camera 14 is provided facing the front surface of the display unit 11. The in-camera 14 is equipped with an optical imaging element such as a CCD (Charge Coupled Device), and is an example of a second imaging unit that captures an image of a customer. Specifically, the in-camera 14 captures an image of the face of the customer C who is imaging the product barcode while looking at the display unit 11.

[0032] The communication unit 15 is an interface for communicating with external devices such as the accounting device 2 and the server device 3. By connecting the control unit 10 to external devices via the communication unit 15, the control unit 10 can send and receive information (data) to and from the external devices.

[0033] The storage unit 16 is configured with a storage medium such as a HDD or flash memory, and maintains its stored contents even when the power is turned off. The storage unit 16 stores a control program 161 and a product master 162.

[0034] The control program 161 is a shopping application or the like that causes the mobile terminal 1 to function as a product registration device. The product master 162 is a master file that stores information about products sold by the store. As described above, the product master 162 stores product codes in association with product information including product names, prices, etc. As the products handled by the store change daily, the product master 162 is updated by the server device 3 as appropriate.

[0035] The storage unit 16 may be configured not to store the product master 162. In this case, the mobile terminal 1 can handle the product master of the server device 3 in the same way as in the configuration in which the product master 162 is stored, by appropriately communicating with the server device 3 and referring to the product master stored in the server device 3.

[0036] Next, a description will be given of the main functional configuration of the control unit 10 of the mobile terminal 1. Fig. 5 is a block diagram showing the functional configuration of the control unit 10 of the mobile terminal 1. The control unit 10 functions as a reception unit 1001, an acquisition unit 1002, a determination unit 1003, a registration unit 1004, a transmission / reception unit 1005, and a display control unit 1006, as a result of the CPU 101 operating in accordance with a control program stored in the ROM 102 or the storage unit 16. Note that each of these functions may be configured using hardware such as a dedicated circuit.

[0037] The reception unit 1001 receives various information from the operation unit 12. For example, the reception unit 1001 receives instructions from the operation unit 12 to launch a shopping app, start scanning, send transaction information, and so on. The shopping app launch instruction is information that instructs the device to launch the shopping app. The scan start instruction is information that instructs the device to operate the outer camera 13 and start scanning product barcodes, i.e., to start reading product codes. The transaction information send instruction is information that instructs the device to send transaction information, including information registered in the product information section 1031, to the checkout device 2.

[0038] The acquisition unit 1002 acquires product identification information that identifies a product. Specifically, the acquisition unit 1002 acquires, from the outer camera 13, a product code read by the outer camera 13. The acquisition unit 1002 functions as a first acquisition unit. Note that, when a decoding processing unit (not shown) of the control unit 10 recognizes a product barcode based on a product image captured by the outer camera 13 and decodes the product barcode, the acquisition unit 1002 can acquire the product code from the decoding processing unit.

[0039] The acquisition unit 1002 also acquires a customer image, which is an image of customer C capturing a product image for recognizing product identification information. Specifically, the acquisition unit 1002 acquires a customer image, such as the face of customer C, captured by the in-camera 14 from the in-camera 14. The customer image is an image of a product barcode captured by customer C using the out-camera 13. The acquisition unit 1002 also functions as a second acquisition unit.

[0040] The determination unit 1003 determines whether to permit registration by the registration unit 1004 based on the customer image acquired by the second acquisition unit (acquisition unit 1002). Specifically, the determination unit 1003 performs image processing on the customer image acquired by the acquisition unit 1002 from the in-camera 14, and determines whether customer C, who is the subject of the customer image, consciously read the product code. The determination unit 1003 then determines that the product code that is determined to have been consciously read by customer C is the subject of a transaction.

[0041] For example, if it is determined that customer C is looking at the display unit 11 in the customer image when the acquisition unit 1002 acquires the product code, i.e., in the customer image when customer C reads the product code, the determination unit 1003 permits product registration by the registration unit 1004. In other words, in this case, the determination unit 1003 determines that the product code acquired by the acquisition unit 1002 was intentionally read by customer C and is the product code of the product that customer C intends to purchase, and permits product registration.

[0042] On the other hand, if it is determined that customer C is not looking at the display unit 11 in the customer image when the acquisition unit 1002 acquires the product code, the determination unit 1003 does not permit product registration by the registration unit 1004. In other words, in this case, the determination unit 1003 determines that the product code acquired by the acquisition unit 1002 is a product code that was read unintentionally by customer C, and does not permit product registration.

[0043] Here, the determination of whether customer C is looking at the display unit 11 is performed using a known image processing technique. For example, the determination unit 1003 extracts a facial area from the customer image, and if the eye area can be recognized, identifies the viewing direction from the eye area or the position of the pupils. If the eye area cannot be recognized, the determination unit 1003 identifies the viewing direction from the position of the nose or mouth. Then, the determination unit 1003 determines whether customer C is looking at the display unit 11 based on the identified viewing direction. Furthermore, if the determination unit 1003 cannot extract a facial area from the customer image, it determines that customer C is not looking at the display unit 11.

[0044] The determination unit 1003 may use another method to determine whether the product code acquired by the acquisition unit 1002 was intentionally read by the customer C. For example, if the determination unit 1003 recognizes that the image captured by the outer camera 13 is an image of a person's hand holding a product, it may determine that the product barcode was intentionally read by the customer C. This is because, when customer C captures an image of the product barcode of a product he or she wishes to purchase, he or she is likely to pick up the product and then capture the image of the product barcode.

[0045] The determination of whether customer C intentionally scanned the product barcode may be performed using machine learning. Machine learning is a well-known technique that uses a large amount of learning data to learn patterns and then uses the learning results to infer unknown data. For example, the determination unit 1003 may use a learning model that infers whether the product with the scanned product code was purchased based on a customer image.

[0046] The registration unit 1004 registers, as a transaction object, the product information of the product identified by the product identification information acquired by the first acquisition unit. Specifically, the registration unit 1004 executes product registration for the product for which registration of product information has been permitted by the determination unit 1003, i.e., the product determined by the determination unit 1003 to be a transaction object. More specifically, the registration unit 1004 reads out, from the product master 162, product information corresponding to the product code determined to have been intentionally read by the customer C, and registers the product information in the product information unit 1031. When the registration of one product is completed, the registration unit 1004 outputs an alert sound from a speaker unit (not shown) to notify the customer C that the product has been registered.

[0047] The registration unit 1004 may also store product information of products determined not to be eligible for trading in the product information unit 1031. In this case, the registration unit 1004 stores product information of products determined to be eligible for trading separately from product information of products determined not to be eligible for trading. In this specification, storing product information of products determined not to be eligible for trading in the product information unit 1031 is not included in the concept of product registration.

[0048] Furthermore, if the product master 162 is not stored in the storage unit 16, the registration unit 1004 can also communicate with the server device 3 that stores the product master 162 and acquire product information from the server device 3.

[0049] The transmission / reception unit 1005 transmits and receives information to and from external devices. For example, the transmission / reception unit 1005 transmits transaction information to the checkout device 2, including product information registered in the product information unit 1031. The transmission / reception unit 1005 also receives a payment completion notification from the checkout device 2, indicating that payment has been completed during the transaction process. The transmission / reception unit 1005 also receives the product master from the server device 3.

[0050] The display control unit 1006 causes various information to be displayed on the display unit 11. For example, the display control unit 1006 causes the display unit 11 to display product information registered by the registration unit 1004. The display control unit 1006 also causes the display unit 11 to display product information for products for which registration by the registration unit 1004 has been denied by the determination unit 1003, along with information indicating that the product will not be registered. The display control unit 1006 also causes the display unit 11 to display product images captured by the outer camera 13. The display control unit 1006 also causes the display unit 11 to display other information, such as various controls for inputting information such as instructions to launch an app or to send transaction information.

[0051] Next, a description will be given of the process for registering a product by the mobile terminal 1 configured as described above. Fig. 6 is a flowchart showing the flow of the registration process by the control unit 10 of the mobile terminal 1. Here, it is assumed that the mobile terminal 1 has a shopping app running.

[0052] The control unit 10 determines whether the reception unit 1001 has received a scan start instruction from the operation unit 12 (step S1), and if the scan start instruction has not been received (N in step S1), the control unit 10 returns to the processing of step S1 and waits.

[0053] When the reception unit 1001 receives a scan start instruction (Y in step S1), the control unit 10 determines whether the acquisition unit 1002 has acquired a product code from the outer camera 13 (step S2), and if the product code has not been acquired (N in step S2), the control unit 10 returns to the processing of step S2 and waits. Note that the processing of step S1 may be omitted, and in this case, the control unit 10 may execute the processing from step S2 onwards when the shopping app is launched.

[0054] When the acquisition unit 1002 acquires the product code (Y in step S2), it acquires a customer image from the in-camera 14 (step S3). The acquisition unit 1002 acquires an image of the customer C capturing an image of the product barcode using the out-camera 13 as the customer image.

[0055] Next, the determination unit 1003 determines whether the customer image acquired by the acquisition unit 1002 is an image of customer C looking at the display unit 11 (step S4). If the image is not of customer C looking at the display unit 11 (N in step S4), the display control unit 1006 causes the display unit 11 to display an unregistered product display (step S5). The unregistered product display is a display indicating that product registration has not been performed based on the product code acquired by the acquisition unit 1002, i.e., the product code read by the outer camera 13. For example, the unregistered product display displays a message indicating that product registration has not been performed based on the product code read by the outer camera 13, along with product information for the product associated with the product code.

[0056] After displaying the unregistered product in step S5, the control unit 10 returns to the processing of step S1. Displaying the unregistered product prevents customer C from mistakenly recognizing an unregistered product as a registered product. For example, even if customer C reads the product code of a product he or she is trying to purchase without visually checking the display unit 11, displaying the unregistered product allows customer C to recognize that the product is not registered. Therefore, customer C can read the product barcode again for the product displayed as an unregistered product.

[0057] In this embodiment, when a product is registered, an audible alarm is emitted to notify customer C that the product has been registered. As a result, if customer C reads the product code without visually checking the display unit 11, customer C can recognize that the product has not been registered because no audible alarm is emitted. However, the audible alarm may not be heard in a noisy sales floor. Alternatively, if the mobile terminal 1 is a smartphone and is in silent mode, the audible alarm is not emitted. Therefore, when registering a product using customer C's smartphone in a sales floor, it may be difficult for customer C to recognize that the product has not been registered based on the audible alarm. In such an environment, it is particularly effective to notify customer C that the product has not been registered by displaying an "unregistered" message.

[0058] Additionally, in this embodiment, customer C issues a command to start scanning each time a product code is read. Therefore, customer C visually checks display unit 11 each time a product code is read. This prevents customer C from overlooking the unregistered product display, and customer C can more reliably recognize that the product is not registered.

[0059] In step S4, if the customer image acquired by the acquisition unit 1002 is an image of customer C looking at the display unit 11 (Y in step S4), the registration unit 1004 registers the product (step S6). Next, the display control unit 1006 causes the display unit 11 to display product information for the registered product (step S7). At this time, the display control unit 1006 also causes the display unit 11 to display operators for entering a command to start scanning and an operator for issuing a command to send transaction information.

[0060] Next, the control unit 10 determines whether the reception unit 1001 has received an instruction to send transaction information from the operation unit 12 (step S8), and if no instruction to send transaction information has been received (N in step S8), the control unit 10 returns to step S1.

[0061] When the reception unit 1001 receives the transaction information transmission instruction (Y in step S8), the transmission / reception unit 1005 reads the information stored in the product information section 1031 and sends transaction information including this information to the transaction device 2 (step S9). The transaction information may be sent to the transaction device 2 via network N, or may be sent to the transaction device 2 using short-range wireless communication such as NFC (Near Field Communication). The control unit 10 then ends the registration process.

[0062] Through the above registration process, the mobile terminal 1 registers only the products whose product codes are determined to have been read intentionally by customer C. This prevents unintended product registration by customer C, thereby reducing the occurrence of discrepancies between the products that customer C places in the shopping cart as purchased products and the products that are actually registered. This makes it possible to prevent products purchased by customer C from being purchased at an incorrect transaction price.

[0063] (Second embodiment) Next, we will explain the second embodiment. The second embodiment differs from the first embodiment in that, after the acquisition unit 1002 acquires a product code but does not include the product in the transaction, the product information for the product is displayed on the display unit 11 for customer C to review after the transaction information transmission instruction is issued. Below, explanations of configurations and functions that are similar to those of the first embodiment may be omitted. This is to avoid unnecessary redundancy in the following explanation and to make it easier for those skilled in the art to understand.

[0064] The product information unit 1031 in this embodiment stores product information for all products associated with product codes read by the outer camera 13. Specifically, the product information unit 1031 stores product information for products that are subject to trade and products that are not subject to trade, distinguishing between these products. FIG. 7 is a diagram showing the data configuration of the product information unit 1031. The product information unit 1031 stores product information and a trade target flag in association with the product code that identifies the product.

[0065] The product information includes the product name and price. The trading target flag is information indicating whether the product has been determined to be a trading target by the determination unit 1003. The trading target flag is registered as "1" for products determined to be a trading target, and "0" for products determined not to be a trading target.

[0066] Next, the registration process in this embodiment will be described. Fig. 8 is a flowchart showing the flow of the registration process by the control unit 10 of the mobile terminal 1. The same processes as in the first embodiment are denoted by the same reference numerals. Also, as in the first embodiment, it is assumed that the mobile terminal 1 has a shopping app running.

[0067] The processes of steps S1 to S4 are the same as those of the first embodiment. In the process of step S4, if the image does not show customer C looking at the display unit 11 (N in step S4), the registration unit 1004 performs transaction non-eligible registration (step S11). Transaction non-eligible registration is a process of storing product information of a product related to the product code acquired by the acquisition unit 1002 in association with information indicating that the product is not a transaction target.

[0068] Specifically, the registration unit 1004 reads out the product information corresponding to the product code acquired by the acquisition unit 1002 from the product master 162 and registers it in the product information unit 1031. At this time, the registration unit 1004 registers "0" indicating that the product is not a transaction target in the transaction target flag associated with the product information. After registering the product as not a transaction target in step S11, the control unit 10 returns to the processing of step S1.

[0069] In the process of step S4, if the image shows the customer C looking at the display unit 11 (Y in step S4), the control unit 10 executes the processes of steps S6 to S8 in the same manner as in the first embodiment.

[0070] If an instruction to send transaction information is issued in step S8 (Y in step S8), the display control unit 1006 displays product information for non-transaction eligible items on the display unit 11 (step S12). Non-transaction eligible item information is displayed for items stored in the product information unit 1031 that have a transaction eligible flag set to "0." In this case, the display control unit 1006 may display a message in addition to the product information. For example, the display control unit 1006 may display a message encouraging the user to register an unregistered item, such as "If you wish to purchase the displayed item, please scan the barcode again."

[0071] Next, the control unit 10 determines whether the receiving unit 1001 has received a confirmation input from the operation unit 12 (step S13). The confirmation input is input by customer C confirming the display of a product that is not subject to a transaction and inputting information indicating that the product related to the displayed product information will not be purchased. In other words, the confirmation input is input by customer C declaring that the customer C will not purchase the product that is registered in the product information unit 1031 as not subject to a transaction. The confirmation input may be input by any method, but may be input by, for example, operating a confirmation button (operator) that is displayed together with the product information in the display of a product that is not subject to a transaction.

[0072] When the reception unit 1001 receives confirmation input from the operation unit 12 (Y in step S13), the transmission / reception unit 1005 reads out information stored in the product information unit 1031 for which the transaction object flag is registered as "1" and sends transaction information including this information to the checkout device 2 (step S9). In other words, the transmission / reception unit 1005 sends information about the products registered as transaction objects in the product information unit 1031 to the checkout device 2. The control unit 10 then ends the registration process.

[0073] If the reception unit 1001 does not receive a confirmation input in the processing of step S13 (N in step S13), the control unit 10 determines whether the reception unit 1001 has received a correction input (step S14). The correction input is input when customer C checks a product display that is not subject to a transaction and purchases a product related to the displayed product information, and is input by inputting information that identifies the product to be purchased. In other words, the correction input is input by inputting information that identifies the product that customer C will purchase from among the products registered in the product information unit 1031 as not subject to a transaction. The method of operating the correction input is not critical, but for example, it may be performed by operating a purchase button (operator) that is displayed in association with the product information in the product display that is not subject to a transaction.

[0074] If no correction input is made (N in step S14), the control unit 10 returns to the processing of step S13. If correction input is made (Y in step S14), the registration unit 1004 performs additional product registration for the product that has been corrected as a product to be purchased (step S15). Specifically, the registration unit 1004 rewrites the transaction object flag in the product information unit 1031 for the product designated as a product to be purchased from "0" to "1." The product whose transaction object flag has been rewritten to "1" is registered as a transaction object. In other words, the product is registered. Once the additional product registration is made, the control unit 10 returns to the processing of step S12.

[0075] As with the first embodiment, the above registration process makes it possible to prevent transactions for products purchased by customer C at an incorrect transaction price. Furthermore, before sending transaction information to the checkout device 2, the mobile terminal 1 displays product information for products associated with product codes acquired by the acquisition unit 1002 that are determined not to be eligible for the transaction. The mobile terminal 1 then accepts product registration for the displayed product information. This allows customer C to confirm that the product they are attempting to purchase is not registered and to register the product before paying for it. Therefore, this also makes it possible to prevent transactions at an incorrect transaction price.

[0076] As described above, the mobile terminal 1 in each embodiment is an information processing device operated by a customer C moving around a store, and is equipped with a first acquisition unit (acquisition unit 1002) that acquires product identification information (product code) that identifies a product, a second acquisition unit (acquisition unit 1002) that acquires a customer image that is an image of customer C taking a product image for recognizing the product identification information, a registration unit 1004 that registers product information of a product identified by the product identification information acquired by the first acquisition unit as a transaction object, and a judgment unit 1003 that judges whether or not to allow registration by the registration unit 1004 based on the customer image acquired by the second acquisition unit.

[0077] This allows the mobile terminal 1 to prevent product registration that is not intended by the customer C, and makes it possible to prevent transactions at an incorrect transaction amount.

[0078] Moreover, the mobile terminal 1 of each embodiment further includes a display unit 11 that displays product information of a product that the determination unit 1003 has determined not to permit registration by the registration unit 1004 .

[0079] This allows customer C to recognize that the product that customer C is trying to purchase is not eligible for the transaction if the determination unit 1003 determines that the product is not eligible for the transaction. This allows customer C to register the displayed product again, further preventing transactions at an incorrect transaction price.

[0080] Furthermore, the mobile terminal 1 in each embodiment further includes a first imaging unit (outer camera 13) that captures product images and a second imaging unit (inner camera 14) that captures customer images, and the display unit 11 displays the product images captured by the first imaging unit.

[0081] This allows the mobile terminal 1 to provide the customer C with good operability when reading the product code.

[0082] In addition, the judgment unit 1003 of the mobile terminal 1 in each embodiment judges to allow registration by the registration unit 1004 when the customer image acquired by the second acquisition unit (acquisition unit 1002) is an image of customer C looking at the display unit 11.

[0083] This allows the mobile terminal 1 to easily determine whether or not the product associated with the product code acquired by the acquisition unit 1002 is a product to be traded. Specifically, by noting that when customer C reads the product code, customer C visually checks the display unit 11 displaying the product image, the determination unit 1003 can easily determine whether or not the product is a product to be traded. More specifically, the determination unit 1003 can determine whether or not to permit registration by the registration unit 1004 by utilizing a configuration that improves operability when customer C reads the product code, i.e., the display unit 11 displaying the product image.

[0084] Furthermore, the determination unit 1003 of the mobile terminal 1 in each embodiment recognizes the orientation of the face of customer C in the customer image.

[0085] This allows the mobile terminal 1 to use known image processing technology to determine whether the product associated with the product code acquired by the acquisition unit 1002 is a product subject to transaction, making it easier to make this determination.

[0086] In the above embodiment, the control program executed by the mobile terminal 1 may be provided by being recorded on a computer-readable recording medium such as a CD-ROM. Also, the control program executed by the mobile terminal 1 of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network, or may be provided via a network such as the Internet.

[0087] Although the embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. [Explanation of symbols]

[0088] 1. Mobile terminal (information processing device) 11 Display section 13 Outer camera (first imaging unit) 14 In-camera (second imaging unit) 1002 Acquisition Department (1st Acquisition Department, 2nd Acquisition Department) 1003 Judgment section 1004 Registration Department [Prior art documents] [Patent documents]

[0089] [Patent Document 1] Patent Publication No. 2021-124894

Claims

1. An information processing device operated by a customer moving around a store, a first acquisition unit that acquires product identification information that identifies a product; a second acquisition unit that acquires a customer image that is an image of a customer; a registration unit that registers product information of a product identified by the product identification information acquired by the first acquisition unit as a trading object; a determination unit that determines whether or not to permit registration by the registration unit based on the customer image acquired by the second acquisition unit; An information processing device comprising:

2. a display unit that displays product information of a product that has been denied registration by the registration unit by the determination unit, The information processing device according to claim 1 .

3. a first imaging unit that captures a commodity image for recognizing the commodity identification information; a second imaging unit that captures the customer image, The display unit displays the product image captured by the first imaging unit. The information processing device according to claim 2 .

4. the determination unit determines to permit registration by the registration unit when the customer image acquired by the second acquisition unit is an image of the customer looking at the display unit. The information processing device according to claim 3 .

5. the determination unit recognizes the orientation of the customer's face in the customer image; The information processing device according to claim 4 .

6. A program for controlling, by a computer, an information processing device operated by a customer moving around a store, The computer a first acquisition unit that acquires product identification information that identifies a product; a second acquisition unit that acquires a customer image that is an image of a customer; a registration unit that registers product information of a product identified by the product identification information acquired by the first acquisition unit as a trading object; a determination unit that determines whether or not to permit registration by the registration unit based on the customer image acquired by the second acquisition unit; A program that makes it work.

Images