Information processing device and information processing program

Abstract

This system prevents errors in registering products for transactions from being left unaddressed, especially when the customer performs the registration process themselves. [Solution] The information processing device of the embodiment comprises an acquisition means, a detection means, and a generation means. The acquisition means acquires an image of the inside of a container for holding goods that have been registered as the subject of a transaction. The detection means detects goods placed in the container based on the image acquired by the acquisition means. A screen is generated in which an image that allows the user to identify the registered goods is superimposed on the image acquired by the acquisition means. When a new item placed in the container is detected by the detection means, if the item to be traded is registered at a predetermined registration timing relative to the detection timing, a new screen is generated that allows the user to identify the new item placed in the container.

Description

Technical Field

[0001] Embodiments of the present invention relate to an information processing apparatus and an information processing program.

Background Art

[0002] There is known a system that performs transaction processing using an information communication terminal owned by a customer, such as a smartphone, or an information communication terminal lent to the customer from a store or the like as a user interface. Alternatively, there is known a transaction processing system that performs transaction processing using an information communication terminal attached to a shopping cart provided in a store as a user interface with the customer.

[0003] In these transaction processing systems, since the operation for registering a product as a transaction target is performed by the customer anywhere in the store, even if the product registration is not properly performed due to an incorrect operation, there are cases where the customer or the store clerk cannot notice it. Under such circumstances, it has been desired to prevent an operation error from being overlooked in a situation where the operation for registering a product as a transaction target is performed by the customer.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The problem to be solved by the present invention is to provide an information processing apparatus and an information processing program that can prevent an operation error from being overlooked in a situation where the operation for registering a product as a transaction target is performed by the customer.

Means for Solving the Problems

[0006] The information processing device of this embodiment comprises an acquisition means, a detection means, and a generation means. The acquisition means acquires an image of the inside of a container for holding goods that have been registered as the subject of a transaction. The detection means detects goods placed in the container based on the image acquired by the acquisition means. The generation means generates a screen in which an image that allows the user to identify the registered goods is superimposed on the image acquired by the acquisition means. When a newly placed goods in the container is detected by the detection means, and the goods to be traded are registered at a predetermined registration timing relative to the detection timing, the generation means generates a new screen that allows the user to identify the newly placed goods in the container. [Brief explanation of the drawing]

[0007] [Figure 1] A block diagram illustrating the schematic configuration of a transaction processing system according to one embodiment. [Figure 2] A block diagram showing the main circuit configuration of the transaction processing device in Figure 1. [Figure 3] A diagram schematically representing the structure of transaction data. [Figure 4] A schematic diagram representing the structure of a single data record included in a tag table. [Figure 5] A block diagram showing the main circuit configuration of the attendant terminal in Figure 1. [Figure 6] A block diagram showing the main circuit configuration of the cart terminal in Figure 1. [Figure 7] A perspective view showing an example of how a cart terminal is attached to a shopping cart. [Figure 8] Transaction processing flowchart. [Figure 9] A flowchart for the tag management process. [Figure 10] A diagram showing an example of a camera image. [Figure 11] A diagram showing an example of the resulting crop. [Figure 12] A diagram showing an example of a camera image. [Figure 13] A diagram showing an example of the resulting crop. [Figure 14] An example of an AR screen is shown in this diagram. [Figure 15] An example of an AR screen is shown in this diagram. [Figure 16] An example of an AR screen is shown in this diagram. [Figure 17] A diagram showing an overall overview of a monitoring screen as an example. [Figure 18] A diagram showing a detail of one of the display areas in Figure 17. [Modes for carrying out the invention]

[0008] An example of an embodiment will be described below with reference to the drawings. Figure 1 is a block diagram showing the schematic configuration of the transaction processing system 1 according to this embodiment. The transaction processing system 1 is configured such that the transaction processing device 100, accounting machine 200, attendant terminal 300, and cart terminal 400 can communicate with each other via the communication network 2. Communication network 2 can use the internet, VPN (virtual private network), LAN (local area network), public communication network, mobile communication network, etc., individually or in appropriate combinations. As an example, communication network 2 can be used in combination with the internet and a mobile communication network. Note that the transaction processing unit 100, accounting machine 200, attendant terminal 300, and cart terminal 400 may be included in any number of each in the transaction processing system 1, but only one of each is shown in Figure 1.

[0009] The transaction processing device 100 is an information processing device that performs information processing for providing a transaction processing service for processing a merchandise sales transaction between a customer and a store in accordance with operations performed by a customer of the store using the accounting machine 200 and the cart terminal 400 as user interface terminals. The transaction processing device 100 is realized, for example, as a cloud server and provides transaction processing services at multiple stores. The transaction processing device 100 may also be realized, for example, as a local server and provide transaction processing services at only one store.

[0010] The accounting machine 200 is installed in the store and executes accounting processing related to the accounting of transactions processed by the transaction processing device 100. The accounting machine 200 receives operations by an operator during accounting processing. The operator of the accounting machine 200 is mainly a customer. In some cases, a store clerk may be the operator of the accounting machine 200.

[0011] The attendant terminal 300 is an information processing terminal operated by a store clerk. The attendant terminal 300 is a terminal device for a user interface for information processing to support the work of the store clerk related to transactions processed by the transaction processing system 1. The work of the store clerk is, for example, to monitor the execution status of a transaction being processed and appropriately support the customer.

[0012] The cart terminal 400 is an information processing terminal attached to a shopping cart (hereinafter referred to as a cart) provided in the store. The cart terminal 400 is lent to the customer together with the cart. The cart terminal 400 is a terminal device that receives operations by the customer for transaction processing by the transaction processing device 100. The cart terminal 400 may include an information communication terminal lent to the customer by the store and carried and used by the customer.

[0013] Figure 2 is a block diagram showing the main circuit configuration of the transaction processing device 100. The transaction processing device 100 includes a processor 101, a main storage unit 102, an auxiliary storage unit 103, a communication unit 104, and a transmission line 105, etc. The processor 101, the main storage unit 102, the auxiliary storage unit 103, and the communication unit 104 are able to communicate with each other via the transmission line 105.

[0014] By connecting the processor 101, the main storage unit 102, and the auxiliary storage unit 103 with a transmission line 105, a computer is configured to perform information processing for controlling the transaction processing device 100. The processor 101 corresponds to the central part of the computer described above. The processor 101 performs information processing to control each part in order to realize various functions as a transaction processing device 100, in accordance with information processing programs such as the operating system and application programs.

[0015] The main memory unit 102 corresponds to the main memory portion of the computer described above. The main memory unit 102 includes a read-only memory area and a rewritable memory area. The main memory unit 102 stores a portion of the information processing program described above in the read-only memory area. The main memory unit 102 may also store data necessary for the processor 101 to perform processing to control each part in the read-only memory area or the rewritable memory area. The main memory unit 102 uses the rewritable memory area as a work area for the processor 101.

[0016] The auxiliary storage unit 103 corresponds to the auxiliary storage portion of the computer described above. The auxiliary storage unit 103 can utilize, for example, an EEPROM (electric erasable programmable read-only memory), an HDD (hard disk drive), an SSD (solid state drive), or various other well-known storage devices. The auxiliary storage unit 103 stores data used by the processor 101 in performing various processes and data generated by the processing performed by the processor 101. The auxiliary storage unit 103 may also store the information processing program described above. In this embodiment, the auxiliary storage unit 103 stores the transaction processing program PRA, which is one of the information processing programs. The transaction processing program PRA is an application program that describes the procedures for information processing (hereinafter referred to as transaction processing) for registering goods that are the subject of a transaction (hereinafter referred to as transaction goods) and settling the payment for registered transaction goods. The transaction processing program PRA also describes the procedures for tag management processing, which will be described later. A portion of the storage area of ​​the auxiliary storage unit 103 is used as an area for storing transaction data DAA and tag table TAA. Transaction data DAA is data that represents the details of a single transaction. Tag table TAA is a data table for managing tags that are set for a single transaction as described later.

[0017] The communication unit 104 performs communication processing for data communication via the communication network 2. For example, an existing wired communication device for the internet can be used as the communication unit 104. Alternatively, a wireless communication device connected to the communication network 2 via wireless communication may be used as the communication unit 104, either in place of or in addition to the wired communication device.

[0018] The transmission line 105 includes an address bus, a data bus, and control signal lines, and transmits data and control signals exchanged between the connected parts.

[0019] Figure 3 is a schematic diagram illustrating the structure of the transaction data DAA. Transaction data DAA is generated for each transaction being processed by the transaction processing unit 100 and stored in the auxiliary storage unit 103. Thus, there may be cases where no transaction data DAA is stored in the auxiliary storage unit 103, or multiple transaction data DAA may be stored in the auxiliary storage unit 103 simultaneously.

[0020] The transaction data DAA includes fields FAA, FAB, and FAC. The transaction data DAA may contain any number of fields starting from field FAD. Field FAA is set to the transaction code as the identifier of the transaction in question. Field FAB is set to the terminal code as the identifier of the cart terminal 400 used by the customer making the transaction in question. Field FAC is set to the customer code as the identifier of the customer making the transaction in question. If there are products already registered as transaction products, fields FAD, FAE, ... associated with each of those transaction products are added to the transaction data DAA. Fields FAD, FAE, ... each contain product data for a separate transaction product. The product data includes the product code and quantity as identifiers for the transaction product in question. The product data may also include various other information, such as product name, unit price, and discount information.

[0021] Figure 4 schematically represents the structure of a single data record REA contained in the tag table TAA. The tag table TAA contains data records REA, which have the structure shown in Figure 4, associated with each of the tags set for a single transaction as described below. In other words, the number of data records REA contained in the tag table TAA varies depending on the number of tags set.

[0022] The data record REA includes fields FBA, FBB, FBC, and FBD. Field FBA contains the tag codes, which serve as identifiers for each tag set for a single transaction. The tag codes are expected to be, for example, numbers representing the order in which the tags are set. Field FBC represents the display position of the AR tag corresponding to the associated tag on the AR (augmented reality) screen, which will be described later. Field FBA also contains a flag indicating whether or not to display the AR tag corresponding to the associated tag on the AR screen.

[0023] Now, as the hardware for the transaction processing device 100, for example, a general-purpose server device can be used. Generally, the transfer of the transaction processing device 100 is carried out with the transaction processing program PRA stored in the auxiliary storage unit 103, but without the transaction data DAA and tag table TAA. However, the hardware in a state where the transaction processing program PRA is not stored in the auxiliary storage unit 103, or in a state where a different version of the same type of application program is stored in the auxiliary storage unit 103, and the transaction processing program PRA may be transferred separately. Furthermore, the transaction processing device 100 may be configured by writing the transaction processing program PRA to the auxiliary storage unit 103 in response to the operation of any operator. The transfer of the transaction processing program PRA can be carried out by recording it on a removable recording medium such as a magnetic disk, magneto-optical disk, optical disk, or semiconductor memory, or by communication over a network.

[0024] Figure 5 is a block diagram showing the main circuit configuration of the attendant terminal 300. The attendant terminal 300 includes a processor 301, a main memory unit 302, an auxiliary memory unit 303, a touch panel 304, a communication unit 305, and a transmission line 306, among other things.

[0025] The general functions of the processor 301, main memory unit 302, auxiliary memory unit 303, communication unit 305, and transmission line 306 are the same as those of the processor 101, main memory unit 102, auxiliary memory unit 103, and transmission line 105, so their explanation will be omitted. However, the auxiliary memory unit 303 stores the attendant terminal program PRB instead of the transaction processing program PRA. The attendant terminal program PRB is an application program that describes the information processing procedure of the processor 301 to operate the attendant terminal 300 as a user interface for store employees that supports transactions processed by the transaction processing device 100.

[0026] The touch panel 304 displays a screen for presenting information to the operator of the attendant terminal 300. The touch panel 304 also accepts input from the operator via touch operations on the screen. The hardware for the attendant terminal 300 may include, for example, a stationary computer device. Alternatively, the hardware for the attendant terminal 300 may include a tablet-type information processing device or a portable information processing device such as a smartphone.

[0027] Figure 6 is a block diagram showing the main circuit configuration of the cart terminal 400. The cart terminal 400 includes a processor 401, a main memory unit 402, an auxiliary memory unit 403, a touch panel 404, a camera 405, an interface unit 406, a wireless communication unit 407, and a transmission line 408, among other things.

[0028] The general functions of the processor 401, main memory unit 402, auxiliary memory unit 403, and transmission line 408 are equivalent to those of the processor 101, main memory unit 102, auxiliary memory unit 103, and transmission line 105. The general functions of the touch panel 404 are also equivalent to those of the touch panel 304. However, the auxiliary memory unit 403 stores the cart terminal program PRC instead of the transaction processing program PRA. The cart terminal program PRC is an application program that describes the information processing procedure of the processor 401 to operate the cart terminal 400 as a user interface for transaction processing by the transaction processing device 100.

[0029] Camera 405 captures an image by photographing a range that encompasses the entire interior space of the shopping basket placed on the cart. The image obtained by camera 405 will be referred to as the camera image below. The interface unit 406 is connected to external devices such as the scanner 499. The interface unit 406 interfaces with the connected external devices for data exchange. An existing USB (universal serial bus) controller can be used as the interface unit 406. The scanner 499 is mounted on the cart and optically scans 1D and 2D barcodes.

[0030] The wireless communication unit 407 performs communication processing for wireless data communication via the communication network 2. For example, an existing wireless communication device for a wireless LAN can be used as the wireless communication unit 407. Alternatively, a communication unit connected to the communication network 2 via a wired connection may be used instead of, or in addition to, the wireless communication unit 407. The basic hardware for the cart terminal 400 is expected to be, for example, the hardware of a tablet-type information processing device.

[0031] Figure 7 is a perspective view showing an example of how the cart terminal 400 is attached to the cart. The cart 900 comprises a caster section 910, a handle frame section 920, and a basket support section 930. The caster section 910 has four wheels 911 for smoothly moving the cart 900 on the floor. The wheels 911 are mounted on the frame 912 so as to be rotatable around a vertical axis.

[0032] The handle frame section 920 includes a pair of vertical frames 921, 921 and a handlebar 922. The vertical frames 921, 921 are erected above the two wheels of the caster section 910. The handlebar 922 connects the upper ends of the vertical frames 921, 921. The basket support section 930 is provided horizontally from the middle of the handle frame section 920. The basket support section 930 holds the shopping basket SB for storing goods. The caster section 910 can also hold the shopping basket SB on top of the frame 912.

[0033] A customer using the cart 900 is typically positioned in front of the handle frame 920 in Figure 7. The customer then pushes the cart 900 while gripping the handlebar 922. In this case, the direction in which the basket support 930 protrudes relative to the handle frame 920 is the direction in which the cart 900 moves forward.

[0034] A scanner 499 is attached to the middle of the handlebar 922. A pole 940 is attached to one of the vertical frames 921 such that its tip is positioned above the handlebar 922. The aforementioned cart terminal 400 is attached to the tip of this pole 940, with the touch panel 404 facing away from the direction of travel. The camera 405 of the cart terminal 400 is positioned facing the back of the touch panel 404 and is not shown in Figure 7. This camera 405 can take an overhead, bird's-eye view of the inside of the shopping basket SB, which is held by the basket support 930. A battery case 950 is mounted between vertical frames 921, 921 on the lower end of the handle frame section 920. The battery case 950 houses a battery used as an external power source for the cart terminal 400.

[0035] Next, the operation of the transaction processing system 1 configured as described above will be explained. Note that the content of the various processes described below is merely an example, and it is possible to change the order of some processes, omit some processes, or add other processes as appropriate. For example, in the following explanation, some processes have been omitted in order to clearly illustrate the characteristic operation of this embodiment. For example, if an error occurs, processing to deal with that error may be performed, but such processing has been omitted from the description.

[0036] The customer performs a predetermined operation to begin using the unused cart terminal 400. As part of the operation to begin using the cart terminal 400, the customer may also perform an operation to enter a customer code into the cart terminal 400. Upon receiving such an operation from the cart terminal 400, the processor 401 requests a check-in from the transaction processing unit 100. That is, the processor 401 sends, for example, request data for the check-in request to the transaction processing unit 100 from the wireless communication unit 407 to the communication network 2. The processor 401 includes the check-in data stored in the auxiliary storage unit 403 and the terminal code of the cart terminal 400 in the above request data. The terminal code is pre-assigned to the relevant cart terminal 400 in a way that allows it to be identified from other cart terminals 400 and is stored, for example, in the auxiliary storage unit 403. If a customer code is entered, the processor 401 also includes that customer code in the request data. The check-in data includes at least information to identify the store where the cart terminal 400 is used. This information is, for example, a store code defined as a store identifier. The information in question may be, for example, a company code to identify a company, if there is no need to distinguish between multiple stores operated by the same company. Furthermore, if the same store code can be used by different companies, the information may include, for example, a company code and a store code.

[0037] When check-in data is transmitted to the transaction processing unit 100 via the communication network 2, the transaction processing unit 100 receives the check-in data via the communication unit 104 and temporarily stores it in the main storage unit 102 or the auxiliary storage unit 103. In this manner, when the check-in data sent from the cart terminal 400 is received by the transaction processing device 100, the processor 101 starts transaction processing for that transaction according to the transaction processing program PRA.

[0038] If processor 101 is already executing a transaction for a different customer, it will start a new transaction as a separate thread from that transaction. In other words, processor 101 may execute multiple transaction processes in parallel. However, the following explanation will focus only on the processing of a transaction for a single customer. Thus, in the following explanation, "cart terminal 400" refers to a single cart terminal 400 used by the customer for the transaction processing of interest. Also, in the following explanation, "transaction" refers to the transaction that is the subject of the processing of interest.

[0039] Figure 8 is a flowchart of the transaction processing. As ACT1, processor 101 generates new transaction data DAA for the target transaction. Specifically, processor 101 determines a new transaction code, different from the transaction code used to identify other transactions, according to predetermined rules, sets this transaction code in field FBA, and generates new transaction data DAA by setting the terminal code included in the request data in field FBB, and stores it in auxiliary storage unit 103. If the request data includes a customer code, processor 101 sets that customer code in field FBC of the newly generated transaction data DAA. If the request data does not include a customer code, processor 101 sets field FBC of the newly generated transaction data DAA to, for example, null. Since the transaction products have not yet been registered, processor 101 does not include product data in the newly generated transaction data DAA.

[0040] The customer searches for the product they wish to purchase within the store. If the customer wishes to register a product as a new purchase, they perform a predetermined operation on the cart terminal 400 to instruct the registration of that product. This operation may include, for example, scanning a barcode representing a product code, such as a JAN (Japanese article number) code, with the scanner 499. Another operation may involve tapping a preset button associated with a product displayed on the touch panel 404. Yet another operation may involve directly entering the product code on the touch panel 404. In response to such an operation, the processor 401 on the cart terminal 400 requests the transaction processing unit 100 to register the product, along with notification of the product code.

[0041] In the transaction processing unit 100, the processor 101 generates transaction data as ACT1 in Figure 8, and then proceeds to ACT2. In ACT2, processor 101 checks whether product registration has been requested. If processor 101 cannot confirm the relevant event, it determines NO and proceeds to ACT3. In ACT3, processor 101 checks whether or not accounting commencement has been requested. If processor 101 cannot confirm the relevant event, it determines NO and returns to ACT2. Thus, processor 101 waits for a request for registration or accounting as ACT2 and ACT3. If registration is requested as described above, processor 101 determines YES in ACT2 and proceeds to ACT4. As ACT4, processor 101 registers a product as a trading product by updating the transaction data DAA to include product data related to the requested product in response to a product registration request.

[0042] As ACT5, processor 101 updates the tag table TAA. When processor 101 executes ACT5 for the first time in response to the instruction to register the initial trading product, it generates a new tag table TAA associated with the transaction being processed and stores it in the auxiliary storage unit 103. This new tag table TAA includes one data record REA associated with one tag assigned to the trading product registered in ACT3.

[0043] The processor 101 sets a tag code determined by a predetermined rule in the field FBA of the data record REA. This rule is expected to be a predetermined initial value, such as "1". However, this rule may be determined as appropriate by, for example, the creator of the transaction processing program PRA.

[0044] The processor 101 sets the field FBB of the data record REA with display information determined by a predetermined rule regarding the product that was designated as a traded product by the product registration that triggered the YES determination in ACT24. This rule is expected to be, for example, a string representing the product code and product name of the product in question. However, this rule may be determined as appropriate by, for example, the creator of the transaction processing program PRA.

[0045] The processor 101 sets information in the field FBC of the data record REA indicating that the display position is undetermined. This information may be appropriately defined, for example, by the creator of the transaction processing program PRA, so as to be distinguishable from the information indicating the actual display position. The processor 101 sets a display flag in the field FBD of the data record REA to indicate a "hidden" state.

[0046] Once processor 101 has finished updating the tag table TAA, it returns to the waiting state for ACT2 and ACT3. Thus, each time product registration is requested, processor 101 repeats ACT4 and ACT5, adding information about the new trading product to the transaction data DAA and the tag table TAA. When processor 101 executes ACT5 for the second time or later, it adds one data record REA to the tag table TAA generated as described above, associating it with one tag to be assigned to the newly registered trading product. Processor 101 sets the field FBA of the data record REA to be added with a tag code determined by a predetermined rule. This rule is expected to be, for example, the maximum value of the tag codes already set in the field FAA of the data record REA included in the tag table TAA plus "1". However, this rule may be determined as appropriate by, for example, the creator of the transaction processing program PRA.

[0047] Once the customer has finished registering the items to be traded, they perform a predetermined operation on the cart terminal 400 to instruct the start of checkout. In response, the processor 401 on the cart terminal 400 requests the transaction processing unit 100 to start checkout. In response to this request, the processor 101 in the transaction processing unit 100 determines YES in ACT3 in Figure 8 and proceeds to accounting processing. The accounting processing may be, for example, the same as existing processing, and its illustration is omitted.

[0048] For example, the processor 101 instructs the cart terminal 400 to display the accounting screen. The accounting screen is a screen for handing over the accounting processing related to a transaction to the accounting machine 200. The accounting screen displays a barcode that represents information for the accounting machine 200 to make inquiries about the relevant transaction to the transaction processing device 100. When the processor 401 in the cart terminal 400 is instructed to display the accounting screen, it updates the display screen of the touch panel 404 to the accounting screen.

[0049] If there are multiple payment machines 200 installed in the store, the customer can choose any unused payment machine 200 and have the barcode scanner on that payment machine 200 read the barcode displayed on the payment screen. In response, the payment machine 200 requests payment data from the transaction processing unit 100 based on the information represented by the barcode read by the barcode scanner. When the transaction processing unit 100 receives a request for accounting data, the processor 101 sends accounting data to the requesting accounting machine 200 so that the accounting machine 200 can settle the requested transaction. In this manner, the accounting machine 200, in response to the accounting data transmitted from the transaction processing device 100, displays a screen as appropriate, receives customer input regarding accounting, and performs the processing to account for the transaction based on the accounting data. The processing performed by this accounting machine 200 may be similar to the processing performed by the accounting machine in an existing POS system, for example.

[0050] Furthermore, regarding the transfer of accounting processing related to transactions from the cart terminal 400 to the accounting machine 200, in addition to the above, this may also be achieved, for example, by transferring accounting data wirelessly when the cart terminal 400 approaches a specific accounting machine 200. Also, if the accounting machine 200 is in the form of a gate, accounting data related to transactions may be transferred when the cart terminal 400 approaches the gate.

[0051] Now, in the transaction processing device 100, the processor 101 starts the transaction processing described above, and at the same time, starts tag management processing for the same transaction according to the transaction processing program PRA. Thus, if the processor 101 is executing multiple transaction processing operations in parallel, it will also execute multiple tag management operations in parallel. However, the following explanation will focus only on tag management processing for a single transaction. Thus, in the following explanation, "cart terminal 400" refers to a single cart terminal 400 used by a customer for the transaction targeted by the tag management processing under consideration. Also, in the following explanation, "transaction" refers to the transaction that is the target of the tag management processing under consideration. Furthermore, the tag management process may be executed based on an application program separate from the transaction processing program PRA.

[0052] Figure 9 is a flowchart of the tag management process. As ACT11, processor 101 acquires camera images from cart terminal 400. For example, processor 101 sends instruction data to cart terminal 400 via communication unit 104 to communication network 2 to instruct it to transmit the camera images. When this instruction data is transmitted to the cart terminal 400 via the communication network 2 and received by the wireless communication unit 407, the processor 401 sends the camera image obtained by the camera 405 to the transaction processing unit 100 via the wireless communication unit 407 to the communication network 2.

[0053] In the transaction processing device 100, when the above camera image is transmitted to the transaction processing device 100 via the communication network 2 and received by the communication unit 104, the processor 101 stores it in the main storage unit 102 or the auxiliary storage unit 103. Here, the shopping basket SB is a container for holding goods that have been registered as the subject of a transaction. The camera image is an image of the inside of the shopping basket SB, which is said to be the container. Thus, by having the processor 101 perform information processing based on the transaction processing program PRA, the computer with the processor 101 as its central component functions as an acquisition means.

[0054] As ACT12, processor 101 performs cropping processing on the camera image acquired as described above. Cropping processing is the process of extracting individual objects that are captured in the camera image. For cropping processing, well-known image processing techniques for extracting objects captured in an image can be used. Note that cropping processing does not determine what the extracted objects are.

[0055] In ACT13, processor 101 checks whether or not any objects were successfully extracted through the extraction process described above. If at least one object was extracted, processor 101 determines the result is YES and proceeds to ACT14. The tag management process starts when the cart terminal 400 is put into use, and product registration has not started when the camera image acquired by ACT11 is taken. In this state, there should be no products in the shopping cart SB. It is desirable to establish a rule for using the cart terminal 400 that the shopping cart SB should be left empty when it is put into use, and to make this known to the customer. To make this known, the processor 401 of the cart terminal 400 may display a screen on the touch panel 404 instructing the customer to leave the shopping cart SB empty when it receives an operation to put it into use. This screen display may be performed autonomously by the processor 401, or it may be performed in response to instructions from the processor 101 of the transaction processing device 100. For this reason, there should be no objects in the camera image acquired by ACT11, and if an object is extracted by the extraction process of ACT12, it means that there is an object in the shopping cart SB, contrary to the above rule.

[0056] As ACT14, processor 101 performs predetermined error handling. This error handling is for responding to violations of the above rules, and the specific actions to be taken may be determined as appropriate by, for example, the creator of the transaction processing program PRA. For example, processor 101 instructs cart terminal 400 to display a screen on touch panel 404 instructing the customer to empty shopping cart SB. Then, if predetermined release conditions are met, processor 101 returns to ACT11. One example of a release condition is when the cart terminal 400 notifies that a predetermined operation has been performed to declare that the customer has confirmed the instructions on the screen instructing them to empty shopping cart SB. Alternatively, another example of a release condition is when a predetermined waiting period has ended.

[0057] If the processor 101 is unable to extract any objects during the extraction process in ACT12, it determines NO in ACT13 and proceeds to ACT15. As ACT15, the processor 101 stores the data representing the result of the extraction process in ACT12 as first extracted data in the main storage unit 102 or the auxiliary storage unit 103.

[0058] As ACT16, processor 101 waits for the acquisition timing. The acquisition timing is, for example, the time after a predetermined period of time has elapsed since the last camera image was acquired. However, the acquisition timing may be determined as appropriate by, for example, the creator of the transaction processing program PRA. When the acquisition timing is reached, processor 101 determines YES in ACT16 and proceeds to ACT17. As ACT17, processor 101 acquires the camera image. This process may be similar to, for example, ACT11.

[0059] As ACT18, processor 101 performs the extraction process. This process may be similar to, for example, ACT12. As ACT19, processor 101 stores the data representing the result of the extraction process in ACT18 as second extracted data in main storage unit 102 or auxiliary storage unit 103.

[0060] In ACT20, processor 101 checks whether there is any change between the first extracted data and the second extracted data. If there is no difference between the first and second extracted data, processor 101 determines NO and proceeds to ACT21. As ACT21, processor 101 overwrites the second extracted data with new first extracted data. After this, processor 101 returns to the waiting state of ACT16. Thus, the processor 101 repeatedly acquires camera images at each acquisition timing, while waiting for any changes to occur in the cropping result for the latest camera image compared to the cropping result for the previously acquired camera image.

[0061] Figure 10 is a diagram showing an example of a camera image IMA. Note that Figure 10 and the camera image diagrams described later are simplified representations. The camera image IMA was taken with items ITA, ITB, ITC, and ITD placed in shopping cart SB.

[0062] Figure 11 shows an example of the resulting crop. The cropping results shown in Figure 11 are an example of the results obtained by performing the cropping process on the camera image IMA shown in Figure 10. Therefore, objects OBA, OBB, OBC, and OBD are cropped corresponding to products ITA, ITB, ITC, and ITD, respectively.

[0063] Figure 12 shows an example of a camera image (IMB). Camera image IMB was taken with products ITA, ITB, ITC, ITD, and ITE in the shopping cart SB. In other words, camera image IMB was taken after camera image IMA was taken, with product ITE newly added.

[0064] Figure 13 shows an example of the resulting crop. The cropping results shown in Figure 13 are an example of the results obtained by performing the cropping process on the camera image IMB shown in Figure 12. Therefore, objects OBA, OBB, OBC, OBD, and OBE are cropped corresponding to the products ITA, ITB, ITC, ITD, and ITE, respectively.

[0065] Thus, if the first extracted data shows the extraction result as shown in Figure 11, and the second extracted data shows the extraction result as shown in Figure 13, there is a significant difference between the two extracted data. In such a case, the processor 101 determines YES in ACT20 and proceeds to ACT22. At this point, the processor 101 has detected the products placed in the shopping basket SB based on the image taken of the inside of the shopping basket SB, which serves as the container. Thus, by executing information processing based on the transaction processing program PRA, the computer with the processor 101 as its central component functions as a detection means.

[0066] As ACT22, processor 101 increments the shopping cart count by one. Although not shown in the diagram, processor 101 clears the shopping cart count to 0 at a time before executing ACT22 for the first time, such as at the start of the tag management process. In other words, the shopping cart count is the number of items detected to have been added to shopping cart SB. As ACT23, processor 101 determines which of the objects cut out in the cutting process in ACT18 is the newly added product. If the first cutting data represents the cutting result shown in Figure 11 and the second cutting data represents the cutting result shown in Figure 13, processor 101 determines that object OBE is the added product.

[0067] As ACT24, processor 101 checks whether product registration has been performed for the input product determined as described above. For example, the usage rules for the cart terminal 400 stipulate that products that have been registered as transaction products should be promptly placed in the shopping cart SB, or that products placed in the shopping cart should be promptly registered as transaction products. Then, processor 101 determines that product registration has been performed for the input product if the registration of the transaction product is performed at a time close to the input timing of the input product determined as described above. More specifically, processor 101 determines that product registration has been performed if, for example, a new product registration is performed in a transaction process targeting the same transaction within an allowable period from a predetermined time before the timing of the camera image taken in ACT17 to a predetermined time after the timing of the camera image taken, and proceeds to ACT25.

[0068] As ACT25, processor 101 updates the tag table TAA. For example, in relation to the product registration that triggered the YES determination in ACT24, the processor 101 sets a display position determined according to a predetermined rule in the field FBC of the data record REA added to the tag table TAA in ACT5 in Figure 8. This rule is predetermined so that the display position is near the location where the object corresponding to the product being entered is extracted in the camera image. For example, the processor 101 determines the display position so that the centroid of the area extracted as the object is at the center of the tag. However, this rule may be determined as appropriate by, for example, the creator of the transaction processing program PRA. Furthermore, the processor 101, for example, changes the display flag set in the field FBD of the data record REA from a state representing "hidden" to a state representing "displayed".

[0069] As ACT26, processor 101 generates an AR screen. The AR screen will be described later. Once processor 101 has finished generating the AR screen, it repeats steps ACT21 and beyond in the same manner as described above. When processor 101 executes ACT25 for the second time or later, it determines the display position to set in field FBC to minimize overlap with other tags, taking into consideration, for example, the display position already set in field FBC of data record REA included in tag table TAA. Processor 101 may also change the display position already set in field FBC of data record REA included in tag table TAA at this time. Furthermore, if there is another tag whose overlap state with the tag associated with the data record REA to be added satisfies a predetermined condition, processor 101 changes the display flag set in field FBD of the data record REA associated with that tag to a state representing "hidden". The above condition is expected to be, for example, "the size of the overlapping area is greater than or equal to a predetermined threshold". However, this condition may be determined as appropriate by, for example, the creator of the transaction processing program PRA.

[0070] Figure 14 shows an example of an AR screen SCA. The AR screen SCA is an example of a screen generated by the processor 101 after acquiring the camera image IMA shown in Figure 10 in ACT17, proceeding through ACT25 and then to ACT26. The processor 101 generates the AR screen SCA by overlaying a computer graphics (CG) image representing the four AR tags ATA, ATB, ATC, and ATD, the indicator INA, and the string CSA onto the camera image IMA shown in Figure 10.

[0071] Processor 101 places AR tags ATA, ATB, ATC, and ATD in the CG image according to each of the four data records REA included in the tag table TAA, where the display flag indicating "display" is set in the field FBD. The processor 101 displays the AR tags ATA, ATB, ATC, and ATD as strings corresponding to the display information set in the field FBB of the corresponding data record REA. For example, AR tag ATA indicates that the product code is "4510JAN" and the product name is "AAAAA". However, "4510JAN" and "AAAAA" are simplified illustrations of the display status of the display information. The processor 101 places each of the AR tags ATA, ATB, ATC, and ATD at the display position set in the field FBC of the corresponding data record REA.

[0072] If the number of items registered as trading products matches the number of items in the shopping cart, the processor 101 places the indicator INA on the CG image as shown in Figure 14. Processor 101 places the string "CSA" on the CG image to represent the number of items registered as trading products (hereinafter referred to as "registered items") and the number of items in the shopping cart.

[0073] Figure 15 is an example of an AR screen SCB. The AR screen SCB is an example of a screen generated by the processor 101 after acquiring the camera image IMB shown in Figure 12 in ACT17, proceeding through ACT25 to ACT26. The CG image that makes up the AR screen SCB represents five AR tags ATA, ATB, ATC, ATD, ATE, the indicator INA, and the string CSB. In other words, AR screen SCB is an example of a screen that is updated from AR screen SCA when the extraction result represented by the first extraction data is the situation shown in Figure 11, the extraction result represented by the second extraction data is the situation shown in Figure 13, and product registration has been performed.

[0074] The processor 101 generates the AR screen SCB by overlaying a CG image representing five AR tags ATA, ATB, ATC, ATD, ATE, an indicator INA, and a string CSB onto the camera image IMB shown in Figure 12. The processor 101 represents the CG image for AR screen SCB by adding the AR tag ATE in association with the newly extracted object represented by the extraction result in the second extraction data, in addition to the AR tags ATA, ATB, ATC, and ATD represented by the CG image in AR screen SCA. Furthermore, the processor 101 will replace the string CSA represented by the CG image in the AR screen SCA with the string CSB in the CG image for the AR screen SCA, in accordance with the increase in the number of registered points and the number of basket points.

[0075] If processor 101 cannot confirm that product registration has been performed for the input product determined in ACT23, it will determine NO in ACT24, skip ACT25, and proceed to ACT26. At this point, since the processor 101 has not updated the tag table TAA, it generates a CG image similar to the one generated when ACT26 was executed last time. Then, the processor 101 generates an AR screen by overlaying this CG image onto the camera image that has been modified.

[0076] Figure 16 is a diagram showing an example of an AR screen SCC. AR screen SCC is an example of a screen generated by the processor 101 after acquiring the camera image IMA shown in Figure 12 in ACT17, and proceeding to ACT26 without going through ACT25. In other words, AR screen SCC is an example of a screen updated from AR screen SCA when the cropping result represented by the first cropping data is the situation shown in Figure 11, the cropping result represented by the second cropping data is the situation shown in Figure 13, and it could not be confirmed that product registration had been performed.

[0077] The processor 101 generates the AR screen SCB by overlaying a CG image representing the four AR tags ATA, ATB, ATC, ATD, ATE, the indicator INB, and the string CSC, similar to those in Figure 14, onto the camera image IMB shown in Figure 12. The processor 101 assumes that the CG images for AR screen SCC represent the same AR tags ATA, ATB, ATC, and ATD as those represented by the CG images in AR screen SCA.

[0078] Furthermore, the processor 101 sets the CG image for the AR screen SCC to "4 points," the same as the AR screen SCA, because the number of registered points has not increased, and represents the string CSC to "5 points" in accordance with the increase in the number of basket points. The processor 101 then determines that the CG image for the AR screen SCC will represent an indicator INB indicating that a discrepancy has occurred, since the registered points and the basket points do not match.

[0079] As described above, the AR screens SCA, SCB, and SCC are screens that overlay CG images, which allow users to identify registered products, onto the acquired camera images. When a product is detected being added to a shopping basket SB, which is an example of a container, the screen is updated to a new screen that allows the user to identify the product newly added to the shopping basket SB, as shown in the AR screens SCA, SCB, and SCC, provided that the product to be traded is registered at a predetermined registration timing relative to the detection timing. Thus, by having the processor 101 execute information processing based on the transaction processing program PRA, the computer with the processor 101 as its central component functions as a generation means. The above registration timing is any timing within the aforementioned allowable period in this embodiment.

[0080] Thus, by sending the AR screen generated by the transaction processing device 100 to the cart terminal 400 at an appropriate time and displaying it on the touch panel 404, the customer can confirm the number of registered items. The customer can then recognize that the product registration has not been performed correctly because the number of registered items does not match the number of items in the shopping cart SB.

[0081] Furthermore, based on an AR screen like the AR screen SCC shown in Figure 16, customers can recognize which of the products captured in the camera image have not been properly registered. In other words, customers can easily identify products that are captured on the AR screen but do not have an AR tag attached to that AR screen, thus indicating that they have not been properly registered.

[0082] Furthermore, by sending the AR screen generated by the transaction processing device 100 to the attendant terminal 300 at an appropriate time and displaying it on the touch panel 304, the attendant can confirm the number of registered items. The attendant can also recognize if there are items in the shopping cart SB that are not properly registered by visually checking the number of items in the shopping cart SB and comparing it with the number of registered items. In addition, the attendant can easily recognize items that are not properly registered, even if they are visible on the AR screen, because they do not have an AR tag attached to that AR screen, and can point this out to the customer.

[0083] Furthermore, by checking the camera image displayed on the AR screen, attendants can confirm the status of items being placed in the shopping basket SB, even when they cannot directly see the shopping basket SB. The display on the attendant terminal 300 is expected to be as follows, for example. First, the touch panel 304 of the attendant terminal 300 displays a monitoring screen that shows an overview and list of the operating status of each cart terminal 400 currently in use within the store.

[0084] Figure 17 is a diagram showing an overall overview of an SCD monitoring screen as an example. The monitoring screen SCD includes a display area ARA associated with each of the cart terminals 400 that are currently checking in. The display area ARA shows the status of the transaction currently running for the associated cart terminal 400. The monitoring screen SCD can display the operating status of up to 15 cart terminals 400 at once, but it also displays a display area ARA associated with each of the operational cart terminals 400. Therefore, the number of display area ARAs included in the monitoring screen SCD may be less than the state shown in Figure 17.

[0085] Figure 18 is a diagram showing one detail of the display area ARA in Figure 17. Figure 18 shows an example of the changes in display within a single display area (ARA). The display area ARA contains the string CSD, the border FRA, and the display objects DOA, DOB, and DOC, as shown in the figure. The string CSD represents the terminal code of the cart terminal 400. The border FRA represents the area indicating the normal / abnormal status of adding items to the shopping cart SB. The display objects DOA, DOB, and DOC are associated with the states "Waiting," "Registering," and "Checkout," respectively. One of the display objects DOA, DOB, and DOC is set to the active state, and the other two are set to the inactive state. In Figure 18, the border of the display object in the active state is shown as a solid line, and the border of the display object in the inactive state is shown as a dashed line. The inactive state is expected to be displayed as, for example, grayed out. The display object DOA is set to the active state before product registration for the relevant transaction begins. The display object DOB is set to the active state while product registration for the relevant transaction is in progress. The display object DOC is set to the active state during checkout for the products subject to the relevant transaction.

[0086] On the attendant terminal 300, the processor 301 displays the string "Registration: Normal" within the border FRA of the display area ARA associated with the cart terminal 400 where the registered points and the cart points match, for example, as shown in the upper part of Figure 18. On the other hand, the processor 301 displays the string "Registration: Abnormal" within the border FRA of the display area ARA associated with the cart terminal 400 where the registered points and the cart points do not match, for example, as shown in the middle and lower parts of Figure 18. The monitoring screen SCD may be generated by the processor 101 in the transaction processing unit 100, transmitted to the attendant terminal 300, and displayed on the touch panel 304 under the control of the processor 301. Alternatively, the processor 301 in the attendant terminal 300 may generate the monitoring screen SCD after acquiring various information from the transaction processing unit 100.

[0087] Based on the display in the display area ARA shown in the upper part of Figure 18, the attendant can recognize that product registration is being performed by the associated cart terminal 400 and that the number of registered items matches the number of items in the cart. Based on the display in the ARA display area shown in the middle of Figure 18, the attendant can recognize that the associated cart terminal 400 is in a state where product registration is being performed, and that the number of registered items and the number of items in the cart do not match. Based on the display in the display area ARA shown in the lower part of Figure 18, the attendant can recognize that the associated cart terminal 400 is in a state where accounting is being processed using the accounting machine 200 identified by the number "No. 2", and that the registered points and the number of items in the basket do not match.

[0088] Therefore, if the attendant recognizes, for example, from the display area ARA shown in the lower part of Figure 18, that accounting is about to be performed for a transaction in which the registration of the transaction product may not have been done properly, they will instruct the display of the AR screen by performing a predetermined operation, such as tapping inside the border FRA of the relevant display area ARA. The attendant may also instruct the display of the AR screen at any other time of their choosing. In response to such instructions, the processor 301 in the attendant terminal 300 requests an AR screen from the transaction processing unit 100, along with specifying the cart terminal 400 associated with the relevant display area ARA. In response to this request, the processor 101 in the transaction processing unit 100 sends the most recently generated AR screen to the attendant terminal 300. Then, the processor 301 in the attendant terminal 300 displays the AR screen sent from the transaction processing unit 100 on the touch panel 304. This allows the attendant to perform the above verification process for customers using payment machine 200, which is identified by the number "No. 2".

[0089] Such verification by an attendant may, in some cases, involve the attendant speaking to the customer and then picking up the items placed in the shopping basket SB. Such actions by the attendant may cause discomfort to the customer. Therefore, it is preferable to display a screen on the touch panel 404 at the start of use of the cart terminal 400, etc., informing the customer that if they do not follow the aforementioned rules for using the cart terminal 400, a store employee may perform an inspection. This screen display may be performed autonomously by the processor 401, or it may be performed in response to instructions from the processor 101 of the transaction processing unit 100.

[0090] The processor 101 will not increase the number of items in the shopping cart unless the second extraction result changes to the first extraction result due to an item being placed in the shopping cart SB. In other words, if an item registered as a transaction item is not placed in the shopping cart SB, the processor 101 will not increase the number of items in the shopping cart, resulting in a difference between the registered number of items and the number of items in the shopping cart. For example, it is normal for large items that are difficult to fit into the shopping cart SB not to be placed in the shopping cart SB. To confirm that such a situation is normal, it is convenient to identify items that are allowed not to be placed in the shopping cart SB by adding a flag to the product database, and to allow the attendant terminal 300 to confirm that the relevant item is registered as a transaction item. Specifically, knowing in advance on the attendant terminal that there are large items that may not have their AR tags displayed helps in visually confirming that the customer has brought such items into the cart during checkout.

[0091] Thus, the processor 101 performs information processing based on the transaction processing program PRA, and outputs the AR screen to the attendant terminal 300 as described above, thereby allowing the computer with the processor 101 as its central component to function as an output means.

[0092] This embodiment can be modified in various ways as follows: The tag management process may be performed in whole or in part by the processor 401 on the cart terminal 400.

[0093] The scanner 499 may be made detachable from the cart 900, allowing customers to read the barcodes of items placed in the shopping basket SB using the scanner 499 after removing it from the cart 900. Furthermore, if an infrared scanning type scanner 499 is used, and the scanning light from the scanner 499 is captured in the camera image, the position of a newly placed object in the shopping basket SB may be determined based on the position where the scanning light was emitted. Alternatively, a handheld scanner that is not detachable from the cart 900 may be provided in place of, or in addition to, the scanner 499. Furthermore, an infrared scanning type scanner may be used as such a handheld scanner, and the position of a newly placed object in the shopping basket SB may be determined based on the position where the scanning light is irradiated, similar to the above.

[0094] Alternatively, a cart may be used that is equipped with a smartphone, allowing the camera on the smartphone to be used to photograph the inside of the shopping cart SB, and the customer's own smartphone may be used instead of the cart terminal 400.

[0095] Camera images may be captured using a camera mounted on the cart.

[0096] Each function realized by the processor 101 through information processing can also be partially or entirely realized by hardware that performs non-program-based information processing, such as logic circuits. Furthermore, each of the above functions can also be realized by combining the above-mentioned hardware, such as logic circuits, with software control.

[0097] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0098] 1...Transaction processing system, 2...Communication network, 100...Transaction processing unit, 101, 301, 401...Processor, 102, 302, 402...Main memory unit, 103, 303, 403...Auxiliary memory unit, 104, 305...Communication unit, 105, 306, 408...Transmission line, 200...Accounting machine, 300...Attendant terminal, 304, 404...Touch panel, 405...Camera, 400...Cart terminal, 406...Interface unit, 407...Wireless communication unit, 499...Scanner, 900...Cart.

Claims

1. An acquisition means for obtaining images of the inside of a container used to hold goods that have been registered as the subject of a transaction, A detection means for detecting the product placed in the container based on the image acquired by the acquisition means, The system generates a screen in which an image that allows the user to identify the registered product is superimposed on the image acquired by the acquisition means, and when a product newly placed in the container is detected by the detection means, if the product to be traded is registered at a predetermined registration timing relative to the detection timing, the generation means generates a new screen in which the user can identify the product newly placed in the container. An information processing device equipped with the following.

2. The detection means detects the entry of a product based on the difference between images taken at different times and acquired by the acquisition means. The information processing apparatus according to claim 1.

3. The generation means displays on the screen the number of registered products and the number of products newly placed in the container that have been detected by the detection means. The information processing apparatus according to claim 1.

4. The generation means displays on the screen that the number of registered products does not match the number of products newly added to the container and detected by the detection means. The information processing apparatus according to claim 1.

5. An output means that outputs instruction data to an information and communication terminal used by a user responsible for monitoring transaction status, which instructs the display of the screen generated by the generation means. The information processing apparatus according to claim 1, further comprising:

6. Computers, An acquisition means for obtaining images of the inside of a container used to hold goods that have been registered as the subject of a transaction, A detection means for detecting the product placed in the container based on the image acquired by the acquisition means, The system generates a screen in which an image that allows the user to identify the registered product is superimposed on the image acquired by the acquisition means, and when a product newly placed in the container is detected by the detection means, if the product to be traded is registered at a predetermined registration timing relative to the detection timing, the generation means generates a new screen in which the user can identify the product newly placed in the container. An information processing program that enables a function to work.

Images