Information processing terminal, registration system, and information processing program

Abstract

To enable an article reflected in an image to be identified with accuracy based on the image.SOLUTION: An article recognition device of an embodiment includes selection means, text recognition means, narrow-down means, and identifying means. Based on a characteristic of an article appearing in an image, the selection means selects one or more candidate articles for the article. The text recognition means recognizes text appearing in the image for the article. When a plurality of candidate articles are selected by the selection means, the narrow-down means narrows down the plurality of candidate articles into a fewer number of candidate articles based on the text recognized by the text recognition means. The identifying means identifies the article appearing in the image as the candidate article if one candidate article is selected by the selection means, and when a plurality of candidate articles are selected by the selection means, identifies the article appearing in the image as one of the candidate articles narrowed down by the narrow-down means.SELECTED DRAWING: Figure 5

Description

Technical Field

[0004] , , , , , , , , Information processing terminal , , ,

[0003] , , , Inside , , , 、 , ,

[0005] , , ,

[0001] Embodiments of the present invention relate to Information processing terminal a registration system and an information processing program.

Background Art

[0002] In a store of the display sales type, a registration system has already been proposed in which a product placed by a purchaser in a container is recognized from an image taken inside the shopping basket and automatically registered as a product to be purchased. However, there are cases where the appearances of a plurality of different products are similar, and it may be difficult to narrow down which product an article is based only on the features shown in the image. Under such circumstances, it has been desired to accurately identify an article shown in an image based on the image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to accurately identify an article shown in an image based on the image Information processing terminal and provide a registration system and an information processing program.

Means for Solving the Problems

[0005] The information processing terminal of the embodiment includes a detection unit, a selection unit, a character recognition unit, a narrowing-down unit, and a specifying unit 、 a registration unit and cancellation means The detection unit, when an article is put into a container that houses a product that a purchaser plans to purchase, the container InsideThe items placed in the container were detected from the image obtained by taking a photograph. Furthermore, when an item is removed from a container, the system detects the item removed from the container based on the difference between a new image obtained by photographing the inside of the container and a previous image. The selection means selects one or more candidate items for the item detected by the detection means based on the features that appear in the image. The character recognition means recognizes the characters that appear in the image for the item. If multiple candidate items are selected by the selection means, the narrowing means narrows down the candidate items from among the multiple candidate items to a smaller number based on the characters recognized by the character recognition means. If one candidate item is selected by the selection means, the identification means identifies the item that appears in the image as that candidate item, and if multiple candidate items are selected by the selection means, it identifies the item that appears in the image as one of the candidate items narrowed down by the narrowing means. If an item placed in a container is detected by the detection means, the registration means registers the item identified by the identification means as a product to be purchased. The cancellation means removes the item identified by the identification means from the items to be purchased when the item removed from the container is detected by the detection means. [Brief explanation of the drawing]

[0006] [Figure 1] A block diagram showing the schematic configuration of a transaction processing system according to one embodiment, and the main circuit configuration of a cart terminal. [Figure 2] A schematic diagram illustrating one of the data record structures included in the dictionary database shown in Figure 1. [Figure 3] A perspective view showing an example of how the cart terminal in Figure 1 is attached to the cart. [Figure 4] Flowchart of the registration process. [Figure 5] Flowchart for product identification process. [Figure 6] A diagram illustrating an example of products that can be filtered by text. [Figure 7] A diagram illustrating an example of a selection screen. [Figure 8] A diagram showing an example of a guidance screen. [Modes for carrying out the invention]

[0007] An example of an embodiment will be described below with reference to the drawings. Figure 1 is a block diagram showing the schematic configuration of the transaction processing system 1 according to this embodiment and the main circuit configuration of the cart terminal 100. The transaction processing system 1 is configured so that the cart terminal 100, accounting machine 200, POS server 300, and attendant terminal 400 can communicate with each other via the communication network 2.

[0008] Communication network 2 can use the internet, VPN (virtual private network), LAN (local area network), public communication network, mobile communication network, etc., individually or in appropriate combinations. As an example, communication network 2 can be used in combination with the internet and a mobile communication network. Note that the transaction processing system 1 may include any number of cart terminals 100, accounting machines 200, POS servers 300, and attendant terminals 400, but Figure 1 shows only one of each.

[0009] The cart terminal 100 is an information processing terminal attached to a shopping cart (hereinafter referred to as "cart") provided in the store. The cart terminal 100 is lent to customers along with the cart when they shop in the store. The cart terminal 100 is a terminal device that, upon receiving an operation from the customer, performs a registration process to register the items that the customer intends to purchase as purchased items. As will be described later, the cart terminal 100 performs a process to recognize the items to be registered as purchased items in relation to objects shown in an image. In other words, the cart terminal 100 has the function of an item recognition device. Furthermore, as will be described later, the cart terminal 100 performs a process to register the recognized items as purchased items. In other words, the cart terminal 100 has the function of a registration system.

[0010] The payment terminal 200 is installed in the store and performs accounting processing related to the payment of purchased items registered by the shopping cart terminal 100. The payment terminal 200 is operated by an operator during the accounting process. The operator of the payment terminal 200 is mainly a customer. In some cases, a store employee may operate the payment terminal 200. The POS server 300 is an information processing device that executes management processing for managing the sales and purchase transactions of products processed by the registration processing in the cart terminal 100 and the accounting processing in the accounting machine 200. The attendant terminal 400 is an information processing terminal operated by a store clerk. The attendant terminal 400 is a terminal device for a user interface related to information processing for supporting the work of the store clerk regarding transactions processed by the transaction processing system 1. The work of the store clerk is, for example, the work of monitoring the execution status of a transaction being processed and appropriately supporting customers.

[0011] The cart terminal 100 includes a processor 101, a main storage unit 102, an auxiliary storage unit 103, a touch panel 104, an interface unit 105, a wireless communication unit 106, a transmission path 107, and the like. The processor 101 and the main storage unit 102, the auxiliary storage unit 103, the touch panel 104, the interface unit 105, and the wireless communication unit 106 are communicable via the transmission path 107.

[0012] By connecting the processor 101, the main storage unit 102, and the auxiliary storage unit 103 with the transmission path 107, a computer for performing information processing for controlling the cart terminal 100 is configured. The processor 101 corresponds to the central part of the above computer. The processor 101 executes information processing for controlling each part in order to realize various functions as the cart terminal 100 according to information processing programs such as an operating system and application programs.

[0013] The main memory unit 102 corresponds to the main memory part of the above computer. The main memory unit 102 includes a read-only memory area and a rewritable memory area. The main memory unit 102 stores a part of the above information processing program in the read-only memory area. Also, the main memory unit 102 may store data necessary for the processor 101 to execute processes for controlling each part in the read-only memory area or the rewritable memory area. The main memory unit 102 uses the rewritable memory area as a work area by the processor 101.

[0014] The auxiliary storage unit 103 corresponds to the auxiliary storage part of the above computer. The auxiliary storage unit 103 can use, for example, EEPROM (electric erasable programmable read-only memory), HDD (hard disc drive), SSD (solid state drive), or other various well-known storage devices. The auxiliary storage unit 103 stores data used by the processor 101 to perform various processes and data generated by the processes in the processor 101. The auxiliary storage unit 103 may store the above information processing program. In this embodiment, the auxiliary storage unit 103 stores a cart terminal program PRA which is one of the information processing programs. The cart terminal program PRA is an application program that describes the procedure of the registration process. A part of the storage area of the auxiliary storage unit 103 is used as an area for storing the dictionary database DBA and the transaction data DAA. The dictionary database DBA is a set of dictionary data referred to for identifying which product the article reflected in the image is. The transaction data DAA is data representing the content of one transaction.

[0015] The interface unit 105 is connected to an external device such as a camera 199. The interface unit 105 interfaces with the connected external device for data exchange. An existing USB (universal serial bus) controller can be used as the interface unit 105. The camera 199 captures an image of the entire interior space of the shopping basket placed on the cart. The camera 199 then outputs image data representing the image obtained through the capture. The camera 199 may repeat the capture at a fixed interval, or it may take a picture in response to instructions from the processor 101.

[0016] The wireless communication unit 106 performs communication processing for wireless data communication over the communication network 2. For example, an existing wireless communication device for a wireless LAN can be used as the wireless communication unit 106. Alternatively, a communication unit connected to the communication network 2 via a wired connection may be used instead of, or in addition to, the wireless communication unit 106. The transmission line 107 includes an address bus, a data bus, and control signal lines, and transmits data and control signals exchanged between the connected parts. The basic hardware for the cart terminal 100 is expected to be, for example, the hardware of a tablet-type information processing device.

[0017] Figure 2 schematically represents one configuration of a data record REA included in the dictionary database DBA. The dictionary database DBA is a collection of data records REA associated with each product sold in a store where the cart terminal 100 is used. Each data record REA includes fields FAA, FAB, FAC, and FAD. Field FAA contains the product code as an identifier for the associated product. Field FAB contains the product name of the associated product. Field FAC contains object information containing multiple feature points that should be referenced in order to recognize the associated product by the object recognition process described later. Field FAD contains character information containing multiple string information representing multiple strings that are represented in the appearance of the associated product.

[0018] The dictionary database DBA is created by an appropriate worker, taking into account the products sold at the store, for example, when the transaction processing system 1 is introduced to the store, and is written to the auxiliary storage unit 103 of the cart terminal 100 used at that store. The dictionary database DBA may also be updated as needed by the worker during maintenance work, even after the transaction processing system 1 has been put into use.

[0019] Figure 3 is a perspective view showing an example of how the cart terminal 100 is attached to the cart. The cart 900 comprises a caster section 910, a handle frame section 920, and a basket support section 930. The caster section 910 has four wheels 911 for smoothly moving the cart 900 on the floor. The wheels 911 are mounted on the frame 912 so as to be rotatable around a vertical axis.

[0020] The handle frame section 920 includes a pair of vertical frames 921, 921 and a handlebar 922. The vertical frames 921, 921 are erected above the two wheels of the caster section 910. The handlebar 922 connects the upper ends of the vertical frames 921, 921. The basket support section 930 is provided horizontally from the middle of the handle frame section 920. The basket support section 930 holds the shopping basket SB, which serves as a container for holding goods. The caster section 910 can also hold the shopping basket SB on top of the frame 912.

[0021] A customer using the cart 900 is typically positioned in front of the handle frame 920 in Figure 3. The customer then pushes the cart 900 while gripping the handlebar 922. In this case, the direction in which the basket support 930 protrudes relative to the handle frame 920 is the direction in which the cart 900 moves forward. A camera 199 is mounted on the middle of the handlebar 922 in a position that allows it to take an overhead, bird's-eye view of the inside of the shopping basket SB, which is held in place by the basket support 930. A pole 940 is attached to one of the vertical frames 921 such that its tip is positioned above the handlebar 922. The aforementioned cart terminal 100 is mounted on the tip of this pole 940, with the touch panel 104 facing away from the direction of forward movement. A battery case 950 is mounted between vertical frames 921, 921 on the lower end of the handle frame 920. The battery case 950 houses a battery used as an external power source for the cart terminal 100.

[0022] Next, the operation of the transaction processing system 1 configured as described above will be explained. Note that the content of the various processes described below is merely an example, and it is possible to change the order of some processes, omit some processes, or add other processes as appropriate. For example, in the following explanation, some processes have been omitted in order to clearly illustrate the characteristic operation of this embodiment. For example, if an error occurs, processing to deal with that error may be performed, but such processing has been omitted from the description. The customer performs a predetermined operation to begin using the unused cart terminal 100. Once this operation is performed, the processor 101 starts the registration process based on the cart terminal program PRA.

[0023] Figure 4 is a flowchart of the registration process. As ACT11, processor 101 generates new transaction data DAA for the target transaction. Specifically, processor 101 determines a new transaction code, for example, that is different from the transaction code used to identify other transactions, according to predetermined rules, generates new transaction data DAA that includes this transaction code but does not include product information about purchased goods, and stores it in auxiliary storage unit 103.

[0024] As ACT12, processor 101 checks whether it is time to acquire an image. If processor 101 cannot confirm the event, it determines NO and proceeds to ACT13. The acquisition timing is expected to be, for example, the timing immediately after moving from ACT11 to ACT12, and at regular time intervals thereafter, and may be determined as appropriate by, for example, the creator of the cart terminal program PRA. As ACT13, processor 101 checks whether accounting has been requested. If processor 101 cannot confirm the relevant event, it determines NO and returns to ACT12. Thus, processor 101 waits for the acquisition timing or the accounting request to occur as ACT12 and ACT13.

[0025] When it's time to acquire the data, processor 101 determines YES in ACT12 and proceeds to ACT14. As ACT14, the processor 101 acquires image data output by the camera 199. For example, if the camera 199 is taking pictures at regular intervals, the processor 101 acquires the image data output by the camera 199 corresponding to the most recent picture taken by the camera 199. For example, if the camera 199 takes pictures in response to instructions from the processor 101, the processor 101 instructs the camera 199 to take a picture and acquires the image data output by the camera 199 in response. The processor 101 then stores the acquired image data in the main storage unit 102 or the auxiliary storage unit 103.

[0026] In ACT15, processor 101 checks whether the product has been placed in shopping cart SB. If processor 101 cannot confirm the event, it determines NO and proceeds to ACT16. As ACT16, processor 101 checks whether the product has been removed from shopping cart SB. If processor 101 cannot confirm the event, it determines NO and returns to the waiting state for ACT12 and ACT13.

[0027] If the first acquisition timing occurs immediately after moving from ACT11 to ACT12, then the image data acquired by executing ACT14 for the first time will not contain any items other than the shopping basket SB. For this reason, when processor 101 first executes ACT15 and ACT16, it determines NO for both ACT15 and ACT16 and returns to the waiting state for ACT12 and ACT13. Then, processor 101 repeats ACT14 each time it is an acquisition timing. At this time, processor 101 leaves in main storage unit 102 or auxiliary storage unit 103 any image data (hereinafter referred to as past data) that has already been acquired by executing ACT14 in a predetermined period in the past.

[0028] The customer searches for the product they wish to purchase within the store. Then, the customer takes the product they intend to buy from the display and places it in their shopping basket. For example, in ACT14, the processor 101 checks whether an item has been newly placed in the shopping basket SB based on the difference between the newly acquired image data (hereinafter referred to as current data) and past data, and if the item exists, it is determined that it has been placed. Alternatively, the processor 101 may monitor the movement of the hand holding the item based on the current data and multiple past data, and determine that the item has been placed based on the results as well. If the processor 101 has confirmed an event that can be determined to be placed, it determines YES in ACT15 and proceeds to ACT17.

[0029] As ACT17, processor 101 detects the items placed in the shopping cart (hereinafter referred to as "items placed"). For example, processor 101 detects items placed based on the difference between current data and past data, such as items that are not visible in the image represented by past data but are visible in the image represented by current data. As a method for this processing, well-known methods such as deep learning using CNN (convolutional neural network) can be used. Thus, by having the processor 101 perform information processing based on the cart terminal program PRA, the computer with the processor 101 as its central component functions as a detection means. As ACT18, processor 101 performs product identification processing to determine which product the input item detected in ACT17 is.

[0030] Figure 5 is a flowchart of the product identification process. As ACT31, processor 101 performs object recognition processing on the input items detected in ACT17. Object recognition processing is a process that selects products that are thought to match the input items as candidate products by comparing the external features of the input items appearing in the images of the input items included in the image currently represented by the data with the feature points represented by the object information contained in each data record REA of the dictionary database DBA. In this object recognition processing, one or more candidate products may be selected. In addition, in the object recognition processing, no candidate products may be selected. As an image recognition method in object recognition processing, well-known methods such as feature point matching can be used. Furthermore, as a method for selecting candidate products according to the image recognition results, well-known methods such as deep learning using CNNs can be used. Thus, by having the processor 101 perform information processing based on the cart terminal program PRA, the computer with the processor 101 as its central component functions as a selection means for selecting candidate products as candidate items.

[0031] In ACT32, processor 101 checks whether there is only one candidate product selected in ACT31. If multiple candidate products were selected in ACT31, or if no candidate products were selected, processor 101 determines NO, indicating that there is more than one candidate product, and proceeds to ACT33. In ACT33, processor 101 checks whether there are multiple candidate products selected in ACT31. If processor 101 has selected multiple candidate products in ACT31, it determines YES and proceeds to ACT34.

[0032] As ACT34, processor 101 performs character recognition processing on the input items. In other words, processor 101 recognizes the characters appearing in the image of the input items contained in the image currently represented by the data, and extracts the string displayed on the input items. For the specific method of character recognition processing, well-known methods such as deep learning using CNNs can be used. Thus, by having the processor 101 perform information processing based on the cart terminal program PRA, the computer with the processor 101 as its central component functions as a character recognition means.

[0033] As ACT35, processor 101 narrows down the multiple candidate products selected in ACT31, taking into account the results of the character recognition processing in ACT34. For example, processor 101 finds data record REA associated with each of the multiple candidate products in the dictionary database DBA. Then, processor 101 selects, for example, a data record REA from among the relevant data records REA in which any of the strings represented by the string information contained in the character information set in field FAD matches a string recognized by the character recognition processing in ACT34, and the products associated with that data record REA become the narrowed-down candidate products. If the number of matches between the strings represented by the string information contained in the character information set in field FAD and the strings recognized by the character recognition processing in ACT34 differs for each data record REA, processor 101 may also select the product associated with the data record REA with the larger number of matches as the narrowed-down candidate product.

[0034] Figure 6 shows an example of products that can be filtered by text. For example, as shown in Figure 6, suppose two product COA and COB products of the same brand but different capacities are sold in a store. Different products of the same brand have the same package design, as shown in the example in Figure 6, and may not be distinguishable by object recognition processing. In other words, in object recognition processing when product COA, as shown in Figure 6, is placed in shopping cart SB, either product COA or COB could be considered a candidate product. In such a case, if the string "350mL" can be obtained from the image of the item through character recognition processing, the candidate product will be narrowed down to product COA. However, if the string "350mL" displayed on the item is not visible in the image due to being hidden by the customer's hand or for other reasons, the candidate product cannot be narrowed down. Thus, by having the processor 101 perform information processing based on the cart terminal program PRA, the computer with the processor 101 as its central component functions as a filtering means.

[0035] In ACT36, processor 101 checks whether there is only one candidate product after narrowing down the candidates in ACT35. If processor 101 has not been able to narrow down the candidates to one product, it determines NO and proceeds to ACT37. As ACT37, processor 101 displays a selection screen on touch panel 104. The selection screen is a GUI (graphical user interface) screen that allows the customer to select one of several candidate products.

[0036] Figure 7 shows an example of a selection screen. The selection screen shown in Figure 7 is an example where the two product COA and COB shown in Figure 6 are selected as candidate products. The selection screen includes four buttons, BUA, BUB, BUC, and BUD, each assigned to a different product. In the example in Figure 7, products COA and COB are assigned to buttons BUA and BUB, respectively. Buttons BUA, BUB, BUC, and BUD are soft keys that allow the customer to specify the assigned product.

[0037] When the selection screen is displayed, the customer selects one of the candidate products by tapping the button assigned to the product they have placed in their shopping cart SB. With the selection screen displayed, processor 101 proceeds to ACT38 in Figure 5. As ACT38, processor 101 waits for one of the candidate products to be specified. If a product is specified through the above operation, processor 101 determines YES and proceeds to ACT39.

[0038] As ACT39, processor 101 sets only the specified products as candidate products. In other words, the processor 101 accepts an operation on the selection screen to specify one of several candidate products. Thus, by having the processor 101 perform information processing based on the cart terminal program PRA, the computer with the processor 101 as its central component functions as a means of receiving operations.

[0039] On the other hand, if the object recognition process in ACT31 fails to select any candidate products, the processor 101 determines NO in ACT33 and proceeds to ACT40. As ACT40, processor 101 displays a guidance screen on touch panel 104. The guidance screen is designed to instruct the customer to hold the barcode (hereinafter referred to as the product barcode) displayed on the item they just placed in shopping basket SB up to camera 199.

[0040] Figure 8 shows an example of a guidance screen. The guidance screen shown in Figure 8 includes area ARA. The processor 101 acquires image data output by the camera 199 and displays the image represented by that image data in area ARA. Processor 101, with the guidance screen displayed, proceeds to ACT41 in Figure 5.

[0041] As ACT41, processor 101 waits for the timing to acquire an image. When the timing to acquire the image is reached, processor 101 determines YES and proceeds to ACT42. The acquisition timing is expected to be, for example, at regular time intervals, and may be determined as appropriate by, for example, the creator of the cart terminal program PRA.

[0042] As ACT42, processor 101 acquires image data output by camera 199. At this time, processor 101 displays the image represented by the newly acquired image data in area ARA. This makes it easy for customers to hold the product up to camera 199 while confirming that the barcode is captured in the image from camera 199 by the image displayed in area ARA.

[0043] As ACT43, processor 101 performs product barcode extraction processing on the image data acquired by ACT42. In other words, processor 101 detects product barcodes that are visible in the image data acquired by ACT42, and if a product barcode is detected, it extracts that product barcode.

[0044] As ACT44, processor 101 checks whether the product barcode was extracted in ACT43. If the product is not properly positioned in front of camera 199 so that the product barcode is visible, processor 101 cannot extract the product barcode in ACT43. In such cases, processor 101 determines NO in ACT44, returns to the waiting state of ACT41, and repeats ACT41 to ACT44 until the product barcode can be extracted in ACT43. If the product barcode is extracted in ACT43, processor 101 determines YES in ACT44 and proceeds to ACT45.

[0045] As ACT45, processor 101 sets the product identified by the product code represented by the extracted product barcode as a candidate product. Thus, by having processor 101 perform information processing based on the cart terminal program PRA, the computer with processor 101 as its central component functions as a determination means for determining a candidate product as one of the candidate items based on the barcode.

[0046] Once processor 101 has finished setting one product as a candidate product in ACT39 or ACT45 as described above, it terminates the product identification process. If processor 101 has selected only one candidate product in ACT31, it determines YES in ACT32, skips the processes from ACT33 onwards, and terminates the product identification process. Also, if processor 101 has narrowed down the candidate products to only one, it determines YES in ACT36, skips the processes from ACT37 onwards, and terminates the product identification process. In other words, processor 101 terminates the product identification process with only one candidate product identified. After processor 101 has completed the product identification process as ACT18 in Figure 4 in this way, it proceeds to ACT19.

[0047] In this way, after going through ACT35, the processor 101 identifies the product as an item, as one of the candidate products narrowed down based on the characters recognized by the character recognition process. Thus, by having the processor 101 perform information processing based on the cart terminal program PRA, the computer with the processor 101 as its central component functions as an identification means.

[0048] As ACT19, processor 101 registers one candidate product identified by the product identification process as a purchased product. In other words, processor 101 updates the transaction data to include the product code of one candidate product identified by the product identification process as the product code of the purchased product. After this, processor 101 returns to the waiting state for ACT12 and ACT13. Thus, by processor 101 executing information processing based on the cart terminal program PRA, the computer with processor 101 as its central component functions as a registration means.

[0049] If a customer decides not to purchase an item they have placed in their shopping cart (SB), they must remove the item from the cart and return it to the sales floor. For example, in ACT14, the processor 101 checks whether an item has been removed from the shopping cart SB based on the difference between the newly acquired current data and past data. If it is confirmed that an item has been removed, it determines that the item should be taken out. Alternatively, the processor 101 may monitor the movement of the hand holding the item based on the current data and multiple past data points, and also determine whether the item should be taken out based on the results. If the processor 101 determines that the item should be taken out, it determines YES in ACT16 and proceeds to ACT20.

[0050] As ACT20, processor 101 detects items removed from shopping basket SB (hereinafter referred to as "removed items"). For example, processor 101 detects removed items based on the difference between current data and past data, identifying items that appear in the image represented by past data but not in the image represented by current data. As a method for this processing, well-known methods such as deep learning using CNNs can be used.

[0051] As ACT21, processor 101 performs product identification processing to determine which product the retrieved item detected in ACT21 is. This product identification processing is the same as ACT18 and is shown in Figure 5. As ACT22, processor 101 removes one candidate product identified by the product identification process from the purchased products. In other words, processor 101 updates the transaction data to reduce the quantity of the product code of the candidate product identified by the product identification process by one. After this, processor 101 returns to the waiting state for ACT12 and ACT13. Thus, by having processor 101 perform information processing based on the cart terminal program PRA, the computer with processor 101 as its central component functions as a cancellation mechanism.

[0052] Once the customer has finished registering their purchased items, they perform a predetermined operation, for example, on the touch panel 104, to initiate checkout. In response, the processor 101 determines YES in ACT 13 and proceeds to the checkout process. The checkout process may be the same as that performed in an existing cart POS system, and its illustration is omitted. For example, the processor 101 performs the process of transferring transaction data to the checkout machine 200 as the checkout process. Alternatively, if a payment terminal (not shown) is attached to the cart 900 and this payment terminal is connected to the cart terminal 100, the process of performing credit card payment or code payment using this payment terminal may be performed as the checkout process.

[0053] As described above, if the cart terminal 100 cannot narrow down the candidate products to one through object recognition processing, it narrows down the candidate products based on the characters obtained through character recognition processing, and identifies the product placed in the shopping cart SB as one of the narrowed-down candidate products. This allows for accurate identification of items that appear in the image when placed in the shopping cart SB, based on the image itself.

[0054] Furthermore, the cart terminal 100 identifies the product placed in the shopping cart SB as one of several candidate products in response to the customer's instructions on the selection screen. Therefore, even if it is not possible to narrow down the candidate products to one based on text, the product placed in the shopping cart SB can be identified. In this case, the customer needs to make a selection, but since the number of candidate products can be reduced by narrowing down based on text, the burden on the customer to select one of the candidate products is small.

[0055] Furthermore, if the cart terminal 100 fails to select any candidate products through object recognition processing, it identifies the products placed in the shopping cart SB based on their barcodes. This ensures that even if object recognition fails, the products in the shopping cart SB can be identified.

[0056] Furthermore, the cart terminal 100 registers the items identified as being placed in the shopping basket SB as purchased items. This makes it possible to realize the transaction processing system 1 as a registration system that can accurately and automatically register items as purchased items by recognizing items placed in a shopping basket by a customer in a display-type store from an image of the shopping basket.

[0057] This embodiment can be modified in various ways as follows: It is also acceptable to recognize items other than products.

[0058] The results of item recognition can be used for any purpose, not limited to purchase registration for transaction processing.

[0059] In the above embodiment, some of the processing performed by the processor 101 may be performed by the POS server 300 or any other one or more information processing devices. For example, the cart terminal 100 may only perform user interface operations, and the various information processing for identifying purchased items may be performed by a computer provided in the POS server 300. In other words, each of the selection means, character recognition means, filtering means, identification means, operation receiving means, determination means, and registration means may be provided in any information processing device other than the cart terminal 100 included in the transaction processing system 1, or may be appropriately distributed and provided in multiple information processing devices included in the transaction processing system 1.

[0060] Each function realized by the processor 101 through information processing can also be partially or entirely realized by hardware that performs non-program-based information processing, such as logic circuits. Furthermore, each of the above functions can also be realized by combining the above-mentioned hardware, such as logic circuits, with software control.

[0061] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. The invention described in the original claims of this application is listed below. [Note 1] A selection means for selecting one or more candidate items for the item based on the characteristics of the item shown in the image, A character recognition means for recognizing characters that appear in the image with respect to the said article, When multiple candidate items are selected by the selection means, a narrowing means is provided to narrow down the number of candidate items from among the multiple candidate items based on the characters recognized by the character recognition means. If one candidate item is selected by the selection means, the item appearing in the image as that candidate item is identified; if multiple candidate items are selected by the selection means, the identification means identifies the item appearing in the image as one of the candidate items narrowed down by the narrowing means. An item recognition device equipped with the following. [Note 2] When there are multiple candidate items narrowed down by the narrowing means, the system further includes an operation receiving means for receiving an operation to specify one of the multiple candidate items, The identification means identifies the article that appeared in the image as a candidate article specified by the operation received by the operation receiving means. The item recognition device described in Appendix 1. [Note 3] Further comprising a determination means for determining one candidate item based on a barcode, The identification means identifies the item that appeared in the image as a candidate item determined by the determination means when no candidate item is selected by the selection means. The item recognition device described in Appendix 1. [Note 4] When an item is placed in a container that holds the goods that the buyer intends to purchase, a detection means for detecting the item placed in the container from an image obtained by photographing the container, A selection means for selecting one or more candidate items for an item detected by the detection means based on features appearing in the image, A character recognition means for recognizing characters that appear in the image with respect to the said article, When multiple candidate items are selected by the selection means, a narrowing means is provided to narrow down the number of candidate items from among the multiple candidate items based on the characters recognized by the character recognition means. If one candidate item is selected by the selection means, the item appearing in the image as that candidate item is identified; if multiple candidate items are selected by the selection means, the identification means identifies the item appearing in the image as one of the candidate items narrowed down by the narrowing means. When the detection means detects an item placed in the container, the registration means registers the item identified by the identification means as an item to be purchased. A registration system equipped with the following features. [Note 5] When an article is removed from the container, the detection means detects the removed article from an image obtained by photographing the container before the removal. When the detection means detects an item removed from the container, a cancellation means removes the item identified by the identification means from the items to be purchased. The registration system described in Appendix 4 further includes the following: [Note 6] The computer installed in the item recognition device, A selection means for selecting one or more candidate items for an item based on the characteristics of the item shown in the image, A character recognition means for recognizing characters that appear in the image with respect to the said article, When multiple candidate items are selected by the selection means, a narrowing means is provided to narrow down the number of candidate items from among the multiple candidate items based on the characters recognized by the character recognition means. If one candidate item is selected by the selection means, the item appearing in the image as that candidate item is identified; if multiple candidate items are selected by the selection means, the identification means identifies the item appearing in the image as one of the candidate items narrowed down by the narrowing means. An information processing program that enables a function to work. [Explanation of symbols]

[0062] 1...Transaction processing system, 2...Communication network, 100...Cart terminal, 101...Processor, 102...Main memory unit, 103...Auxiliary memory unit, 104...Touch panel, 105...Interface unit, 106...Wireless communication unit, 107...Transmission line, 199...Camera, 200...Accounting machine, 300...POS server, 400...Attendant terminal, 900...Cart.

Claims

1. A detection means that, when an item is placed in a container that holds the goods that a buyer intends to purchase, detects the item placed in the container from an image obtained by photographing the inside of the container, and when an item is removed from the container, detects the item removed from the container based on the difference between a new image obtained by photographing the inside of the container and a previous image. A selection means for selecting one or more candidate items for an item detected by the detection means based on features appearing in the image, A character recognition means for recognizing characters that appear in the image with respect to the said article, When multiple candidate items are selected by the selection means, a narrowing means is provided to narrow down the number of candidate items from among the multiple candidate items based on the characters recognized by the character recognition means. If one candidate item is selected by the selection means, the item appearing in the image as that candidate item is identified; if multiple candidate items are selected by the selection means, the identification means identifies the item appearing in the image as one of the candidate items narrowed down by the narrowing means. When the detection means detects an item placed in the container, the registration means registers the item identified by the identification means as an item to be purchased. When an item removed from the container is detected by the detection means, a cancellation means is provided to remove the item identified by the identification means from the items to be purchased. An information processing terminal equipped with [the following features].

2. The system further includes an operation receiving means that receives an operation to specify one of the candidate items when there are multiple candidate items narrowed down by the aforementioned narrowing means, The identification means identifies the article that appeared in the image as a candidate article specified by the operation received by the operation receiving means. The information processing terminal according to claim 1.

3. The system further comprises a determination means for determining one candidate item based on a barcode, The identification means identifies the item that appeared in the image as a candidate item determined by the determination means when no candidate item is selected by the selection means. The information processing terminal according to claim 1.

4. A detection means that, when an item is placed in a container that holds the goods that a buyer intends to purchase, detects the item placed in the container from an image obtained by photographing the inside of the container, and when an item is removed from the container, detects the item removed from the container based on the difference between a new image obtained by photographing the inside of the container and a previous image. A selection means for selecting one or more candidate items for an item detected by the detection means based on features appearing in the image, A character recognition means for recognizing characters that appear in the image with respect to the said article, When multiple candidate items are selected by the selection means, a narrowing means is provided to narrow down the number of candidate items from among the multiple candidate items based on the characters recognized by the character recognition means. If one candidate item is selected by the selection means, the item appearing in the image as that candidate item is identified; if multiple candidate items are selected by the selection means, the identification means identifies the item appearing in the image as one of the candidate items narrowed down by the narrowing means. When the detection means detects an item placed in the container, the registration means registers the item identified by the identification means as an item to be purchased. When an item removed from the container is detected by the detection means, a cancellation means is provided to remove the item identified by the identification means from the items to be purchased. A registration system equipped with the following features.

5. The computer installed in the information processing terminal, A detection means that, when an item is placed in a container that holds the goods that a buyer intends to purchase, detects the item placed in the container from an image obtained by photographing the inside of the container, and when an item is removed from the container, detects the item removed from the container based on the difference between a new image obtained by photographing the inside of the container and a previous image. A selection means for selecting one or more candidate items for an item detected by the detection means based on features appearing in the image, A character recognition means for recognizing characters that appear in the image with respect to the said article, When multiple candidate items are selected by the selection means, a narrowing means is provided to narrow down the number of candidate items from among the multiple candidate items based on the characters recognized by the character recognition means. If one candidate item is selected by the selection means, the item appearing in the image as that candidate item is identified; if multiple candidate items are selected by the selection means, the identification means identifies the item appearing in the image as one of the candidate items narrowed down by the narrowing means. When the detection means detects an item placed in the container, the registration means registers the item identified by the identification means as an item to be purchased. When an item removed from the container is detected by the detection means, a cancellation means is provided to remove the item identified by the identification means from the items to be purchased. An information processing program that enables a function to work.

Images