Item recognition system, information processing program, and transaction processing system

Abstract

To increase the success rate of recognition when recognizing an article from an image showing the article moved by a hand.SOLUTION: The object recognition system according to an embodiment includes recognition means and guidance means. The recognition means recognizes an article on the basis of an image showing the article. The guidance means shows a user how to act to increase the accuracy of recognition by the recognition means according to the recognition means having failed the recognition of the article more than one time.SELECTED DRAWING: Figure 4

Description

Technical Field

[0004] , , , , , , , , , , , ,

[0005]

[0001] Embodiments of the present invention relate to an article recognition system, an information processing program, and a transaction processing system.

Background Art

[0002] In a store of the display sales type, a registration system has already been proposed that recognizes products placed by a customer in the basket part of a shopping cart from an image and automatically registers them as products to be purchased by the customer. A POS system using such a registration system may sometimes be called a frictionless POS system or the like. However, depending on how the customer holds the product, many of the products may be hidden by the hand and not fully reflected in the image, making it impossible to correctly recognize them. Due to such circumstances, it has been desired to improve the recognition success rate when recognizing an article from an image in which the article being held and moved is reflected.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to provide an article recognition system, an information processing program, and a transaction processing system capable of improving the recognition success rate when recognizing an article from an image in which the article being held and moved is reflected.

Means for Solving the Problems

[0005] The object recognition system of the embodiment comprises recognition means and guidance means. The recognition means recognizes an object based on an image in which the object is reflected. The guidance means guides the user to take actions to improve the accuracy of recognition by the recognition means in response to repeated failures by the recognition means to recognize the object. The recognition means starts processing for recognizing an item when the item appears in the image and ends when recognition is successful or when the item is no longer visible in the image. The guidance means considers the termination of the item recognition process when the item is no longer visible in the image as a failure of item recognition by the recognition means. [Brief explanation of the drawing]

[0006] [Figure 1] A block diagram showing the schematic configuration of a transaction processing system according to one embodiment, and the main circuit configuration of a cart terminal. [Figure 2] A perspective view showing an example of how a cart terminal is attached to a cart. [Figure 3] Flowchart of the registration process. [Figure 4] Flowchart of the registration process. [Figure 5] A diagram showing an example of a guidance screen. [Modes for carrying out the invention]

[0007] An example of an embodiment will be described below with reference to the drawings. This embodiment will describe an example of a transaction processing system that incorporates the functions of an item recognition system. Figure 1 is a block diagram showing the schematic configuration of the transaction processing system 1 according to this embodiment and the main circuit configuration of the cart terminal 100.

[0008] Transaction processing system 1 is an information processing system for processing transactions involving the sale of goods to customers in a store. Transaction processing system 1 is configured so that a cart terminal 100, a checkout machine 200, an attendant terminal 300, a management terminal 400, and a POS server 500 can communicate with each other via a communication network 2. Communication network 2 can use the internet, VPN (virtual private network), LAN (local area network), public communication network, mobile communication network, etc., individually or in appropriate combinations. As an example, communication network 2 can be used by combining a LAN and the internet. Note that the transaction processing system 1 may contain any number of cart terminals 100, accounting machines 200, attendant terminals 300, management terminals 400, and POS servers 500, but Figure 1 shows only one of each.

[0009] The cart terminal 100 is an information processing terminal attached to a shopping cart (hereinafter referred to as "cart") provided in the store. The cart terminal 100 is lent to customers along with the cart when they purchase goods in the store. The cart terminal 100 is a terminal device that, upon receiving an operation from the customer, performs a registration process to register the goods that the customer intends to purchase as transaction items. As described later, the cart terminal 100 performs a process to recognize the goods to be registered as transaction items from an image in which the goods are captured. In other words, the cart terminal 100 has the function of a product recognition device.

[0010] The accounting machine 200 is installed in the store and performs accounting processing related to the accounting and settlement of the price of the traded goods registered by the cart terminal 100. The accounting machine 200 is operated by an operator during the accounting process. The operator of the accounting machine 200 is mainly a customer. In some cases, a store employee may operate the accounting machine 200. For example, the accounting machine 200 can be the same one used in an existing cart POS system.

[0011] The attendant terminal 300 is an information processing terminal operated by a store employee. The attendant terminal 300 is a terminal device for a user interface related to information processing, which supports the work of the store employee regarding transactions processed by the transaction processing system 1. The work of the store employee is, for example, to monitor the progress of a transaction in progress and to support the customer as appropriate. Various general-purpose information processing devices such as desktop, book, tablet, and mobile types can be used as the basic hardware for the attendant terminal 300. The attendant terminal 300 can then be realized by installing an information processing program that realizes the various functions of the attendant terminal 300 on the computer equipped with such basic hardware.

[0012] The management terminal 400 is an information terminal that serves as a user interface for administrators managing the transaction processing system 1. Various general-purpose information processing devices, such as desktop, book-type, tablet, and mobile types, can be used as the basic hardware for the management terminal 400. The management terminal 400 can then be realized by installing information processing programs that implement the various functions of the management terminal 400 onto the computer of such basic hardware.

[0013] The POS server 500 is an information processing device that performs management processing to manage the buying and selling transactions of goods processed by the registration process at the cart terminal 100 and the accounting process at the accounting machine 200. As the basic hardware for the POS server 500, a suitable information processing device such as a general-purpose server device can be used. The POS server 500 can then be realized by installing an information processing program that implements the various functions of the POS server 500 onto the computer of such basic hardware.

[0014] The cart terminal 100 includes a processor 101, a main memory unit 102, an auxiliary memory unit 103, a touch panel 104, a sound unit 105, an interface unit 106, a wireless communication unit 107, and a transmission line 108. The processor 101 and the main memory unit 102, auxiliary memory unit 103, touch panel 104, sound unit 105, interface unit 106, and wireless communication unit 107 are able to communicate with each other via the transmission line 108.

[0015] By connecting the processor 101, the main memory unit 102, and the auxiliary memory unit 103 with a transmission line 108, a computer is configured to perform information processing for controlling the cart terminal 100. The processor 101 corresponds to the central part of the computer described above. The processor 101 performs information processing to control each part in order to realize various functions as a cart terminal 100, in accordance with information processing programs such as the operating system and application programs.

[0016] The main memory unit 102 corresponds to the main memory portion of the computer described above. The main memory unit 102 includes a read-only memory area and a rewritable memory area. The main memory unit 102 stores a portion of the information processing program described above in the read-only memory area. The main memory unit 102 may also store data necessary for the processor 101 to perform processing to control each part in the read-only memory area or the rewritable memory area. The main memory unit 102 uses the rewritable memory area as a work area for the processor 101.

[0017] The auxiliary storage unit 103 corresponds to the auxiliary storage part of the above computer. The auxiliary storage unit 103 can use, for example, an EEPROM (electric erasable programmable read-only memory), an HDD (hard disc drive), an SSD (solid state drive), or other various well-known storage devices. The auxiliary storage unit 103 stores the data used by the processor 101 to perform various processes and the data generated by the processes in the processor 101. The auxiliary storage unit 103 may also store the above information processing program. In this embodiment, the auxiliary storage unit 103 stores a cart terminal program PRA, which is one of the information processing programs. The cart terminal program PRA is an application program that describes the procedure of the registration process. A part of the storage area of the auxiliary storage unit 103 is used as an area for storing transaction data DAA. The transaction data DAA is data representing the content of one transaction.

[0018] The touch panel 104 displays a screen for presenting information to the operator. The touch panel 104 also inputs an instruction by a touch operation on the screen by the operator. The sound unit 105 outputs sounds for various guides and alarms. As the sound unit 105, for example, various well-known sound devices such as a voice synthesis device and a buzzer can be used alone or in combination.

[0019] The interface unit 106 is connected to an external device such as a camera 199. The interface unit 106 interfaces the exchange of data with the connected external device. As the interface unit 106, an existing USB (universal serial bus) controller or the like can be used. Note that the camera 199 may repeat shooting at a fixed cycle or may perform shooting according to an instruction from the processor 101. Instead of the camera 199, a camera built in the cart terminal 100 may be used.

[0020] The wireless communication unit 107 performs communication processing for wireless data communication via the communication network 2. For example, an existing wireless communication device for a wireless LAN can be used as the wireless communication unit 107. Alternatively, a communication unit connected to the communication network 2 via a wired connection may be used instead of, or in addition to, the wireless communication unit 107. The transmission line 108 includes an address bus, a data bus, and control signal lines, and transmits data and control signals exchanged between the connected parts.

[0021] The basic hardware of the cart terminal 100 is expected to be, for example, the hardware of a tablet-type information processing device. Generally, the cart terminal 100 is transferred with the cart terminal program PRA stored in the auxiliary storage unit 103. However, the hardware and the cart terminal program PRA may be transferred separately, either without the cart terminal program PRA stored in the auxiliary storage unit 103, or with a different version of the same type of application program stored in the auxiliary storage unit 103. Furthermore, the cart terminal 100 may be configured by writing the cart terminal program PRA to the auxiliary storage unit 103 in response to an operation by any worker. The cart terminal program PRA can be transferred by recording it on a removable recording medium such as a magnetic disk, magneto-optical disk, optical disk, or semiconductor memory, or by communication over a network.

[0022] Figure 2 is a perspective view showing an example of how the cart terminal 100 is attached to the cart. The cart 900 comprises a caster section 910, a handle frame section 920, a basket section 930, a support section 940, and a battery case 950. The caster section 910 has four wheels 911 for smoothly moving the cart 900 on the floor. The wheels 911 are mounted on the frame 912 so as to be rotatable around a vertical axis.

[0023] The handle frame section 920 includes a pair of vertical frames 921, 921 and a handlebar 922. The vertical frames 921, 921 are erected above the two wheels of the caster section 910. The handlebar 922 connects the upper ends of the vertical frames 921, 921. The basket section 930 is provided horizontally from the middle of the handle frame section 920. The basket section 930 is basket-shaped with an opening at the top that allows goods to be placed inside, and functions as a container for holding goods that a customer intends to purchase. A camera 199 is mounted on the upper end of the basket section 930 on the side closest to the handle bar 922, in a manner that allows it to photograph the opening of the basket section 930 from the side. The shooting range of the camera 199 is set so that the goods placed inside the basket section 930 are captured through the opening of the basket section 930.

[0024] The support section 940 includes a pole 941. The pole 941 is attached to one of the vertical frames 921 such that its tip is positioned above the handlebar 922. The aforementioned cart terminal 100 is attached to the tip of this pole 941. As a result, the support section 940 supports the cart terminal 100 in the state shown in Figure 2. The battery case 950 is mounted between the vertical frames 921, 921 at the lower end of the handle frame 920. The battery case 950 houses the battery used as an external power source for the cart terminal 100.

[0025] Next, the operation of the transaction processing system 1 configured as described above will be explained. Note that the content of the various processes described below is just an example, and it is possible to change the order of some processes, omit some processes, or add other processes as appropriate. For example, in the following explanation, the explanation of some processes has been omitted in order to explain the characteristic operation of this embodiment in an easy-to-understand manner. For example, if some error occurs, processing to deal with that error may be performed. Also, for example, in various waiting states, the system may wait for the occurrence of events other than those described and proceed to processing according to the event that occurred.

[0026] Customers of a retail store, that is, customers who intend to purchase goods displayed and sold at the store, perform predetermined operations to start using the unused shopping cart terminal 100. Once these operations are performed, the processor 101 starts the registration process based on the shopping cart terminal program PRA.

[0027] Figures 3 and 4 are flowcharts of the registration process. As ACT11 in Figure 3, the processor 101 generates new transaction data DAA for the target transaction. Specifically, the processor 101 determines a new transaction code, for example, that is different from the transaction code used as an identifier to identify other transactions, according to predetermined rules, generates new transaction data DAA that includes this transaction code but does not include product information about the traded product, and stores it in the auxiliary storage unit 103.

[0028] As ACT12, processor 101 displays a registration screen on touch panel 104. The registration screen is a screen that allows the customer to recognize the registration status of the trading products. In other words, the registration screen displays, for example, a list of registered trading products. However, when ACT12 is executed, the trading products have not yet been registered, so the screen is used to let the customer know that the trading products are not yet registered. In addition to the list of trading products, the registration screen may also include various display objects of any choice. An example of a display object that the registration screen may include is a string representing the provisional price of the registered trading products. Another example of a display object that the registration screen may include is a soft key for receiving various specifications from the operator. The content of the registration screen may be determined as appropriate by, for example, the creator of the shopping cart terminal program PRA.

[0029] As ACT13, processor 101 checks if it is time to acquire an image. If processor 101 cannot confirm the event, it determines NO and proceeds to ACT14. As ACT14, processor 101 checks whether the re-recognition described later has been specified. If processor 101 cannot confirm the relevant event, it determines NO and proceeds to ACT15. As ACT15, processor 101 checks if accounting has been requested. If processor 101 cannot confirm the event, it determines NO and returns to ACT13. Thus, processor 101 waits for either an acquisition timing, a re-recognition to be specified, or an accounting to be specified as ACT13 to ACT15.

[0030] The acquisition timing is expected to be, for example, immediately after progressing from ACT12 to ACT13, and at regular time intervals thereafter. The acquisition timing may be determined as appropriate by, for example, the creator of the cart terminal program PRA. However, the acquisition timing should be determined so that it occurs at least once, preferably multiple times, while the product to be placed in the basket section 930 is passing through the shooting range of the camera 199.

[0031] When it's time to acquire the data, processor 101 determines YES in ACT13 and proceeds to ACT16. As ACT16, the processor 101 acquires image data output by the camera 199. For example, if the camera 199 is taking pictures at regular intervals, the processor 101 acquires the image data output by the camera 199 in accordance with the first picture taken after the acquisition timing. For example, if the camera 199 takes pictures in response to instructions from the processor 101, the processor 101 instructs the camera 199 to take a picture, and acquires the image data output by the camera 199 in response. The processor 101 then stores the acquired image data in the main storage unit 102 or the auxiliary storage unit 103.

[0032] As ACT17, processor 101 detects the area in which the product is included (hereinafter referred to as the product area) from the image represented by the acquired image data. For determining the area to be detected, it is possible to use well-known methods such as deep learning using a CNN (convolutional neural network) or graph cuts, and the method used is not limited.

[0033] As ACT18, processor 101 checks whether it successfully detected the product area. If it failed to detect the product area in ACT17, processor 101 determines NO and proceeds to ACT19. Thus, processor 101 proceeds to ACT19 when the product is not visible in the image captured by camera 199. As ACT19, processor 101 checks if the product is being recognized. If processor 101 cannot confirm the relevant event, it determines NO and returns to the waiting state of ACT13-ACT15.

[0034] The customer searches for the item they wish to purchase within the store. The customer then takes the item they intend to buy from the display and places it in the shopping basket 930. At this point, the item enters the field of view of camera 199 and is captured in the image data output by camera 199. In other words, until the customer attempts to place the item in the shopping basket 930, the processor 101 does not detect the item area in ACT17, even when it is time to acquire the data. Therefore, product recognition has not begun, and the processor 101 determines NO in both ACT18 and ACT19, returning to the waiting state of ACT13 to ACT15. When the item that the customer is about to place in the shopping basket 930 appears in the image, the processor 101 detects the item area in ACT17. In this case, the processor 101 determines YES in ACT18 and proceeds to ACT20. As ACT20, processor 101 performs recognition start processing. Recognition start processing is a process to confirm that the product that has entered the shooting range of camera 199 is being recognized. For example, processor 101 sets the recognition flag to a state that indicates that recognition is in progress.

[0035] As ACT21, processor 101 performs recognition processing. Recognition processing is the process of recognizing products that are captured in the product area based on the image of the product area detected in ACT17. This recognition processing may be either product recognition based on the appearance of the product, product recognition based on the product barcode displayed on the product, or both. For product recognition based on the appearance of the product, it is expected that deep learning using CNN, a method suitable for image recognition, will be used. Alternatively, for product recognition based on the appearance of the product, the method may be limited to using OCR (optical character recognition) or feature point matching. Thus, by processor 101 executing information processing based on the cart terminal program PRA, the computer with processor 101 as its central component functions as a recognition means. Furthermore, if product recognition based on the product barcode displayed on the product is applied to the recognition processing, another device such as a laser scanner that detects the pattern of the product barcode without relying on an image may be used instead of or in addition to camera 199.

[0036] As ACT22, processor 101 checks whether recognition in ACT20 was successful. If recognition was successful, processor 101 determines YES and proceeds to ACT23. As ACT23, processor 101 registers the product recognized in ACT21 as a trading product. In other words, processor 101 updates the transaction data DAA to include the product information of the product recognized in ACT21 as the product information of the trading product. Processor 101 includes at least the product code as a product identifier in the product information. Processor 101 may also include the product name and unit price in the product information. The information that processor 101 includes in the product information may be determined as appropriate by, for example, the creator of the shopping cart terminal program PRA. Thus, by having the processor 101 perform information processing based on the cart terminal program PRA, the computer with the processor 101 as its central component functions as a registration means.

[0037] As ACT24, processor 101 updates the registration screen. For example, processor 101 updates the registration screen to display on touch panel 104 a list that includes the trading products registered in ACT23. As ACT25, processor 101 performs recognition termination processing. Recognition termination processing is a process to confirm that recognition of the next product to enter the shooting range of camera 199 has not yet begun. For example, processor 101 sets the recognition in progress flag to a state that recognition is not in progress.

[0038] As ACT26, processor 101 waits for the recognized product to complete its passage through the camera 199's field of view. For example, as long as processor 101 can successfully detect the product area using the same process as in ACT17, it repeats the same process as in ACT16-ACT18 each time it is time to acquire the product. When processor 101 can no longer detect the product area, it determines that passage is complete and returns to the waiting state of ACT13-ACT15.

[0039] If the processor 101 fails to recognize the product during the recognition process in ACT21, it will determine NO in ACT22, skip ACT23 to ACT26, and return to the waiting state of ACT13 to ACT15. In other words, even if a product is visible in the image, if the processor 101 is unable to recognize the product from the image, it will return to the waiting state of ACT13 to ACT15 while remaining in the recognition in progress state.

[0040] While a single trading item passes through the camera 199's field of view, multiple acquisition opportunities may arise. The way the item is captured in the image data obtained at these multiple acquisition opportunities will change. Therefore, in the recognition process in ACT21, which is repeated by the processor 101 each time an acquisition opportunity arises, the item may not have been recognized at previous acquisition opportunities, but may be recognized at the most recent acquisition opportunity. In this case, the processor 101 executes ACT23 to ACT26 and registers the recognized item as a trading item.

[0041] However, in cases where a customer is holding an item with their hands covering most of it, the processor 101 may not be able to recognize the item in the recognition process at ACT21, regardless of the timing of acquisition while a single transaction item passes through the camera 199's field of view. If the processor 101 proceeds to ACT19 because the item has moved out of the camera 199's field of view and the item area can no longer be detected at ACT17, it determines that the item is being recognized, for example, based on the recognition flag indicating that the item is being recognized, and proceeds to ACT31 in Figure 4.

[0042] As ACT31, the processor 101 displays a recognition error screen on the touch panel 104. The recognition error screen is a screen that makes the customer aware that the product placed in the basket 930 immediately before could not be recognized. The recognition error screen is expected to be a screen that displays a text message to make the customer aware that the product placed in the basket 930 immediately before could not be recognized. The content of the recognition error screen may be determined as appropriate by, for example, the creator of the cart terminal program PRA. At this time, the processor 101 may also execute a notification operation other than screen display in combination with the display of the recognition error screen as appropriate. A notification operation other than screen display may be, for example, an alarm sound or voice message output from the sound unit 105. A notification operation other than screen display may be, for example, the lighting or flashing of an alarm lamp, which is not shown in Figure 1. A notification operation other than screen display may be, for example, the generation of vibration by a vibrator, which is not shown in Figure 1.

[0043] As ACT32, processor 101 updates the failure management information. Failure management information is information for managing the number of times a product has failed to be recognized. Processor 101 updates the failure management information so that the cumulative number of failures can be determined based on the updated failure management information. Typically, processor 101 increases the value represented by the variable as failure management information by one. However, processor 101 may also increase the value represented by the variable as failure management information by two or more fixed numbers. Alternatively, processor 101 may also decrease the value represented by the variable as failure management information by a fixed number. Alternatively, processor 101 may use information other than a variable as failure management information and change that information according to predetermined rules.

[0044] As ACT33, processor 101 checks whether predetermined guidance execution conditions have been met. Guidance execution conditions are criteria for determining whether or not to guide the customer on actions to improve the accuracy of product recognition. For example, if a variable has been increased in ACT32, the guidance execution conditions are expected to be defined as "the value represented by the variable as failure management information is greater than or equal to a specified value." However, the guidance execution conditions may be defined as "the value represented by the variable as failure management information is greater than a default value," "the variable as failure management information represents a default value," or "the failure management information matches the specified information," as appropriate by the creator of the cart terminal program PRA. However, the guidance execution conditions are defined to be met when the number of failures that can be determined from the failure management information is 2 or more, as arbitrarily determined. For example, processor 101 updates the variable VA, which has an initial value of "0", to "VA+1" as ACT32. Then, as ACT33, processor 101 checks whether "VA≧3" is true.

[0045] Then, if the processor 101 cannot confirm that the guidance execution conditions are met, it determines NO and proceeds to ACT34. In other words, the first time the processor 101 proceeds to ACT32, it determines NO and proceeds to ACT34. As ACT34, processor 101 displays the registration screen on touch panel 104. In other words, processor 101 displays the registration screen on touch panel 104 again, showing a list of traded products that have been registered at this point. After this, processor 101 returns to the standby state of ACT13~ACT15 in Figure 3.

[0046] When a customer learns that recognition has failed via the recognition error screen, they remove the item from the basket unit 930, and after the registration screen is displayed by ACT34 or ACT39 in Figure 4, they place the item back into the basket unit 930. At this time, the processor 101 attempts to recognize the item again using ACT13, ACT16~ACT18, and ACT21, and if recognition is successful, it registers it as a transaction item as ACT23. In this case, since the way the item is reflected in the image obtained by the camera 199 is different from the previous time, there is a possibility that it can be recognized correctly.

[0047] Now, in the recognition process in ACT21 in Figure 3, processor 101 may mistakenly recognize an item as a different item from the one pictured in the image. However, even in such cases, processor 101 will still determine YES in ACT22 and register it as a traded item. If the customer then confirms, based on the list displayed on the registration screen, that a different product than the one placed in the shopping cart 930 has been registered as the transaction item, they can, for example, specify a re-recognition of the incorrectly registered transaction item through a predetermined operation on the touch panel 104. Once this specification is made, the processor 101 determines YES at ACT14 in Figure 3 and proceeds to ACT35 in Figure 4.

[0048] As ACT35, processor 101 removes the product designated for re-recognition from the trading products. For example, processor 101 deletes the product information for the product in question from the trading data DAA. After this, processor 101 executes ACT32 and subsequent steps in the same manner as described above. In other words, processor 101 counts the number of failures, considering two cases as recognition failures: when it fails to recognize the product while the product is passing through the camera 199's field of view, and when the customer determines that the recognition result is incorrect.

[0049] If the customer requests re-recognition, they will first remove the item from the shopping cart 930 and then place it back into the cart 930. At this time, the processor 101 attempts to recognize the item again using ACT13, ACT16-ACT18, and ACT21. If recognition is successful, it registers the item as a transaction item using ACT23. In this case, the way the item is reflected in the image obtained by the camera 199 is different from the previous time, so there is a possibility of correct recognition. If the correct transaction item is registered through this process, the aforementioned incorrect recognition result is corrected by the new recognition result. Thus, by having the processor 101 execute information processing based on the cart terminal program PRA, the computer with the processor 101 as its central component functions as a correction mechanism.

[0050] If the camera 199 fails to recognize the product while it is passing through its field of view, or if the customer determines that the recognition result is incorrect, and these conditions for executing the guidance are met multiple times, the processor 101 determines YES in ACT33 in Figure 4 and proceeds to ACT37.

[0051] As ACT37, processor 101 executes guidance processing. Guidance processing is the process of guiding the customer on actions to improve the accuracy of product recognition. For example, as part of the guidance processing, processor 101 displays a predetermined guidance screen on the touch panel 104 for the relevant guidance.

[0052] Figure 5 is a diagram showing an example of a guidance screen (SCA). The SCA (Screen Display A) represents the image IMA (Image Image A), text CSA (Text Script A), and button BUA (Button Script A). Image IMA is a photograph or illustration used to help the customer recognize camera 199, which is used for product recognition. String CSA represents a text message guiding the customer's actions. Button BUA is a soft key used by the customer to indicate that they understand the guidance provided by the guidance screen SCA.

[0053] The processor 101 may, in addition to or instead of displaying the guidance screen, output an audio message to the sound unit 105 as part of the guidance processing. The processor 101 waits for a predetermined termination condition to be met before proceeding to ACT 35. For example, the processor 101 determines that the termination condition has been met when the customer specifies confirmation completion through a predetermined operation such as tapping button BUA. Alternatively, the processor 101 determines that the termination condition has been met when the duration of the guidance screen display reaches a predetermined time. The content of the guidance screen, or the processes performed during the guidance process, may be determined as appropriate by, for example, the creator of the shopping cart terminal program PRA. Thus, by having the processor 101 perform information processing based on the cart terminal program PRA, the computer with the processor 101 as its central component functions as a guidance means.

[0054] As ACT38, processor 101 initializes the failure management information. In other words, if processor 101 updates the variable VA, which has an initial value of "0", to "VA+1" as ACT32, it updates the value represented by variable VA to "0". As ACT39, processor 101 displays the registration screen on touch panel 104. In other words, processor 101 displays the registration screen on touch panel 104 again, showing a list of traded products that have been registered at this point. After this, processor 101 returns to the standby state of ACT13~ACT15 in Figure 3.

[0055] Once the buyer has finished registering their purchased items, they perform a predetermined operation to specify payment, for example, on the touch panel 104. In response, the processor 101 determines YES in ACT 15 in Figure 3 and proceeds to the payment process. The payment process can be the same as that performed in an existing cart POS system, and its illustration is omitted. For example, the processor 101 performs the process of transferring transaction data to the payment machine 200 as the payment process. Alternatively, if a payment terminal (not shown) is attached to the cart 900 and this payment terminal is connected to the cart terminal 100, the payment process may be the process of performing credit card payment or code payment using this payment terminal.

[0056] Thus, in the transaction processing system 1, if recognition failures occur repeatedly, the cart terminal 100 performs guidance processing to guide the customer on actions to improve the accuracy of product recognition. If the customer recognizes the actions to improve the accuracy of product recognition through this guidance processing and takes those actions, the success rate of recognizing the item from the image of the item being moved while held in the customer's hand can be improved.

[0057] Furthermore, the cart terminal 100 counts the number of failures, recording both cases where it fails to recognize the product while it is passing through the camera 199's field of view, and cases where the customer determines that the recognition result is incorrect. This allows the processor 101 to correctly evaluate even cases where it registers a product as a successful recognition despite a misrecognition, and reflect this in determining whether or not guidance processing is necessary.

[0058] This embodiment can be modified in various ways as follows: The processor 101 may perform guidance processing only in response to the repeated occurrence of either the case in which the product could not be recognized while it was passing through the shooting range of the camera 199, or the case in which the customer determined that the recognition result was incorrect.

[0059] The processor 101 may, in response to repeated instances where it fails to recognize the product while it passes through the camera 199's field of view, perform a first guidance action adapted to that situation, and in response to repeated instances where the customer determines that the recognition result is incorrect, perform a second guidance action, different from the first guidance action, adapted to that situation.

[0060] It may be used in combination with other product registration devices or transaction processing devices, and may be implemented as a product recognition device whose primary function is product recognition.

[0061] The results of the item recognition process can be used for any purpose, not limited to registering trading items for transaction processing.

[0062] It may also be implemented as a device that recognizes items other than products.

[0063] In the above embodiment, some of the processing performed by the processor 101 may be performed by the POS server 500 or any other one or more information processing devices. For example, the cart terminal 100 may only perform user interface operations, and the various information processing for identifying the transaction items may be performed by a computer provided in the POS server 500. In other words, the recognition means, guidance means, modification means, and registration means may each be provided in any information processing device other than the cart terminal 100 included in the transaction processing system 1, such as the POS server 500, or they may be appropriately distributed and provided in multiple information processing devices included in the transaction processing system 1.

[0064] Each function realized by the processor 101 through information processing can also be partially or entirely realized by hardware that performs non-program-based information processing, such as logic circuits. Furthermore, each of the above functions can also be realized by combining the above-mentioned hardware, such as logic circuits, with software control.

[0065] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. The invention described in the original claims of this application is listed below. [Note 1] Recognition means for recognizing an object based on an image in which the object is reflected, A guidance means that, in response to repeated failures by the recognition means to recognize an item, guides the user to take actions to improve the accuracy of recognition by the recognition means, An item recognition system equipped with the following features. [Note 2] The recognition means recognizes the article based on the external features of the article captured in the image. The item recognition system described in Appendix 1. [Note 3] The recognition means recognizes an item based on the information represented by a barcode displayed on the item captured in the image. The item recognition system described in Appendix 1 or Appendix 2. [Note 4] The recognition means starts the process for recognizing an article when the article is captured in the image, and ends when recognition is successful or when the article is no longer captured in the image. The guidance means terminates the process for recognizing an item when the item is no longer visible in the image, and this is considered a failure of the recognition of the item by the recognition means. The item recognition system described in Appendix 1. [Note 5] Correction means for correcting the recognition result by the recognition means in accordance with instructions from the operator, Furthermore, The guidance means determines that the correction made by the correction means constitutes a failure of the recognition of the article by the recognition means. The item recognition system described in Appendix 1 or Appendix 4. [Note 6] The computer included in the object recognition system Recognition means for recognizing an object based on an image in which the object is reflected, A guidance means that, in response to repeated failures by the recognition means to recognize an item, guides the user to take actions to improve the accuracy of recognition by the recognition means, An information processing program that enables a function to work. [Note 7] Recognition means for recognizing an object based on an image in which the object is reflected, A registration means for registering an item recognized by the recognition means as a commodity subject to trade, A guidance means that, in response to repeated failures by the recognition means to recognize an item, guides the user to take actions to improve the accuracy of recognition by the recognition means, A transaction processing system equipped with the following features. [Explanation of symbols]

[0066] 1...Transaction processing system, 2...Communication network, 100...Cart terminal, 101...Processor, 102...Main memory unit, 103...Auxiliary memory unit, 104...Touch panel, 105...Sound unit, 106...Interface unit, 107...Wireless communication unit, 108...Transmission line, 199...Camera, 200...Accounting machine, 300...Attendant terminal, 400...Management terminal, 500...POS server, 900...Cart, 910...Caster section, 920...Handle frame section, 930...Basket section, 940...Support section, 950...Battery case.

Claims

1. Recognition means for recognizing an object based on an image in which the object is reflected, A guidance means that, in response to repeated failures by the recognition means to recognize an item, guides the user to take actions to improve the accuracy of recognition by the recognition means, It is equipped with, The recognition means starts processing for recognizing an item when the item appears in the image, and ends processing when recognition is successful or when the item is no longer visible in the image. The guidance means terminates the process for recognizing an item when the item is no longer visible in the image, and this is considered a failure of the recognition of the item by the recognition means. Item recognition system.

2. The recognition means recognizes an article based on the external features of the article captured in the image. The article recognition system according to claim 1.

3. The recognition means recognizes an item based on the information represented by a barcode displayed on the item captured in the image. The article recognition system according to claim 1 or claim 2.

4. The computer in the object recognition system Recognition means for recognizing an object based on an image in which the object is reflected, A guidance means that, in response to repeated failures by the recognition means to recognize an item, guides the user to take actions to improve the accuracy of recognition by the recognition means, and make it work Furthermore, the recognition means starts processing for recognizing an item when the item appears in the image, and ends when recognition is successful or when the item is no longer visible in the image. The guidance means determines that the process for recognizing an item ends when the item is no longer visible in the image, and this is considered a failure of the recognition of the item by the recognition means. Information processing program.

5. Recognition means for recognizing an object based on an image in which the object is reflected, A registration means for registering an item recognized by the recognition means as a commodity subject to trade, A guidance means that, in response to repeated failures by the recognition means to recognize an item, guides the user to take actions to improve the accuracy of recognition by the recognition means, It is equipped with, The recognition means starts processing for recognizing an item when the item appears in the image, and ends processing when recognition is successful or when the item is no longer visible in the image. The guidance means terminates the process for recognizing an item when the item is no longer visible in the image, and this is considered a failure of the recognition of the item by the recognition means. Transaction processing system.

6. The claim further comprises exclusion means for excluding a registered product from the subject of a transaction in response to a re-recognition of the registered product by the registration means, The guidance means, when exclusion is performed by the exclusion means, indicates that the recognition means has failed to recognize the item. The transaction processing system according to claim 5.

Images