Object recognition system, information processing program, and transaction processing system

Abstract

To increase the success rate of recognition when recognizing an article from an image showing the article moved by a hand.SOLUTION: The object recognition system according to an embodiment includes recognition means and guidance means. The recognition means recognizes an article on the basis of an image showing the article. The guidance means shows a user how to act to increase the accuracy of recognition by the recognition means according to the recognition means having failed the recognition of the article more than one time.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] An embodiment of the present invention relates to an item recognition system, an information processing program, and a transaction processing system. [Background technology]

[0002] A registration system has already been proposed for display sales stores that automatically registers the products that customers will purchase by recognizing the images of the products that they put in their shopping carts. POS systems that use such registration systems are sometimes called frictionless POS systems. However, depending on how a customer holds the product, there is a risk that the product will not be fully captured in the image due to circumstances such as the product being hidden by the hand, making it impossible to correctly recognize the product. In view of these circumstances, it has been desirable to improve the success rate of recognizing an object from an image in which the object is captured and moved by hand. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-45990 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem that the present invention aims to solve is to provide an item recognition system, an information processing program, and a transaction processing system that can improve the success rate of recognizing an item from an image that captures the item being held and moved by hand. [Means for solving the problem]

[0005] An embodiment of an item recognition system includes a recognition unit and a guidance unit. The recognition unit recognizes an item based on an image in which the item is captured. The guidance unit guides a user on an action to be taken by the user to improve the accuracy of recognition by the recognition unit in response to repeated failure of the recognition unit to recognize the item. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of a transaction processing system according to an embodiment and a main circuit configuration of a cart terminal. [Diagram 2] FIG. 13 is a perspective view showing an example of a state in which the cart terminal is attached to the cart. [Diagram 3] 13 is a flowchart of a registration process. [Figure 4] 13 is a flowchart of a registration process. [Diagram 5] FIG. 13 is a diagram illustrating an example of a guide screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] An example of an embodiment will be described below with reference to the drawings. Note that as this embodiment, an example of a transaction processing system implemented by incorporating the functions of an item recognition system will be described. FIG. 1 is a block diagram showing a schematic configuration of a transaction processing system 1 according to this embodiment and a main circuit configuration of a cart terminal 100. As shown in FIG.

[0008] The transaction processing system 1 is an information processing system for processing sales of products to customers in a store. The transaction processing system 1 is configured so that a cart terminal 100, a billing machine 200, an attendant terminal 300, a management terminal 400, and a POS server 500 can communicate with each other via a communication network 2. The communication network 2 may be the Internet, a virtual private network (VPN), a local area network (LAN), a public communication network, a mobile communication network, or the like, either alone or in appropriate combination. As an example, the communication network 2 may be a combination of a LAN and the Internet. Although any number of cart terminals 100, accounting machines 200, attendant terminals 300, control terminals 400, and POS servers 500 may be included in transaction processing system 1, only one of each is shown in FIG.

[0009] The cart terminal 100 is an information processing terminal attached to a shopping cart (hereinafter referred to as a cart) provided in a store. The cart terminal 100 is loaned together with the cart to a customer who purchases products at the store. The cart terminal 100 is a terminal device that executes a registration process for registering a product that the customer plans to purchase as a transaction product in response to an operation by the customer. As will be described later, the cart terminal 100 executes a process for recognizing a product to be registered as a transaction product from an image in which the product is reflected. In other words, the cart terminal 100 has a function as a product recognition device.

[0010] The payment machine 200 is installed in a store, and executes payment processing related to the calculation and settlement of the price of the transaction goods registered by the cart terminal 100. The payment machine 200 is operated by an operator during payment processing. The operator of the payment machine 200 is usually a customer. In some cases, the operator of the payment machine 200 is a store clerk. As the payment machine 200, for example, one that is used in an existing cart POS system can be used as is.

[0011] The attendant terminal 300 is an information processing terminal operated by a store clerk. The attendant terminal 300 is a terminal device for a user interface related to information processing to support the work of the store clerk regarding the transaction processed by the transaction processing system 1. The work of the store clerk is, for example, monitoring the implementation status of the transaction being processed and supporting the customer appropriately. As the basic hardware of the attendant terminal 300, various general-purpose information processing devices such as desktop type, notebook type, tablet type, and mobile type can be used. The attendant terminal 300 can be realized by installing an information processing program for realizing various functions as the attendant terminal 300 in a computer equipped with such basic hardware.

[0012] Management terminal 400 is an information terminal that serves as a user interface for an administrator who manages transaction processing system 1. As basic hardware for management terminal 400, various general-purpose information processing devices such as desktop type, notebook type, tablet type, and mobile type can be used. Management terminal 400 can be realized by installing an information processing program for realizing various functions of management terminal 400 in a computer equipped with such basic hardware.

[0013] The POS server 500 is an information processing device that executes management processing for managing merchandise buying and selling transactions that are processed by the registration processing in the cart terminal 100 and the accounting processing in the accounting device 200. As the basic hardware of the POS server 500, an appropriate information processing device such as a general-purpose server device can be used. The POS server 500 can be realized by installing an information processing program for realizing various functions of the POS server 500 in a computer equipped with such basic hardware.

[0014] The cart terminal 100 includes a processor 101, a main memory unit 102, an auxiliary memory unit 103, a touch panel 104, a sound unit 105, an interface unit 106, a wireless communication unit 107, and a transmission path 108. The processor 101 is capable of communicating with the main memory unit 102, the auxiliary memory unit 103, the touch panel 104, the sound unit 105, the interface unit 106, and the wireless communication unit 107 via the transmission path 108.

[0015] The processor 101, the main storage unit 102, and the auxiliary storage unit 103 are connected via a transmission line 108 to constitute a computer that performs information processing for controlling the cart terminal 100. The processor 101 corresponds to the central part of the computer. The processor 101 executes information processing for controlling each part to realize various functions of the cart terminal 100 in accordance with information processing programs such as an operating system and application programs.

[0016] The main memory unit 102 corresponds to the main memory portion of the computer. The main memory unit 102 includes a read-only memory area and a rewritable memory area. The main memory unit 102 stores a part of the information processing program in the read-only memory area. The main memory unit 102 may also store data required for the processor 101 to execute processes for controlling each part in the read-only memory area or the rewritable memory area. The main memory unit 102 uses the rewritable memory area as a work area for the processor 101.

[0017] The auxiliary storage unit 103 corresponds to the auxiliary storage portion of the computer. The auxiliary storage unit 103 may be, for example, an EEPROM (electric erasable programmable read-only memory), a HDD (hard disk drive), an SSD (solid state drive), or any other known storage device. The auxiliary storage unit 103 stores data used by the processor 101 in performing various processes and data generated by the processes in the processor 101. The auxiliary storage unit 103 may also store the information processing program. In this embodiment, the auxiliary storage unit 103 stores a cart terminal program PRA, which is one of the information processing programs. The cart terminal program PRA is an application program in which the procedure of the registration process is described. A part of the storage area of ​​the auxiliary storage unit 103 is used as an area for storing transaction data DAA. The transaction data DAA is data representing the contents of one transaction.

[0018] The touch panel 104 displays a screen for presenting information to the operator, and also inputs instructions via the touch operation on the screen by the operator. The sound unit 105 outputs sounds for various guidance and warnings. As the sound unit 105, various well-known sound devices such as a voice synthesis device and a buzzer can be used alone or in combination.

[0019] An external device such as a camera 199 is connected to the interface unit 106. The interface unit 106 interfaces for sending and receiving data to and from the connected external device. An existing USB (universal serial bus) controller or the like can be used as the interface unit 106. The camera 199 may repeat photographing at regular intervals, or may photograph in response to an instruction from the processor 101. A camera built into the cart terminal 100 may be used instead of the camera 199.

[0020] The wireless communication unit 107 executes communication processing for wirelessly performing data communication via the communication network 2. For example, an existing wireless communication device for wireless LAN can be used as the wireless communication unit 107. Note that instead of or in addition to the wireless communication unit 107, a communication unit connected to the communication network 2 by wire may be used. The transmission path 108 includes an address bus, a data bus, and control signal lines, and transmits data and control signals between the connected components.

[0021] The basic hardware of the cart terminal 100 is assumed to be, for example, the hardware of a tablet-type information processing device. The cart terminal 100 is generally transferred in a state in which the cart terminal program PRA is stored in the auxiliary storage unit 103. However, the cart terminal program PRA may be transferred separately from the hardware in a state in which the cart terminal program PRA is not stored in the auxiliary storage unit 103, or in a state in which a different version of the same type of application program is stored in the auxiliary storage unit 103. The cart terminal 100 may be configured by writing the cart terminal program PRA to the auxiliary storage unit 103 in response to an operation by an arbitrary worker. The cart terminal program PRA may be transferred by recording it on a removable recording medium such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory, or by communication via a network.

[0022] FIG. 2 is a perspective view showing an example of how the cart terminal 100 is attached to the cart. The cart 900 includes a caster portion 910 , a handle frame portion 920 , a basket portion 930 , a support portion 940 , and a battery case 950 . The caster unit 910 has four wheels 911 for smoothly moving the cart 900 on a floor surface. The wheels 911 are attached to a frame 912 in a state capable of rotating around an axis in the vertical direction.

[0023] The handle frame portion 920 includes a pair of vertical frames 921, 921 and a handle bar 922. The vertical frames 921, 921 are erected above the two wheels of the caster portion 910. The handle bar 922 connects the upper ends of the vertical frames 921, 921 together. The basket section 930 is provided horizontally from a midway portion of the handle frame section 920. The basket section 930 is shaped like a basket with an opening at the top to allow products to be placed therein, and functions as a container for storing products that a customer wishes to purchase. A camera 199 is attached to the upper end of the basket section 930 on the side closer to the handle bar 922 in a state in which the camera 199 can photograph the opening of the basket section 930 from the side. The photographing range of the camera 199 is determined so that the products placed inside the basket section 930 are captured through the opening of the basket section 930.

[0024] The support unit 940 includes a pole 941. The pole 941 is attached to one of the vertical frames 921 so that its tip is positioned higher than the handlebars 922. The cart terminal 100 described above is attached to the tip of the pole 941. As a result, the support unit 940 supports the cart terminal 100 in the state shown in FIG. The battery case 950 is attached between the vertical frames 921, 921 on the lower end side of the handle frame portion 920. The battery case 950 houses a battery used as an external power source for the cart terminal 100.

[0025] Next, the operation of the transaction processing system 1 configured as above will be described. Note that the contents of the various processes described below are merely examples, and it is possible to change the order of some of the processes, omit some of the processes, or add other processes as appropriate. For example, in the following description, in order to easily explain the characteristic operations of this embodiment, the description of some of the processes is omitted. For example, when some error occurs, a process to deal with the error may be performed. Also, for example, in various standby states, the system may wait for the occurrence of an event different from that described, and may transition to a process according to the event that has occurred.

[0026] A customer of a retail store, that is, a customer who wishes to purchase a product displayed and sold at the retail store, performs a predetermined operation for starting use on an unused cart terminal 100. When this operation is performed, the processor 101 starts a registration process based on the cart terminal program PRA.

[0027] 3 and 4 are flowcharts of the registration process. 3, the processor 101 generates new transaction data DAA for the target transaction. That is, the processor 101, for example, determines a new transaction code different from the transaction code serving as an identifier for identifying other transactions according to a predetermined rule, generates new transaction data DAA including this transaction code but not including product information related to the transaction product, and stores the new transaction data DAA in the auxiliary storage unit 103.

[0028] As ACT12, the processor 101 displays a registration screen on the touch panel 104. The registration screen is a screen for allowing the customer to recognize the registration status of the trading product. That is, the registration screen shows, for example, a list of registered trading products. However, when ACT12 is executed, the trading product has not yet been registered, so the registration screen is a screen for allowing the customer to recognize that the trading product has not yet been registered. The registration screen may include various display objects other than the list of trading products. An example of a display object included in the registration screen is a character string showing a provisional price for the registered trading product. An example of a display object included in the registration screen is a soft key for receiving various specifications by the operator. The contents of the registration screen may be appropriately determined, for example, by the creator of the cart terminal program PRA.

[0029] In ACT 13, the processor 101 checks whether it is time to capture an image. If the processor 101 cannot check the relevant event, it determines NO and proceeds to ACT 14. In ACT14, the processor 101 checks whether re-recognition, which will be described later, has been specified. If the processor 101 cannot confirm the relevant event, it determines NO and proceeds to ACT15. In ACT15, the processor 101 checks whether a payment has been requested. If the processor 101 cannot check the relevant event, it determines NO and returns to ACT13. Thus, in ACT13 to ACT15, the processor 101 waits for the timing to obtain data, for re-recognition to be specified, or for a transaction to be specified.

[0030] The acquisition timing is assumed to be, for example, immediately after proceeding from ACT12 to ACT13, and at regular time intervals thereafter. The acquisition timing may be determined as appropriate, for example, by the creator of the cart terminal program PRA. However, the acquisition timing should be determined so that the signal arrives at least once, and preferably multiple times, while the commodity to be put into the basket section 930 passes through the shooting range of the camera 199.

[0031] When the acquisition timing arrives, the processor 101 determines YES in ACT13 and proceeds to ACT16. In ACT16, processor 101 acquires image data output by camera 199. For example, if camera 199 repeats shooting at a constant cycle, processor 101 acquires image data output by camera 199 in response to the first shooting after the acquisition timing. For example, if camera 199 shoots in response to an instruction from processor 101, processor 101 instructs camera 199 to shoot, and acquires image data output by camera 199 in response to the instruction. Then, processor 101 stores the acquired image data in main storage unit 102 or auxiliary storage unit 103.

[0032] As ACT17, the processor 101 detects an area in which a product is reflected (hereinafter referred to as a product area) from the image represented by the acquired image data. The area to be detected can be determined by using, for example, deep learning using a convolutional neural network (CNN) or a well-known method using graph cuts, and the method is not limited.

[0033] In ACT 18, the processor 101 checks whether the detection of the product area was successful. If the processor 101 could not detect the product area in ACT 17, the result is NO, and the processor 101 proceeds to ACT 19. Thus, the processor 101 proceeds to ACT 19 when the product is not captured in the image captured by the camera 199. In ACT 19, the processor 101 checks whether the product is being recognized. If the processor 101 cannot check the event, it determines NO and returns to the standby state of ACT 13 to ACT 15.

[0034] A customer searches for a product to be traded in the store. The customer then takes out the product to be purchased from the sales floor and places it in the basket section 930. At this time, the product enters the shooting range of the camera 199 and is reflected in the image represented by the image data output by the camera 199. In other words, until the customer tries to put the product in the basket section 930, the processor 101 does not detect the product area in ACT17 even if the acquisition timing arrives, and product recognition has not started. Therefore, the processor 101 judges NO in both ACT18 and ACT19, and returns to the standby state of ACT13 to ACT15, and repeats this process. Then, when the product that the customer is trying to put in the basket section 930 is reflected in the image, the processor 101 detects the product area in ACT17. In this case, the processor 101 judges YES in ACT18 and proceeds to ACT20. As ACT20, the processor 101 performs a recognition start process. The recognition start process is a process for making it possible to confirm that a commodity that has entered the shooting range of the camera 199 is being recognized. For example, the processor 101 sets a recognition in progress flag to a state indicating that recognition is in progress.

[0035] The processor 101 executes a recognition process as ACT21. The recognition process is a process for recognizing a product reflected in a product area based on an image of the product area detected in ACT17. The recognition process may be either a process for product recognition based on the appearance of the product or a process for product recognition based on a product barcode displayed on the product, or may be both of them. For the process for product recognition based on the appearance of the product, it is assumed that deep learning using CNN, which is a method suitable for image recognition, is used. Alternatively, for the process for product recognition based on the appearance of the product, any method may be used, such as OCR (optical character recognition) or feature point matching. Thus, the processor 101 executes information processing based on the cart terminal program PRA, and the computer with the processor 101 as the central part functions as a recognition means. Furthermore, if a process for product recognition based on a product barcode displayed on the product is applied to the recognition process, another device such as a laser scanner that detects a pattern of a product barcode without being based on an image may be used instead of or in addition to the camera 199.

[0036] In ACT 22, the processor 101 checks whether the recognition in ACT 20 was successful. If the recognition was successful, the processor 101 judges that the result is YES and proceeds to ACT 23. In ACT23, the processor 101 registers the product recognized in ACT21 as a transacted product. That is, the processor 101 updates the transaction data DAA, for example, to include product information of the product recognized in ACT21 as the product information of the transacted product. The processor 101 includes at least a product code as a product identifier in the product information. The processor 101 may also include the product name and unit price in the product information. Here, the type of information that the processor 101 includes in the product information may be appropriately determined by, for example, the creator of the cart terminal program PRA. Thus, the processor 101 executes information processing based on the cart terminal program PRA, so that the computer having the processor 101 as its central part functions as a registration means.

[0037] In ACT24, the processor 101 updates the registration screen. For example, the processor 101 updates the registration screen displayed on the touch panel 104 to show an updated list that includes the trading product registered in ACT23. In ACT25, the processor 101 performs a recognition end process. The recognition end process is a process for making it possible to confirm that recognition has not started for the next product that enters the shooting range of the camera 199. For example, the processor 101 sets the recognition in progress flag to a state indicating that recognition is not in progress.

[0038] In ACT26, the processor 101 waits for the product that has been recognized as described above to complete passing through the shooting range of the camera 199. For example, while the processor 101 is successful in detecting the product area by a process similar to ACT17, the processor 101 repeats processes similar to ACT16 to ACT18 at each acquisition timing. Then, when the processor 101 is no longer able to detect the product area, it determines that the product has completed passing through, and returns to the standby state of ACT13 to ACT15.

[0039] If the processor 101 does not succeed in recognizing the product through the recognition process in ACT21, it judges NO in ACT22, skips ACT23 to ACT26, and returns to the standby state of ACT13 to ACT15. In other words, even if the processor 101 sees a product in an image, if the processor 101 is unable to recognize the product from the image, it returns to the standby state of ACT13 to ACT15 while keeping the product in recognition.

[0040] While one transaction commodity passes through the shooting range of camera 199, the acquisition timing may arrive multiple times. The state in which the commodity is reflected in the image represented by the image data acquired at these multiple acquisition timings changes. For this reason, in the recognition process in ACT21 while processor 101 repeats ACT16 and subsequent steps each time an acquisition timing arrives, even if the commodity was not recognized at the previous acquisition timings, the commodity may be recognized at the latest acquisition timing. In this case, processor 101 executes ACT23 to ACT26 and registers the recognized commodity as a transaction commodity.

[0041] However, in cases such as when a customer holds a commodity with most of the commodity covered by his / her hand, processor 101 may not be able to recognize the commodity in the recognition process in ACT21 at any of the capture timings while a single transaction commodity passes through the shooting range of camera 199. If processor 101 proceeds to ACT19 in response to the commodity area being unable to be detected in ACT17 because the commodity has left the shooting range of camera 199, processor 101 determines that the commodity is being recognized based on, for example, the recognition in progress flag indicating that the commodity is being recognized, and proceeds to ACT31 in FIG.

[0042] As ACT31, the processor 101 displays a recognition error screen on the touch panel 104. The recognition error screen is a screen for making the customer realize that the product previously placed in the basket section 930 could not be recognized. The recognition error screen is assumed to be, for example, a screen displaying a text message for making the customer realize that the product previously placed in the basket section 930 could not be recognized. The contents of the recognition error screen may be determined as appropriate by, for example, the creator of the cart terminal program PRA. In addition, at this time, the processor 101 may execute a suitable combination of an announcing operation other than the screen display together with the display of the recognition error screen. The announcing operation other than the screen display may be, for example, an alarm sound or a voice message output from the sound unit 105. The announcing operation other than the screen display may be, for example, the lighting or blinking of an alarm lamp not shown in FIG. 1. The announcing operation other than the screen display may be, for example, the generation of vibration by a vibrator not shown in FIG. 1.

[0043] As ACT32, the processor 101 updates the failure management information. The failure management information is information for managing the number of times product recognition has failed. The processor 101 updates the failure management information so that the cumulative number of failures can be determined based on the updated failure management information. Typically, the processor 101 increases the value represented by the variable as the failure management information by one, for example. However, the processor 101 may increase the value represented by the variable as the failure management information by a fixed number of two or more, for example. Alternatively, the processor 101 may decrease the value represented by the variable as the failure management information by a fixed number, for example. Alternatively, the processor 101 may use information other than variables as the failure management information and change the information in accordance with a predetermined rule.

[0044] As ACT33, the processor 101 checks whether a predetermined guidance execution condition is satisfied. The guidance execution condition is a judgment condition for whether or not the customer should be guided on the customer's behavior to improve the accuracy of product recognition. For example, if the variable is increased in ACT32, the guidance execution condition is assumed to be set as "the value represented by the variable as failure management information is equal to or greater than a specified value". However, the guidance execution condition may be appropriately set by the creator of the cart terminal program PRA, for example, as "the value represented by the variable as failure management information is greater than a default value", "the variable as failure management information represents a default value", or "the failure management information matches the default information". However, the guidance execution condition is set to be satisfied when the number of failures that can be determined from the failure management information is an arbitrarily set number of 2 or more. As an example, in ACT32, the processor 101 updates the variable VA, whose initial value is "0," to "VA+1." Then, in ACT33, the processor 101 checks whether "VA≧3" is satisfied.

[0045] If the processor 101 cannot confirm that the guidance execution condition is satisfied, the processor 101 judges NO and proceeds to ACT 34. In other words, when the processor 101 proceeds to ACT 32 for the first time, the processor 101 judges NO and proceeds to ACT 34. In ACT34, the processor 101 causes the touch panel 104 to display a registration screen. That is, the processor 101 causes the touch panel 104 to once again display a registration screen showing a list of trading products that have been registered at this point in time. Then, the processor 101 then returns to the standby state of ACT13 to ACT15 in FIG. 3.

[0046] When the customer learns from the recognition error screen that recognition has failed, he or she temporarily removes the relevant item from the basket section 930, and after the registration screen is displayed in ACT34 or ACT39 in Fig. 4, places the item back into the basket section 930. At this time, the processor 101 attempts to recognize the item again in ACT13, ACT16 to ACT18, and ACT21, and if recognition is successful, registers the item as a transaction item in ACT23. At this time, the appearance of the item in the image obtained by the camera 199 is different from the previous time, so there is a possibility that it will be recognized correctly.

[0047] Now, the processor 101 may recognize a product different from the product shown in the image in the recognition process in ACT21 in Fig. 3. However, even in such a case, the processor 101 determines YES in ACT22 and registers the product as a transaction product. If the customer confirms based on the list displayed on the registration screen that a product different from the one he or she placed in the basket section 930 has been registered as the product to be traded, the customer may specify a request for reconfirmation of the product that was erroneously registered, for example, by a predetermined operation on the touch panel 104. When this specification is made, the processor 101 judges YES in ACT 14 in FIG. 3 and proceeds to ACT 35 in FIG. 4.

[0048] In ACT35, the processor 101 excludes the product designated for re-recognition from the trading products. For example, the processor 101 deletes the product information related to the product from the trading data DAA. After that, processor 101 executes ACT32 and subsequent steps in the same manner as described above. That is, processor 101 counts the number of failures as recognition failures when the product cannot be recognized while it passes through the shooting range of camera 199 and when the customer judges that the recognition result is incorrect.

[0049] If the customer specifies re-recognition, the customer removes the target commodity from the basket section 930 and then places it back into the basket section 930. At this time, the processor 101 attempts to recognize the commodity again in ACT13, ACT16 to ACT18, and ACT21, and if the recognition is successful, registers the commodity as a transaction commodity in ACT23. At this time, the commodity is reflected in the image obtained by the camera 199 in a different way from the previous time, so there is a possibility that the commodity can be correctly recognized. If the correct transaction commodity is registered by such processing, the above-mentioned erroneous recognition result is corrected by the new recognition result. Thus, the processor 101 executes information processing based on the cart terminal program PRA, and the computer with the processor 101 as its central part functions as a correction means.

[0050] When the cases where the product cannot be recognized while passing through the shooting range of camera 199, or the customer judges that the recognition result is incorrect, are repeated multiple times until the guidance execution condition is met, processor 101 judges YES in ACT 33 in FIG. 4 and proceeds to ACT 37.

[0051] As ACT37, the processor 101 executes a guidance process. The guidance process is a process for informing the customer of the behavior of the customer for improving the accuracy of product recognition. As the guidance process, for example, the processor 101 displays a guidance screen that is predetermined for the corresponding guidance on the touch panel 104.

[0052] FIG. 5 is a diagram showing an example of the guide screen SCA. The guide screen SCA displays an image IMA, a character string CSA, and a button BUA. The image IMA is a photograph or illustration to allow the customer to recognize the camera 199 used for product recognition. The character string CSA represents a text message to guide the customer on his / her actions. The button BUA is a soft key for the customer to declare that he / she has understood the guidance provided by the guidance screen SCA.

[0053] As the guidance process, the processor 101 may output a voice message to the sound unit 105 instead of or in addition to displaying the guidance screen. The processor 101 waits for a predetermined end condition to be satisfied and then proceeds to ACT35. The processor 101 determines that the end condition is satisfied when the customer designates completion of confirmation by a predetermined operation such as tapping the button BUA. Alternatively, the processor 101 determines that the end condition is satisfied when the duration of the display of the guidance screen reaches a predetermined time. The contents of the guidance screen or the type of processing to be performed in the guidance process may be determined as appropriate by, for example, the creator of the cart terminal program PRA. Thus, the processor 101 executes information processing based on the cart terminal program PRA, and the computer with the processor 101 as its central part functions as a guide means.

[0054] In ACT38, the processor 101 initializes the failure management information. That is, for example, if the processor 101 updates the variable VA, whose initial value is "0", to "VA+1" in ACT32, the processor 101 updates the value represented by the variable VA to "0". In ACT39, the processor 101 causes the touch panel 104 to display a registration screen. That is, the processor 101 causes the touch panel 104 to once again display a registration screen showing a list of trade products that have been registered at this point in time. Then, the processor 101 then returns to the standby state of ACT13 to ACT15 in FIG. 3.

[0055] Once the purchaser has finished registering the purchased items, the purchaser performs a predetermined operation to specify the payment, for example, on touch panel 104. In response, processor 101 determines YES in ACT15 in FIG. 3 and proceeds to payment processing. The payment processing may be the same as that performed in existing cart POS systems, for example, and is not shown in the figures. For example, processor 101 performs processing to transfer transaction data to payment device 200 as payment processing. Alternatively, if a payment terminal (not shown) is attached to cart 900 and the payment terminal is connected to cart terminal 100, the payment processing may be processing to perform credit card payment, code payment, or the like using this payment terminal.

[0056] Thus, in transaction processing system 1, if recognition failures are repeated, cart terminal 100 performs guidance processing to guide the customer on actions to be taken by the customer to improve the accuracy of product recognition. If a customer, who has been recognized as taking a customer action to improve the accuracy of product recognition through this guidance processing, takes the relevant action, it is possible to improve the success rate of recognition of the item from an image in which the item being moved in the customer's hand is captured.

[0057] Furthermore, the cart terminal 100 counts the number of failures as recognition failures when the product cannot be recognized while it passes through the shooting range of the camera 199 and when the customer judges the recognition result to be incorrect. This allows the correct evaluation of a failure as to whether or not to execute guidance processing even when the processor 101 has registered the product as successful despite an incorrect recognition.

[0058] This embodiment can be modified in various ways as follows. The processor 101 may execute the guidance process in response to a repetition of either the case where the product cannot be recognized while passing through the shooting range of the camera 199, or the case where the customer determines that the recognition result is incorrect.

[0059] The processor 101 may perform a first guidance action adapted to the situation in response to repeated instances where the product cannot be recognized while passing through the shooting range of the camera 199, and may perform a second guidance action different from the first guidance action adapted to the situation in response to repeated instances where the customer determines that the recognition result is incorrect.

[0060] It may also be used in combination with another merchandise registration device or transaction processing device, and realized as a merchandise recognition device whose main function is to recognize merchandise.

[0061] The results of the recognition of an item may be used for any purpose other than the registration of a transaction item for transaction processing.

[0062] The present invention may also be realized as a device for recognizing items other than merchandise.

[0063] In the above embodiment, a part of the processing executed by processor 101 may be executed by POS server 500 or any other information processing device or devices. For example, cart terminal 100 may perform only user interface operations, and various information processing for identifying the transaction commodity may be executed by a computer included in POS server 500. In other words, each of the recognition means, the guidance means, the correction means, and the registration means may be included in any information processing device such as POS server 500 other than cart terminal 100 included in transaction processing system 1, or may be appropriately distributed among multiple information processing devices included in transaction processing system 1.

[0064] Each function realized by the processor 101 through information processing can be realized in part or in whole by hardware that executes information processing not based on a program, such as a logic circuit, etc. Also, each of the above functions can be realized by combining the above hardware, such as the logic circuit, with software control.

[0065] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0066] 1...transaction processing system, 2...communication network, 100...cart terminal, 101...processor, 102...main memory unit, 103...auxiliary memory unit, 104...touch panel, 105...sound unit, 106...interface unit, 107...wireless communication unit, 108...transmission path, 199...camera, 200...accounting machine, 300...attendant terminal, 400...management terminal, 500...POS server, 900...cart, 910...caster section, 920...handle frame section, 930...basket section, 940...support section, 950...battery case.

Claims

1. A recognition means for recognizing an object based on an image in which the object is captured; a guidance means for informing a user of an action to be taken by the user to improve the accuracy of recognition by the recognition means in response to repeated failure of the recognition means to recognize an article; An article recognition system comprising:

2. The recognition means recognizes an object based on an external appearance characteristic of the object reflected in the image. The item recognition system according to claim 1 .

3. The recognition means recognizes an object captured in the image based on information represented by a barcode displayed on the object. The article recognition system according to claim 1 or 2.

4. the recognition means starts a process for recognizing an object in response to the object being captured in the image, and ends the process in response to the object being successfully recognized or the object being no longer captured in the image; The guidance means determines that the recognition of the object by the recognition means has failed when the process for recognizing the object is ended in response to the object no longer being reflected in the image. The item recognition system according to claim 1 .

5. a correction means for correcting the recognition result by the recognition means in response to an instruction from an operator; Further equipped with The guidance means determines that the correction by the correction means has been made as a failure of the recognition of the article by the recognition means. The article recognition system according to claim 1 or 4.

6. A computer provided in the item recognition system, A recognition means for recognizing an object based on an image in which the object is captured; a guidance means for informing a user of an action to be taken by the user to improve the accuracy of recognition by the recognition means in response to repeated failure of the recognition means to recognize an article; An information processing program that enables the system to function as such.

7. A recognition means for recognizing an object based on an image in which the object is captured; a registration means for registering the item recognized by the recognition means as a commodity to be traded; a guidance means for informing a user of an action to be taken by the user to improve the accuracy of recognition by the recognition means in response to repeated failure of the recognition means to recognize an article; A transaction processing system comprising:

Images