Commodity registration terminal, commodity registration method, and program

Abstract

To provide a technique capable of canceling an erroneously recognized commodity.SOLUTION: A commodity registration terminal includes object detection means, commodity specification means, cancellation processing means, and output means. The object detection means detects an object based on an image captured by an imaging unit. The commodity identification means identifies a commodity based on an image including the object detected by the object detection means. The cancellation processing means acquires a cancellation instruction to cancel the commodity specified by the commodity specification means based on a user's operation. The output means outputs a notification indicating cancellation of the commodity based on acquisition of the cancellation instruction.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] An embodiment of the present invention relates to a product registration terminal, a product registration method, and a program. [Background technology]

[0002] Among cart POS (Point Of Sales) systems in which a shopping cart is provided with an information terminal equipped with a touch panel, scanner, etc., there are cart POS systems that utilize AI (artificial intelligence) and are called frictionless carts, AI carts, smart carts, intelligent carts, etc. In these cart POS systems, when a user simply places a product in the cart, the AI ​​automatically identifies and specifies the type of product placed in the cart through image recognition of the product package, and automatically registers the product (for example, see Patent Document 1).

[0003] In cart POS systems, if there are similar products that look similar, or if the product cannot be correctly identified due to the way it is inserted, the product may be incorrectly recognized and registered. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-87578 Summary of the Invention [Problem to be solved by the invention]

[0005] Therefore, there is a need for technology to cancel misidentified products.

[0006] The problem to be solved by the embodiments of the present invention is to provide a technique that makes it possible to cancel a product that has been erroneously recognized. [Means for solving the problem]

[0007] In one embodiment, the product registration terminal includes an object detection means, a product identification means, a cancellation processing means, and an output means. The object detection means detects an object based on an image captured by an imaging unit. The product identification means identifies a product based on an image including the object detected by the object detection means. The cancellation processing means acquires a cancellation instruction to cancel the product identified by the product identification means based on a user's operation. The output means outputs a notification indicating the cancellation of the product based on the acquisition of the cancellation instruction. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a cart POS system according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing a main data structure of the information terminal table of the cart controller according to the embodiment. [Diagram 3] FIG. 3 is a block diagram showing a main circuit configuration of the information terminal according to the embodiment. [Figure 4] FIG. 4 is a schematic diagram showing the mounting position of an information terminal on a shopping cart according to the embodiment. [Diagram 5] FIG. 5 is a flowchart showing a procedure of main information processing executed by a processor of an information processing device in an information terminal according to the terminal program. [Figure 6] FIG. 6 is a diagram showing a display example of the information terminal according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating a display example of the store terminal according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] (Embodiment) Hereinafter, the embodiments will be described with reference to the drawings. The scale of each part in each drawing used in the following description of the embodiments may be changed as appropriate. In addition, the configuration of each drawing used in the following description of the embodiments may be omitted for the sake of explanation. The following embodiments are applied to a cart POS system.

[0010] Fig. 1 is an overall configuration diagram of a cart POS system 1 according to an embodiment. The cart POS system 1 includes a store server 2, a cart controller 3, an access point (indicated as AP in Fig. 1) 4, a payment machine 5, and a store terminal 9. The store server 2, the cart controller 3, the access point 4, the payment machine 5, and the store terminal 9 are connected to a network 6 such as a LAN (Local Area Network) or a wireless LAN. The cart POS system 1 includes a plurality of information terminals 7 connected to the network 6 via the access point 4.

[0011] The information terminal 7 is a device that enables a customer, who is a purchaser, to input data related to the registration of purchased items by himself / herself. The information terminal 7 is provided in a shopping cart 8 that the customer uses when shopping in the store. Hereinafter, the shopping cart 8 will be simply referred to as a cart 8. Each of the multiple carts 8 is provided with an information terminal 7. The cart 8 is an example of a carrier that transports purchased items by a customer using the cart 8.

[0012] The information terminal 7 is provided on the cart 8. The information terminal 7 is an information processing terminal that functions as a user interface for operations related to registering purchased items. The information terminal 7 is equipped with a wireless unit. The information terminal 7 performs wireless communication with the access point 4. The access point 4 relays communication between the information terminal 7 and each device connected to the network 6, i.e., the store server 2, the cart controller 3, and the payment device 5. Although only one access point 4 is shown in FIG. 1, there may be two or more access points 4 depending on the size of the store, etc.

[0013] The accounting machine 5 is a device that enables a store clerk or a customer to make a payment for purchased items. The accounting machine 5 is a well-known self-service POS terminal. The accounting machine 5 may be a well-known face-to-face POS terminal. There is no particular limit to the number of accounting machines 5.

[0014] The store terminal 9 is a device used by a store clerk. The store terminal 9 includes an input device capable of inputting information. The store terminal 9 includes a display device capable of displaying images. The store terminal 9 includes a communication interface that communicatively connects the store terminal 9 to other devices via the network 6. The store terminal 9 is a tablet terminal, a smartphone, a PC (Personal Computer), etc., but may also be a dedicated device. A store clerk is an example of a user of the store terminal 9. There is no particular limit to the number of store terminals 9.

[0015] The store server 2 supports store operations. To support the operations, the store server 2 manages a product database (DB) 21, a member database (DB) 22, and the like.

[0016] The product database 21 stores a product record that describes product data for each product sold in the store. The product record includes a product code, price, product name, etc. The product code is an identification code for the product. The sales database stores sales records that describe sales data for each item, such as product, department, time period, date, week, month, etc. The sales data includes sales units, sales amount, etc.

[0017] In the present embodiment, the product database 21 stores pre-trained learning data 211 for recognizing products stored in the product database 21 by AI. The learning data 211 is a collection of learning data for each product sold in a store, and the learning data for each product is incorporated into the learning data 211 in association with, for example, a product code. The learning data 211 can update the learning data for each product in accordance with an update of the product database 21 in response to an addition or replacement of a product sold in the store. The learning data for each product may be created by a provider of the product and provided to the store, or may be created independently by the store. In addition to the above learning data, the learning data 211 may learn data other than products, such as data of a person's hand.

[0018] Member database 22 stores a member record that describes member data for each member. The member data includes a member ID, name, gender, email address, etc. The member ID is a code such as a unique member number that is set for each member to individually identify each member. Members own a recording medium on which the member ID is recorded. The recording medium may be, for example, a magnetic card, a contact type IC (Integrated Circuit) card, a contactless IC card, a smartphone, etc.

[0019] Although not shown in the figure, the store server 2 may further manage a member payment database. The member payment database stores a member payment record that describes data required for payment for each member. In addition to the member ID, the member payment record includes an electronic money identification code and an electronic money user ID as data required for payment. The cart POS system 1 according to this embodiment limits the payment method to one type of electronic money when the member does not use the accounting machine 5. The electronic money identification code is an identification code of electronic money that is registered in advance as the electronic money to be used for payment by the member identified by the member ID. The electronic money user ID is an identification code that the electronic money center EMC uses to identify the user of the electronic money, and is assigned by the electronic money center EMC. In order to enable payment by electronic money, the member registers the electronic money identification code and electronic money ID in advance, for example, when registering as a member. If the electronic money has not been registered in advance, NULL or 0000 is stored in the electronic money identification code and electronic money ID of the member payment record. Furthermore, when electronic money is allowed to be used in this way, the store server 2 has a communication server function, or the cart POS system 1 includes a communication server. The store server 2 or the communication server supports data communication processing between the accounting machine 5 or the cart controller 3 and an electronic money center that supports payments with electronic money, via an external communication network such as the Internet or a network dedicated to electronic money.

[0020] The cart controller 3 cooperates with the information terminal 7 to support the information terminal 7 so that the information terminal 7 appears to function as a POS terminal. The cart controller 3 has an information terminal table 31 corresponding to each information terminal 7. FIG. 2 is a schematic diagram showing an example of the information terminal table 31. As shown in FIG. 2, the information terminal table 31 has a terminal ID area 311, a member ID area 312, one or more purchased product data areas 313, and a subtotal area 314. The terminal ID area 311 stores a terminal ID, which is a code such as a terminal number for uniquely identifying the information terminal 7 corresponding to the information terminal table 31. The member ID area 312 stores a member ID. The purchased product data area 313 stores purchased product data for the product every time a customer registers a product to be purchased on the information terminal 7. The purchased product data includes a product code, a unit price, a sales point (number), a product name, a sales amount, a tax amount, and the like of the purchased product purchased by the customer. The subtotal area 314 is amount data showing the total amount of the sales price and tax for all the purchased items purchased by the customer, calculated based on the purchased item data stored in the purchased item data area 313. This can be updated every time the customer performs the registration operation for the item he or she wishes to purchase.

[0021] FIG. 3 is a block diagram showing the main circuit configuration of the information terminal 7. FIG. 4 is a schematic diagram showing the mounting position of the information terminal 7 on the cart 8. The information terminal 7 includes an information processing device 71, a reader 72, and a camera 73. The information processing device 71 may be a dedicated device or a general-purpose device such as a tablet computer. The information processing device 71 includes a processor 711, a main memory 712, an auxiliary storage device 713, a wireless unit 714, a touch panel 715, a speaker 716, and a system transmission path 717. The system transmission path 717 includes an address bus, a data bus, a control signal line, and the like. The information processing device 71 connects the processor 711, the main memory 712, the auxiliary storage device 713, the wireless unit 714, the touch panel 715, and the speaker 716 to the system transmission path 717. The reader 72 and the camera 73 are connected to the system transmission path 717 via an interface not shown. In the information processing device 71, a computer is configured with a processor 711, a main memory 712, an auxiliary storage device 713, and a system transmission line 717 that connects these together. The information terminal 7 is an example of a product registration terminal.

[0022] The processor 711 corresponds to the central part of the computer. The processor 711 controls each part to realize various functions as the information processing device 71 according to an operating system or an application program. The processor 711 is, for example, a CPU (Central Processing Unit), but is not limited thereto. The processor 711 may be a multi-core / multi-threaded one and can execute multiple processes in parallel. The processor 711 may include an AI chip specialized for AI. Furthermore, the processor 711 may be realized in various other forms including integrated circuits such as a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (field-programmable gate array), and a DSP (Digital Signal processor). The processor 711 is an example of a processing circuit.

[0023] The main memory 712 corresponds to the main storage portion of the computer. The main memory 712 includes a nonvolatile memory area and a volatile memory area. The main memory 712 stores an operating system or an application program in the nonvolatile memory area. The main memory 712 may store data required for the processor 711 to execute processes for controlling each part in the nonvolatile or volatile memory area. The main memory 712 uses the volatile memory area as a work area where data is appropriately rewritten by the processor 711. The nonvolatile memory area is, for example, a ROM (Read Only Memory). The volatile memory area is, for example, a RAM (Random Access Memory).

[0024] The auxiliary storage device 713 corresponds to the auxiliary storage part of the computer. For example, an EEPROM (registered trademark) (Electric Erasable Programmable Read-Only Memory), a HDD (Hard Disc Drive), or an SSD (Solid State Drive) can be the auxiliary storage device 713. The auxiliary storage device 713 stores data used by the processor 711 in performing various processes, or data created by the processes in the processor 711. The auxiliary storage device 713 may store the above application program. In this embodiment, as the application program, a terminal program 7131 for causing the information processing device 71, the reader 72, and the camera 73 to function as the information terminal 7 according to the embodiment is stored. The terminal program 7131 may be stored in the main memory 712. In addition, the auxiliary storage device 713 stores learning data 7132 for commodity recognition by AI for each commodity sold in the store stored in the commodity database 21 of the store server 2. This learning data 7132 may be a copy of the learning data 211 included in the product database 21 of the store server 2. The timing of copying this learning data 211 as the learning data 7132 may be a specific timing such as each time the information processing device 71 is started up, or may be an arbitrary timing such as when the learning data 211 is distributed from the store server 2 to each information processing device 71 when the learning data 211 is updated.

[0025] The wireless unit 714 wirelessly communicates data with the access point 4 in accordance with a wireless communication protocol.

[0026] The touch panel 715 is a device that combines an input device and a display device for the information terminal 7. The touch panel 715 displays an image based on image data output from the processor 711, detects a touch position on the displayed image, and outputs the touch position information to the processor 711. As shown in Fig. 4, the information processing device 71 is attached to the cart 8 at a position and orientation such that a customer who is moving while pushing the cart 8 can see the display of the touch panel 715 and perform touch operations. The touch panel 715 is an example of a display unit.

[0027] The speaker 716 reproduces sounds based on the sound data output from the processor 711. The reproduced sounds include confirmation sounds, alert warning sounds, and the like.

[0028] The reader 72 reads data recorded on the recording medium and outputs the read data to the processor 711. The reader 72 is a magnetic card reader if the recording medium is a magnetic card, and is an IC card reader if the recording medium is a contact IC card. In the case of a recording medium that uses RFID (Radio Frequency Identification) such as a non-contact IC card or a smartphone, an RFID reader is used as the reader 72. The position at which the reader 72 is attached to the cart 8 is not particularly specified. For example, it can be attached next to the information processing device 71.

[0029] As shown in FIG. 4, the camera 73 is provided on the cart 8 so as to photograph the basket receiving section 81 of the cart 8 from above. The camera 73 is for monitoring the insertion or return of products into or from the basket receiving section 81 of the cart 8 or the basket placed in the basket receiving section 81. An LED (Light Emitting Diode) lamp 731 for illumination can be installed on the camera 73. The operating states of the camera 73 and the LED lamp 731 can be controlled by the processor 711 via the system transmission path 717. Although only one camera 73 is shown in FIG. 3 and FIG. 4, two or more cameras may be provided. For example, four cameras 73 may be arranged on the four sides or four corners of the basket receiving section 81 of the cart 8. Depending on the type of camera 73, sufficient brightness may be ensured, so operation may be performed without installing the LED lamp 731. The camera 73 is an example of an imaging section.

[0030] 3, the information terminal 7 may further include a patrol lamp 74 that is connected to the system transmission line 717 via an interface (not shown) and whose operation state is controlled by the processor 711. The patrol lamp 74 is a lamp for issuing a warning to a customer using the cart 8 and nearby store staff.

[0031] FIG. 5 is a flowchart showing the procedure of main information processing executed by the processor 711 of the information processing device 71 in the information terminal 7 according to the terminal program 7131.

[0032] Below, using this flowchart, the main operations of the cart POS system 1 when a customer uses the information terminal 7 of the cart 8 to perform the process from registering purchased items to paying for them by himself / herself will be explained. Note that the operations explained below are just an example. As long as similar results are obtained, the procedure is not particularly limited. Also, it is assumed here that the reader 72 has read the data recorded on the recording medium and that login has already been performed. In other words, the member ID area 312 of the information terminal table 31 of the cart controller 3 corresponding to the information terminal 7 has already stored the member ID read from the customer's recording medium.

[0033] In ACT11, the processor 711 of the information processing device 71 turns on the LED lamp 731 of the camera 73 and causes the camera 73 to capture an image. Image data of the captured image is stored in the main memory 712. Note that an area for storing multiple frames of this image data is secured in the main memory 712, and when the processor 711 stores new image data in that area, it erases the oldest image data and stores the new image data in the secured area. After that, the processor 711 proceeds to ACT12.

[0034] In ACT12, the processor 711 compares the object and detection position of the latest image data stored in the main memory 712 with the object and detection position of the old image data to determine whether or not there is a change. The processing of ACT12 may be processing by an object detection means. The processor 711 detects an object based on the fact that a product is put into the cart 8. Specifically, when a product is put into the basket receiving section 81 of the cart 8 or a basket placed in the basket receiving section 81, or when a product is taken out of the basket receiving section 81 or a basket, that is, when a product is returned, the product being moved for the putting in or returning is photographed by the camera 73. Therefore, when a product is put in or returned, a difference occurs between the object and detection position of the latest image data and the object and detection position of the old image data, so that the processor 711 can determine that there is a change. If it is determined that there is no change, the processor 711 determines NO in ACT12 and proceeds to the above ACT11. Moreover, if it is determined that there has been a change, the processor 711 determines YES in ACT12 and proceeds to ACT13.

[0035] In ACT13, the processor 711 detects an object in the image based on the latest image data that is determined to have changed. The processing in ACT13 may be processing by an object detection means. Specifically, the processor 711 performs object detection on the object in the latest image, sets a rectangular area called a bounding box, and detects the object in the image. The detected object and the detection position of the object in the image data are stored in the main memory 712 in association with the image data stored in the main memory 712. After that, the processor 711 proceeds to ACT14.

[0036] The processor 711 identifies a product based on an image including the detected object as ACT14. The process of ACT14 may be a process by a product identification means. Specifically, the processor 711 performs an identification process to identify a product moving in the image. The identification process indicates identifying an object included in the image as a product. The identification process includes identifying which product the object is. The identification process includes, for example, identifying an object included in the image by information that can identify the product, such as a product name. The processor 711 performs image recognition on the object in the bounding box detected in ACT13 and stored in the main memory 712 based on the learning data 7132 stored in the auxiliary storage device 713, and identifies which product the object is. After that, the processor 711 proceeds to ACT15. The processor 711 may perform product identification process by known techniques, without being limited to the above-mentioned processes.

[0037] In ACT15, the processor 711 determines whether or not the product has been identified. For example, if only a small portion of the product is shown in the image, it is not possible to identify what the product is. If it is determined that the product has not been identified, the processor 711 determines NO in ACT15 and proceeds to ACT11 above. If it is determined that the product has been identified, the processor 711 determines YES in ACT15 and proceeds to ACT16.

[0038] In addition, when the processor 711 determines that the product could not be identified, the processor 711 may display an error notification on the touch panel 715. The error notification includes, for example, a text message indicating that the product could not be identified. Specifically, the error notification may include messages such as "The product could not be identified," "Please insert the product again," and "Please let me read the product code." The processor 711 may superimpose the error notification on the registered product screen. In this case, the customer performs an operation such as inserting the product again. The processor 711 performs processing according to the customer's operation. For example, when the customer inserts the product again, the processor 711 proceeds to ACT11 and takes an image. When the customer performs an operation to read the product code via the camera 73 or a scanner (not shown), the processor 711 may identify the product based on the product code.

[0039] In ACT16, the processor 711 determines whether a product has been put into the basket receiving section 81 of the cart 8 or into the basket placed in the basket receiving section 81. For example, the processor 711 can determine whether a product has been put into or taken out based on the movement direction of the same product shown in the latest image and the past image. Alternatively, it is possible to detect the putting in / taking out of a product by placing a weight scale (not shown) on the bottom surface of the basket receiving section 81 of the shopping cart 8 and determining whether the weight has increased or decreased. If the processor 711 determines that a product has not been put in, that is, that a product has been taken out, it determines NO in ACT16 and proceeds to ACT23. If the processor 711 determines that a product has been put in, it determines YES in ACT16 and proceeds to ACT17.

[0040] In ACT17, the processor 711 transmits a product input notification to the cart controller 3 via the access point 4 by the wireless unit 714. The product input notification may include a terminal ID assigned to the information terminal 7 including the information processing device 71, and a product code indicating the product identified in ACT15. After that, the processor 711 proceeds to ACT18.

[0041] The cart controller 3 that has received the product input notification registers the input product. That is, the cart controller 3 queries the store server 2 for product information using the product code included in the product input notification, and acquires product information stored in the product database 21 managed by the store server 2. The cart controller 3 then additionally stores the acquired product information as new purchased product data in the purchased product data area 313 of the information terminal table 31 having the terminal ID included in the product input notification. The cart controller 3 then creates a registered product screen for notifying the registered product based on the purchased product data stored in the purchased product data area 313, and transmits the registered product screen to the sender of the product input notification.

[0042] In ACT18, the processor 711 receives, by the wireless unit 714, the registered product screen transmitted from the cart controller 3 via the access point 4. After that, the processor 711 proceeds to ACT19.

[0043] In this way, when an item is added to the basket receiving section 81 of the cart 8 or a basket placed in the basket receiving section 81, the information terminal 7 identifies the item through image recognition and sends an item addition notification including an item code indicating the identified item to the cart controller 3, thereby causing the cart controller 3 to register the item as a purchased item.

[0044] In ACT19, the processor 711 presents the received registered product screen to the customer using the cart 8 via the touch panel 715. The registered product screen includes a cancel error recognition button. The cancel error recognition button is an operation key for inputting a cancel instruction to cancel the registered product when the registered product is different from the inserted product. After that, the processor 711 proceeds to the above ACT20.

[0045] The processor 711 determines whether or not a cancel operation has been performed by the user of the cart 8 in ACT20. The process of ACT20 may be a process by a cancel processing means. For example, the processor 711 determines whether or not a cancel instruction to cancel the product identified in ACT15 has been acquired based on the input operation of the user's cancel misrecognition button. When the user inputs the cancel misrecognition button, it is determined that the user has performed a cancel operation. The user may select the product to be canceled from a registered product list included in the registered product screen. The product selected as the product to be canceled by the user is also called a product to be canceled. The selection operation of the product to be canceled by the user includes an operation of selecting the product to be canceled from the registered product list by touch input or the like. In this example, the product identified in ACT15 is an example of a product to be canceled. The user may select the product identified in ACT15 from the registered product list. In this case, the processor 711 determines YES in ACT20 and proceeds to ACT21. Moreover, if it is determined that the user has not performed a cancel operation, the processor 711 determines NO in ACT20 and proceeds to ACT11.

[0046] The processor 711 cancels the product to be canceled as ACT21. The processing of ACT21 may be processing by a cancellation processing means. Specifically, the processor 711 transmits an instruction to cancel the product to be canceled to the cart controller 3 based on the cancellation instruction. In this example, the product to be canceled is the product identified in ACT15. The cart controller 3 deletes the data of the product to be canceled that is registered as a purchased product in the purchased product data area 313. The cart controller 3 updates the purchased product data area 313.

[0047] The user of the cart 8 may select and cancel a product displayed on the registered product screen as a purchased product. For example, the user of the cart 8 selects the Cancel Misrecognition button. The user of the cart 8 selects the cancellation target product whose registration should be cancelled from the registered product list included in the registered product screen. The user's operation of the Cancel Misrecognition button and the selection operation of the cancellation target product may be performed in any order. The processor 711 acquires a cancellation instruction and information on the cancellation target product. The cancellation instruction may include information on the cancellation target product. The information on the cancellation target product is, for example, a product code. The processor 711 transmits the cancellation instruction and information on the cancellation target product to the cart controller 3. The cart controller 3 deletes the purchased product data of the product code included in the cancellation instruction, which is stored in the purchased product data area 313 of the information terminal table 31 having the terminal ID included in the cancellation instruction, from the purchased product data area 313. The cart controller 3 updates the purchased product data area 313.

[0048] The processor 711 may display a notification on the registered item screen indicating that a cancellation operation has been performed by the user. The processor 711 may confirm the cancellation operation based on the fact that a store clerk who has received a cancellation notification, which will be described later, has approved the cancellation operation or performed a correction operation. Approval of the cancellation operation includes an operation in which the store clerk inputs an approval instruction to approve the cancellation operation via the store terminal 9 or the like. Correction operation includes an operation in which the store clerk inputs a correction instruction to correct the cancellation operation via the store terminal 9 or the like. In this case, the cart controller 3 may update the purchased item data area 313 based on the fact that the store clerk has approved the cancellation operation or performed a correction operation.

[0049] The processor 711 outputs a cancellation notification to the store terminal 9 as ACT22. The process of ACT22 may be a process by an output means. The cancellation notification is a notification indicating the cancellation of the product. The cancellation notification includes, for example, a message indicating that the user of the cart 8 has performed a cancellation operation of the registered product. The processor 711 may output the cancellation notification to the store terminal 9 based on the acquisition of a cancellation instruction. The processor 711 may output the cancellation notification to the store terminal 9 based on the cart controller 3 updating the purchased product data area 313. The processor 711 may output the cancellation notification to a mobile terminal such as a smartphone or tablet used by a store clerk. The processor 711 may turn on the LED lamp 731 of the camera 73 as a notification indicating the cancellation of the product. For example, the processor 711 may change the color or blinking pattern of the LED lamp 731 to indicate that the registered product has been cancelled. The processor 711 may play a specific notification sound through the speaker 716. Also, in the case where the cart 8 is equipped with a patrol lamp 74, the patrol lamp 74 may be turned on instead of or in addition to the patrol lamp 74.

[0050] The store clerk who received the cancellation notice judges whether the cancellation operation was performed appropriately. For example, the processor 711 may output information on the product to be canceled and an image including the product to be canceled to the store terminal 9. The information on the product to be canceled is, for example, a product code. The image including the product to be canceled is image data used for object detection of the product to be canceled in ACT12. The processor 711 may obtain the image data from the main memory 712 and output it to the store terminal 9. The store terminal 9 displays the image data and an image including the information on the product to be canceled on the display device based on the operation of the store clerk. The store clerk checks the image data against the information on the product to be canceled and confirms that the product to be canceled is a product that has been erroneously recognized. This allows the store clerk to judge whether the cancellation operation by the user of the cart 8 is appropriate. The confirmation work by the store clerk may be performed at any timing, as long as it is performed before the checkout process by the user of the cart 8 is completed. The processor 711 may display the image data and an image including the information on the product to be canceled on the touch panel 715 based on the operation of the store clerk. Furthermore, the processor 711 may output the image data and information on the product to be canceled to a mobile terminal such as a smartphone or tablet used by a store clerk. The mobile terminal may display an image including the image data and information on the product to be canceled based on an operation by the store clerk.

[0051] Furthermore, if the processor 711 determines in ACT 16 that the product has been removed, in ACT 23, the processor 711 transmits a product return notification to the cart controller 3 via the access point 4 by the wireless unit 714. The product return notification may include a terminal ID assigned to the information terminal 7 including the information processing device 71, and a product code indicating the product identified in ACT 15. After that, the processor 711 proceeds to ACT 24.

[0052] The cart controller 3 that has received the product return notice deletes the returned product from the registered purchased products. That is, the cart controller 3 deletes the purchased product data of the product code included in the product return notice, which is stored in the purchased product data area 313 of the information terminal table 31 having the terminal ID included in the product return notice, from the purchased product data area 313. The cart controller 3 then creates a registered product screen for notifying the registered product based on the purchased product data stored in the purchased product data area 313, and transmits the registered product screen to the sender of the product input notice.

[0053] In ACT24, the processor 711 receives the registered product screen transmitted from the cart controller 3 via the access point 4, by the wireless unit 714. After that, the processor 711 proceeds to ACT25.

[0054] In ACT25, the processor 711 presents the received registered product screen to the customer using the cart 8 via the touch panel 715. After that, the processor 711 proceeds to ACT11.

[0055] The product identification process may be executed by the store server 2. In this case, the store server 2 performs the processes from ACT14 to ACT16. The processor 711 outputs the latest image and the past image that are determined to have changed to the store server 2. The store server 2 identifies the product that is moving in the image. Specifically, the store server 2 performs image recognition on the object in the bounding box set in the latest image based on the learning data 211 stored in the product database 21, and identifies which product the object is. Product identification can be realized by a known technology. The store server 2 creates a registered product screen based on the product identification result and the determination result of whether the product has been inserted, and transmits the registered product screen to the sender of the image. The processor 711 receives the registered product screen as ACT18 or ACT24.

[0056] According to this example, when a product that the user of the cart 8 has inserted is erroneously recognized by the cart POS system, the user can cancel the erroneously recognized product. The information terminal 7 can delete the erroneously recognized product from the registered products based on the user's operation. For example, in a frictionless cart POS system in which a customer inserts a product into the cart and product registration is automatically performed, the product inserted by the user may be erroneously recognized. When the user removes the erroneously recognized product from the cart, the cart POS system may recognize the removed product again, and may correctly recognize the product, different from when it was inserted. In such a case, the product was erroneously recognized when it was inserted and correctly recognized when it was removed, so that a discrepancy occurs between the inserted product and the removed product, and the cancellation process may not be performed appropriately. Even in such a case, the above-mentioned function allows the information terminal 7 to cancel the product that has been erroneously recognized by the customer's operation. This allows the information terminal 7 to appropriately cancel the erroneously recognized product.

[0057] Furthermore, the information terminal 7 can output a notification indicating the cancellation of the product to the store terminal 9, etc., based on the cancellation operation of the erroneously recognized product by the user of the cart 8. This allows the store clerk to recognize that the user has performed the cancellation operation of the erroneously recognized product. Based on the notification, the store clerk can determine whether the cancellation operation of the erroneously recognized product has been performed appropriately.

[0058] The registered product screen for canceling erroneous recognition will now be described. FIG. 6 is a diagram showing a display example of the information terminal 7 according to the embodiment. FIG. 6 shows the registered product screen Ima displayed on the touch panel 715.

[0059] The registered product screen Ima includes a registered product list La and a cancel error recognition button Ba. The registered product list La is a list including information on registered products. The cancel error recognition button Ba is an operation key for inputting a cancellation instruction to cancel a misrecognized product. The user inputs an instruction to cancel a misrecognized product by operating the cancel error recognition button Ba. The processor 711 acquires an instruction to cancel a misrecognized product based on the user's operation of the cancel error recognition button Ba. For example, the user selects a product to be canceled from the registered product list La. The user inputs the cancel error recognition button Ba. As a result, the processor 711 acquires the cancellation instruction and information on the product to be canceled.

[0060] An example will be described in which the product added by the user to cart 8 is a snack food. Assume that the processor 711 mistakenly recognizes the added product as a plastic bottle juice. The processor 711 identifies the added product as "plastic bottle juice." The processor 711 displays "plastic bottle juice" in the registered product list La of the registered product screen Ima. The user of cart 8 determines that the product has been mistakenly recognized because the added product and the product displayed in the registered product list La are different. The user of cart 8 selects the mistaken recognition cancel button Ba by touch input or the like, and inputs an instruction to cancel the mistakenly recognized product. The processor 711 acquires the cancellation instruction and information on the product to be canceled, and performs the cancellation process described above.

[0061] The process of selecting the product to be cancelled may be omitted. For example, when there is only one registered product, the processor 711 may determine that the registered product is the product to be cancelled. When cancelling the latest registered product, the processor 711 may determine that the latest registered product is the product to be cancelled.

[0062] According to this example, when a misrecognized product is registered, the user of cart 8 can easily cancel the misrecognized product by selecting the product to be registered on the registration product screen and selecting the cancel misrecognition button.

[0063] The screen for confirming the operation for canceling the misrecognition will now be described. FIG. 7 is a diagram showing a display example of the store terminal 9 according to the embodiment.

[0064] FIG. 7 shows a cancellation confirmation screen Imb displayed on the display device of the store terminal 9.

[0065] The cancellation confirmation screen Imb includes an erroneous recognition image area Ara and a canceled product display area Arb. The erroneous recognition image area Ara is an area that displays an image including the canceled product captured by the camera 73 when the canceled product is inserted. The canceled product display area Arb is an area that displays information about the canceled product.

[0066] An example will be described using the case of Fig. 6. In Fig. 7, an image captured when a user of cart 8 puts snacks into the cart is displayed in the misrecognition image area Ara as an image including the product to be cancelled. As information on the cancelled product, "bottled juice" is displayed in the cancelled product display area Arb. The store clerk compares the image including the product to be canceled with the information about the canceled product. If the image including the product to be canceled and the information about the canceled product show different products, the store clerk determines that the cancellation operation by the user of the cart 8 is appropriate. In this case, the store clerk may operate the store terminal 9 to approve the cancellation operation by the user. If the image including the product to be canceled and the information about the canceled product show the same product, the store clerk determines that the cancellation operation by the user of the cart 8 is inappropriate. In this case, the store clerk may operate the store terminal 9 to correct the cancellation operation by the user.

[0067] The cancellation confirmation screen Imb may be displayed on a mobile terminal carried by the store clerk or on the touch panel 715. When the cancellation confirmation screen Imb is displayed on the touch panel 715, the store clerk may approve or correct the cancellation operation by the user on the touch panel 715 depending on the store clerk's operation mode or the like.

[0068] According to this example, the store clerk who receives a notification that a cancel operation has been performed by the user of cart 8 can check whether the cancel operation is appropriate based on the image including the product to be canceled and the information of the canceled product. Therefore, the cart POS system can help prevent fraudulent behavior that utilizes a mistakenly recognized cancel operation.

[0069] (Other embodiments) In the above embodiment, the trigger for starting object detection is when there is a change in the image, but this is not limited to this. For example, at least one sensor that optically detects an object present within the field of view of the camera 73 may be disposed at any position on the cart 8, and product recognition may be started in response to the sensor detecting an object being put into or taken out of the cart 8.

[0070] Furthermore, the judgment of whether a product has been put in or taken out is not limited to being judged from an image. For example, it is possible to place a weighing scale on the bottom surface of the basket receiving portion 81 of the shopping cart 8 and judge whether a product has been put in or taken out based on whether the weight has increased or decreased.

[0071] The product registration terminal may be realized as one device as explained by taking the information terminal 7 as an example, or may be realized as a plurality of devices with distributed functions.

[0072] The above-described embodiment may be applied to a method executed by the device. The above-described embodiment may be applied to a program capable of causing a computer of the device to execute each function. The above-described embodiment may be applied to a recording medium storing the program.

[0073] Each of the one or more circuits constituting the processing circuit executes one or more processes among the multiple processes. When the processing circuit is composed of a single circuit, the single circuit executes all of the multiple processes. When the processing circuit is composed of multiple circuits, each of the multiple circuits executes a part of the multiple processes. The part of the multiple processes may be one of the multiple processes, or two or more of the multiple processes. When the processing circuit is composed of multiple circuits, the multiple circuits may be included in one device, or may be distributed across multiple devices.

[0074] The program may be transferred in a state where it is stored in the device according to the embodiment, or in a state where it is not stored in the device. In the latter case, the program may be transferred via a network, or in a state where it is recorded on a recording medium. The recording medium is a non-transitory tangible medium. The recording medium is a computer-readable medium. The recording medium may be in any form, such as a CD-ROM or a memory card, as long as it is capable of storing the program and is readable by a computer.

[0075] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims.

[0076] The above embodiment may be expressed as follows. (1) an object detection means for detecting an object based on an image captured by an imaging unit; a product identification means for identifying a product based on an image including the object detected by the object detection means; a cancellation processing means for acquiring a cancellation instruction for canceling the product identified by the product identification means based on a user operation; an output means for outputting a notice indicating the cancellation of the product based on the acquisition of the cancellation instruction; A product registration terminal comprising: (2) The imaging unit is provided on a cart, The object detection means detects an object based on the fact that a product has been put into the cart. (1) A product registration terminal as described above. (3) The output means outputs an image including the product to be canceled by the cancellation instruction. (1) A product registration terminal as described above. (4) detecting an object based on an image captured by the imaging unit; identifying a product based on an image including the detected object; Obtaining a cancellation instruction to cancel the identified product based on a user operation; outputting a notification indicating the cancellation of the product based on the acquisition of the cancellation instruction; A product registration method comprising: (5) To the computer: A function of detecting an object based on an image captured by an imaging unit; A function of identifying a product based on an image including the detected object; A function of acquiring a cancellation instruction for canceling the identified product based on a user operation; a function of outputting a notification indicating the cancellation of the product based on the acquisition of the cancellation instruction; A program that can execute the following. [Explanation of symbols]

[0077] 1... cart POS system, 2... store server, 3... cart controller, 4... access point, 5... accounting machine, 6... network, 7... information terminal, 8... shopping cart, 9... store terminal, 21... product database, 22... member database, 31... information terminal table, 71... information processing device, 72... reader, 73... camera, 74... patrol lamp, 81... basket receiving section, 211... learning data, 311... terminal ID area, 312... member ID area, 313... purchase Product data area, 314...subtotal area, 711...processor, 712...main memory, 713...auxiliary storage device, 714...wireless unit, 715...touch panel, 716...speaker, 717...system transmission path, 731...LED lamp, 7131...terminal program, 7132...learning data, Ara...misrecognized image area, Arb...cancel product display area, Ba...misrecognition cancel button, Ima...registered product screen, Imb...cancellation confirmation screen, La...registered product list.

Claims

1. an object detection means for detecting an object based on an image captured by the imaging unit; a product identification means for identifying a product based on an image including the object detected by the object detection means; a cancellation processing means for acquiring a cancellation instruction for canceling the product identified by the product identification means based on a user operation; an output means for outputting a notice indicating the cancellation of the product based on the acquisition of the cancellation instruction; A product registration terminal comprising:

2. The imaging unit is provided on a cart, The object detection means detects an object based on the fact that a product has been put into the cart. The product registration terminal according to claim 1.

3. The output means outputs an image including the product to be canceled by the cancellation instruction. The product registration terminal according to claim 1.

4. Detecting an object based on an image captured by an imaging unit; identifying a product based on an image including the detected object; Obtaining a cancellation instruction to cancel the identified product based on a user operation; outputting a notification indicating the cancellation of the product based on the acquisition of the cancellation instruction; A product registration method comprising:

5. On the computer, A function of detecting an object based on an image captured by an imaging unit; A function of identifying a product based on an image including the detected object; A function of acquiring a cancellation instruction for canceling the identified product based on a user operation; a function of outputting a notification indicating the cancellation of the product based on the acquisition of the cancellation instruction; A program that can execute the following.

Images