Article registration device and program

Abstract

To provide an article registration device and a program that prompt a customer to re-put an article when the reliability of recognition of the article that the customer inputs to a basket is deteriorated.SOLUTION: A tablet terminal (article registration device) includes: an imaging control unit that allows at least one camera (imaging device) to capture a plurality of temporally consecutive images of an area including a basket (storage unit) which is provided in a shopping cart and to which an article is inputted; an article recognition unit (recognition unit) that recognizes the article inputted to the basket from the plurality of images captured by the imaging control unit; an article registration unit (registration unit) that registers the article recognized by the article recognition unit as an article to be purchased; a recognition score lowering factor estimation unit (estimation unit) that estimates a cause of a recognition score becoming equal to or less than a threshold value based on a state of an image when the recognition score representing the correctness of recognition by the article recognition unit is equal to or less than the threshold value; and a notification unit that instructs a customer to re-input the article by notifying the customer with a message corresponding to the cause estimated by the recognition score lowering factor estimation unit.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] An embodiment of the present invention relates to a product registration device and a program. [Background technology]

[0002] Recently, a self-shopping system has been proposed that uses a camera to capture and recognize an item that a customer has put into a shopping cart, and then registers the item (for example, Patent Document 1).

[0003] In such self-shopping systems, depending on how a customer places items into a shopping cart, the image may become blurred or the characteristic surfaces needed to identify the product may be covered by the customer's hand, reducing the accuracy of product identification. Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a product registration device and a program that prompt a customer to re-insert a product into a basket when the reliability of recognition of the product that the customer has inserted into the basket has decreased. [Means for solving the problem]

[0005] The product registration device of the embodiment includes an imaging control unit, a recognition unit, a registration unit, an estimation unit, and an alarm unit. The imaging control unit causes at least one imaging device to capture multiple images that are consecutive in time of an area of ​​a shopping cart including a storage unit into which a product to be purchased is inserted. The recognition unit recognizes the product inserted into the storage unit from the multiple images captured by the imaging control unit. The registration unit registers the product recognized by the recognition unit as a product to be purchased. When a recognition score indicating the accuracy of recognition by the recognition unit is equal to or lower than a threshold, the estimation unit estimates a cause of the recognition score being equal to or lower than the threshold from multiple causes prepared in advance based on the state of the image. The alarm unit instructs the customer to reinsert the product by issuing a message according to the cause estimated by the estimation unit. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of a product sales data processing system including a tablet terminal according to the present embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of a shopping cart equipped with a tablet terminal according to the present embodiment. [Diagram 3] FIG. 3 is a hardware block diagram showing an example of the hardware configuration of the tablet terminal of this embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a data structure of the product master. [Diagram 5] FIG. 5 is a diagram illustrating an example of a data structure of the product registration file. [Figure 6] FIG. 6 is a diagram for explaining an outline of image processing for detecting that an article has been put into a shopping cart, based on an image captured by a camera. [Figure 7] FIG. 7 is a diagram showing an example of a cause of a decrease in the recognition score of a recognized product when the product is put into the basket, and a notification content when the cause is estimated. [Figure 8] FIG. 8 is a functional block diagram showing an example of the functional configuration of the tablet terminal of the present embodiment. [Figure 9] FIG. 9 is a flowchart showing an example of the flow of the product registration process performed by the tablet terminal of this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] (Embodiment) Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the present invention is not limited to the embodiments described below.

[0008] (Configuration of product sales data processing system) A schematic configuration of a product sales data processing system 1 including a tablet terminal 10 of the embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing a schematic configuration of a product sales data processing system including a tablet terminal of the embodiment.

[0009] 1, the product sales data processing system 1 includes a tablet terminal 10, a server 2, and a payment machine 4. The server 2 and the payment machine 4 are communicatively connected via a network Na.

[0010] The network Na may be the Internet, a virtual private network (VPN), a local area network (LAN), a public communication network, a mobile communication network, or the like, either alone or in appropriate combination. As an example, the network Na may be a combination of the Internet and a mobile communication network.

[0011] In addition, a plurality of wireless access points 3 are connected to the network Na. The wireless access points 3 are connected via a network Nb such as a wireless LAN to a tablet terminal 10 that is installed in a shopping cart 20 and moves around the store accompanying a customer. Therefore, the tablet terminal 10 can communicate with the server 2 and the cashier 4 via the network Nb and the network Na.

[0012] The shopping cart 20 includes a basket 23 (see FIG. 2) for storing products displayed in the store. The top of the basket 23 is open, and products are placed into the basket 23 from the top. A customer borrows a shopping cart 20 at a store, and while moving around the store with the shopping cart 20, places products to be purchased from those displayed into the basket 23.

[0013] The tablet terminal 10 is installed so as to be attachable to the shopping cart 20. The tablet terminal 10 acquires an image captured by the camera 17 (see FIG. 2), recognizes that a product has been added to the basket 23, and executes a sales registration process for the product. The sales registration process refers to a process for making the product identified by the tablet terminal 10 the target for purchase. Specifically, the sales registration process is a process for registering information about the product recognized as having been added to the basket 23 (hereinafter also referred to as sales registration data) in a product registration file 124 (see FIG. 3) described later. Through the sales registration process, sales registration data including a product code capable of identifying the product type, etc., and the number of purchases of the product, etc., is registered in the product registration file 124.

[0014] Furthermore, when the tablet terminal 10 accepts an operation by a customer to perform a transaction at the payment machine 4, it executes a process to transfer the sales registration data for one transaction to the payment machine 4. For example, the tablet terminal 10 encodes the sales registration data into a code symbol such as a two-dimensional code and displays it on its display device 15 (see FIG. 3). The payment machine 4 reads the code symbol displayed on the tablet terminal 10 to obtain the sales registration data for one transaction that was registered on the tablet terminal 10, and executes payment processing based on the sales registration data. The tablet terminal 10 is an example of a product registration device in this disclosure.

[0015] The server 2 uses the tablet terminal 10 as a user interface terminal and performs information processing to provide a service for carrying out buying and selling transactions of goods in a store where the user interface terminal is used. The server 2 may be, for example, a store server provided in the store, or a server device provided outside the store. Note that in this embodiment, the server 2 is realized by a single device, but it may also be a cloud server (or cloud system) realized by multiple devices connected to a network.

[0016] The server 2 is an information processing device configured as a computer equipped with a processor, a main storage device, an auxiliary storage device, etc. The server 2 stores a product master in which the names and prices of products sold in a store are registered in a database (not shown) and provides the product master to the tablet terminal 10 and the payment machine 4.

[0017] The accounting machine 4 is an accounting device installed in a store. The accounting machine 4 executes the payment process for products based on sales registration data. The payment process includes displaying the total price and tax amount of products related to one transaction, calculating and displaying change based on the deposit received from the customer, and issuing a receipt with product information and payment information (total price, deposit amount, change amount, etc.) printed on it. Note that multiple accounting machines 4 may be installed to make the payment process more efficient.

[0018] (Outline of shopping cart configuration) A schematic configuration of the shopping cart 20 will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of a shopping cart equipped with a tablet terminal according to the present embodiment.

[0019] The shopping cart 20 includes a basket-like basket 23 with an open top, rotating casters 22 arranged below the basket 23, and a handle 21 connected to the basket 23 and the casters 22 and held by a customer when moving the shopping cart 20.

[0020] The basket 23 has a structure in which the side and bottom parts are surrounded by a mesh-like lattice in a substantially rectangular shape and the top is open, so that the products put into the basket 23 can be visually observed from the outside. The side and bottom parts may be configured by walls instead of the mesh-like lattice. The basket 23 is an example of a storage section in the present disclosure.

[0021] The shopping cart 20 also includes a detachable tablet terminal 10 near the handle 21. The shopping cart 20 also includes a camera 17 near the edge of the opening of the basket 23. The camera 17 is an example of an imaging device in the present disclosure, and is realized by, for example, a digital camera. The camera 17 is attached to the shopping cart 20 so that the imaging direction faces the opening of the basket 23. In other words, the imaging direction is set so that at least a portion of the imaging range 18 of the camera 17 includes the interior of the basket 23.

[0022] The camera 17 captures images of products put into the basket 23 or products taken out of the basket 23, and outputs the captured images to the tablet terminal 10. The camera 17 captures images that are continuous over time at a predetermined frame rate, for example, 30 frames per second, and outputs the images to the tablet terminal 10.

[0023] In addition, the shopping cart 20 may be equipped with multiple cameras 17 that capture images of the area including the basket 23 from different positions in order to reduce blind spots when a customer puts items into the basket 23.

[0024] (Hardware configuration of tablet device) The hardware configuration of the tablet terminal 10 will be described with reference to Fig. 3. Fig. 3 is a hardware block diagram showing an example of the hardware configuration of the tablet terminal of this embodiment.

[0025] The tablet terminal 10 includes a control unit 11, a storage unit 12, a peripheral device controller 13, and a communication interface 14.

[0026] The control unit 11 controls the overall operation of the tablet terminal 10. The control unit 11 includes a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, and a RAM (Random Access Memory) 113. The CPU 111 is connected to the ROM 112 and the RAM 113 via internal buses such as an address bus and a data bus. The CPU 111 loads various programs stored in the ROM 112 and the storage unit 12 into the RAM 113. The CPU 111 controls the operation of the tablet terminal 10 by operating in accordance with the various programs loaded into the RAM 113. That is, the control unit 11 has the configuration of a general computer.

[0027] The storage unit 12 is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The storage unit 12 may also be a non-volatile memory such as a flash memory that retains stored information even when the power is turned off. The storage unit 12 stores a control program 121, product recognition dictionary data 122, a product master 123, and a product registration file 124.

[0028] The control program 121 is a program that controls the overall operation of the tablet terminal 10. The control program 121 may be provided in a state stored in the storage unit 12, or may be provided by being recorded in an installable or executable file format on a non-transitory computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD. The control program 121 may also be provided by being stored on a computer connected to a network and downloaded via the network. Furthermore, the control program 121 may be provided or distributed via a network such as the Internet.

[0029] The commodity recognition dictionary data 122 is dictionary data for recognizing a commodity shown in an image captured by the camera 17. The commodity recognition dictionary data 122 may be in any format, and may be, for example, a dictionary in which feature quantities characterizing a commodity are stored for each commodity, or a pattern classification model generated from a plurality of commodity images acquired by machine learning. The pattern classification model is described, for example, in the form of a neural network.

[0030] The product master 123 is a master file that stores product codes that uniquely identify products sold in a store and product information of the products in association with each other. The data structure of the product master 123 will be described in detail later (see FIG. 4).

[0031] The product registration file 124 is a file that stores the product codes of the products that the customer has put into the basket 23, that is, the products that the customer plans to purchase. The data structure of the product registration file 124 will be described in detail later (see FIG. 5).

[0032] The control unit 11 is connected, via a peripheral device controller 13, to a display device 15 which inputs and outputs various types of information, an operation device 16, and a camera 17.

[0033] The display device 15 is a display device such as a liquid crystal monitor or an organic EL monitor provided in the tablet terminal 10. The display device 15 displays a product list of the products that the customer has put in the basket 23, and the like.

[0034] The operation device 16 is an information input device such as a touch panel that is layered on the display device 15. After the customer places the product to be purchased in the basket 23, the customer transmits an instruction to the tablet terminal 10 to perform the transaction at the transaction machine 4 by, for example, pressing an operator such as a checkout button displayed on the display device 15.

[0035] As described above, the camera 17 captures images in the imaging range 18 that are continuous over time.

[0036] The control unit 11 wirelessly communicates with the server 2 and the accounting machine 4 via the communication interface 14. The tablet terminal 10 obtains the latest data of the product recognition dictionary data 122 and the product master 123, which are updated each time, by wirelessly communicating with the server 2. In addition, the tablet terminal 10 wirelessly communicates with the accounting machine 4 to send sales registration data registered in the product registration file 124 to the accounting machine 4, and causes the accounting machine 4 to perform accounting processing.

[0037] (Product master data structure) The data structure of the product master 123 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the data structure of the product master.

[0038] The product master 123 stores a product code that uniquely identifies a product and product information of the product in association with each other. The product code is, for example, a JAN (Japanese Article Number) code.

[0039] The product information includes a product name, a product type, a unit price, and additional information.

[0040] The product name is the name of the product.

[0041] The product type is type information to which the product belongs. The product type includes, for example, major categories such as food and daily necessities, and medium categories within food such as rice, noodles, vegetables, meat, etc. The product type may include further subcategories.

[0042] The unit price is the unit price of the product.

[0043] The additional information is additional information related to the product, and includes, for example, discount information.

[0044] The product master 123 may store other product information than the above.

[0045] (Product registration file data structure) The data structure of the product registration file 124 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the data structure of the product registration file.

[0046] The product registration file 124 is a file that stores product codes in association with the number of items added to the basket 23. The product code is product identification information of a product that has been registered as a sales item, and corresponds to the product code in the product master 123. The number of items is the registered number of items of the same product that has been registered as a sales item.

[0047] The data structure of the product registration file 124 is not limited to the example shown in Fig. 5. For example, the product information corresponding to the product code may be registered in association with the product code. Also, the date and time information indicating the date and time of registration may be registered in association with the product code.

[0048] Furthermore, in this embodiment, the tablet terminal 10 is configured to hold the product registration file 124, but this is not limiting, and the server 2 may be configured to hold the product registration file 124. In this case, the server 2 can manage the product registration file 124 for each transaction, i.e., for each tablet terminal 10, by issuing a transaction ID that can uniquely identify a transaction performed on each tablet terminal 10, and holding the product registration file 124 in association with the transaction ID.

[0049] In such a configuration, the server 2 notifies the tablet terminal 10 of the issued transaction ID, thereby causing the tablet terminal 10 to retain the transaction ID while the transaction of that transaction ID is being carried out. The tablet terminal 10 then transmits the product code of the product whose sales has been registered to the server 2 together with the transaction ID, and instructs the server 2 to register the product in the product registration file 124 corresponding to that transaction ID. This allows the server 2 to centrally manage the products whose sales have been registered on each tablet terminal 10. Note that the tablet terminal 10 is capable of referring to the product registration file 124 of the transaction related to its own terminal from the server 2.

[0050] (Method of detecting items placed in a basket) A method for detecting that an article has been put into the basket 23 will be described with reference to Fig. 6. Fig. 6 is a diagram for explaining an outline of image processing for detecting that an article has been put into the basket of a shopping cart, based on an image captured by a camera.

[0051] The camera 17 captures images Ia, Ib, and Ic, which are successive in time, as shown in Fig. 6. Image Ia is an image captured at time t = t0. Image Ib is an image captured at time t = t0 + Δt. Image Ic is an image captured at time t = t0 + 2Δt. Δt is, for example, 1 / 30 seconds. That is, images Ia, Ib, and Ic are images captured at 1 / 30 second intervals. In this embodiment, images Ia, Ib, and Ic are called successive images in time.

[0052] Note that images Ia, Ib, and Ic show a part of the basket 23 of the shopping cart 20. Therefore, when a customer puts a product 30 into the basket 23, there is a high probability that the product 30 will be shown in images Ia, Ib, and Ic. However, it is not known when and at what timing the product 30 will be put into the basket 23. For this reason, the camera 17 continuously captures images at intervals of, for example, 1 / 30 seconds. Note that images Ia, Ib, and Ic shown in FIG. 6 are examples of images captured at the moment the product 30 is put into the basket 23.

[0053] The tablet terminal 10 acquires the image captured by the camera 17 and performs a detection process to detect whether an article has been placed in the basket 23.

[0054] First, the tablet terminal 10 performs a process of subtracting the image Ib from the acquired image Ia, that is, a so-called frame difference, to generate an image Id. The frame difference is a process of calculating the difference in pixel values ​​between corresponding pixels of two images. That is, the calculation Id=Ia-Ib is performed. As a result of the calculation, if the pixel value of the image Ia is larger (brighter) than the pixel value of the image Ib, the difference value will be a positive value, but if the pixel value of the image Ia is smaller (darker) than the pixel value of the image Ib, the difference value will be a negative value. Therefore, in this embodiment, the result of the frame difference is output as the absolute value of the difference value.

[0055] Therefore, if there is a change in brightness between images Ia and Ib, a large difference value is stored in the area where the change in brightness occurred. The tablet terminal 10 performs a so-called binarization process in which a "1" is assigned to pixels in image Id whose difference value is equal to or greater than a predetermined threshold, and a "0" is assigned to pixels whose difference value is below the predetermined threshold. As a result, an area 32 where there is a large change in brightness is visualized, as in image Id shown in FIG. 6. At this time, a "0" is assigned to the area of ​​the cage 23 where there is no change in brightness between images Ia and Ib, and therefore the area without movement is excluded from the object detection target.

[0056] The tablet terminal 10 also performs a similar frame difference calculation on the images Ib and Ic to generate an image Ie. That is, it performs the calculation Ie=Ib-Ic. Then, as a result of the calculation, it performs a binarization process in which a "1" is assigned to pixels whose difference value is equal to or greater than a predetermined threshold, and a "0" is assigned to pixels whose difference value is below the predetermined threshold. As a result, an area 33 with a large change in brightness is visualized, as in the image Ie shown in FIG. 6.

[0057] 6, it can be seen that the position of the area moves from the top to the bottom of the image, that is, from the outside of the cage 23 to the inside of the cage 23, over time. At this time, the tablet terminal 10 determines that some article has been put into the cage 23 between time t=t0 and time t=t0+2Δt.

[0058] The comparison of the positions of the region 32 and the region 33 can be performed, for example, by comparing the position of the center of gravity Ga of the region 32 with the position of the center of gravity Gb of the region 33. For example, in the example of Fig. 6, the center of gravity Gb of the region 33 is located lower in the image than the center of gravity Ga of the region 32, so it can be determined that the region 32 has moved over time to the lower region 33, i.e., in a direction toward the inside of the cage 23. Mathematically, when the direction of a vector starting from the center of gravity Ga and ending at the center of gravity Gb is downward, i.e., toward the inside of the cage 23, the tablet terminal 10 determines that an article has been put into the cage 23.

[0059] Note that the method of detecting an article shown in Fig. 6 is an example and is not limited to this method. Whether an article has been put into the basket 23 may be determined, for example, by calculating a motion vector (optical flow) of an object between images from only two images Ia and Ib. Specifically, pairs of feature points (or pairs of feature regions) that are considered to represent the same point (or region) between images may be extracted, a motion vector connecting each of the feature points (or a motion vector connecting each of the feature regions) may be calculated, and the moving direction of the object may be estimated from the distribution of the directions of the calculated motion vectors.

[0060] Although not shown, the tablet terminal 10 can also determine in a similar manner whether an article has been removed from the basket 23.

[0061] In this case, if the direction of a vector starting from the center of gravity Ga of area 32 in image Id generated in the same manner as described above and ending at the center of gravity Gb of area 33 in image Ie is upward, i.e., toward the top of the basket 23, the tablet terminal 10 determines that an item has been removed from the basket 23.

[0062] In addition, in Figure 6, an example of a process is described in which three temporally consecutive images are used to detect whether an item has been put into the basket 23 and whether an item has been removed from the basket 23. However, the number of images used is not limited to three, and a greater number of temporally consecutive images may be used as the detection target.

[0063] Furthermore, if shopping cart 20 is equipped with multiple cameras 17, it is desirable for each camera 17 to be synchronized and capture images at the same time.

[0064] (How to recognize items added to the cart) Next, a method in which the tablet terminal 10 recognizes which product the article put in the basket 23 is, after it has been detected, will be described.

[0065] The tablet terminal 10 performs a general object recognition process for each of the multiple images that have led to the determination that an item has been placed in the basket 23, for example, in the example of Figure 6, three images, image Ia, image Ib, and image Ic, to recognize which product is depicted in each image.

[0066] The tablet terminal 10 compares the captured image with the product recognition dictionary data 122 to calculate a recognition score Sc indicating how similar the item in the image is to the product, and performs a general object recognition process to recognize the product based on the calculated recognition score Sc.

[0067] The recognition score Sc is expressed as a value between 0.0 and 1.0, for example, and the closer to 1.0, the more likely the product is. The recognition score Sc is calculated, for example, by the angle between a feature vector formed by a plurality of feature amounts calculated from an item detected from an image and a feature vector formed by a plurality of feature amounts indicating the likeliness of the item, registered in the commodity recognition dictionary data 122. When the two feature vectors completely match, that is, when the feature vector of the captured item completely matches the feature vector of the commodity recognition dictionary data 122, the angle between the two feature vectors is 0 degrees, and the recognition score Sc at this time is 1.0. On the other hand, when the two feature vectors face in opposite directions, that is, when the feature vector of the captured item and the feature vector of the commodity recognition dictionary data 122 face in opposite directions, the angle between the two feature vectors is 180 degrees, and the recognition score Sc at this time is 0.0.

[0068] The tablet terminal 10 determines that the object shown in the three images, image Ia, image Ib, and image Ic, is the specific product if the object has the highest recognition score Sc for being recognized as the specific product and if the recognition score Sc exceeds a predetermined threshold value TH.

[0069] The tablet terminal 10 may recognize a product by recognizing a code symbol, such as a barcode or a two-dimensional code, to which a product code is registered, attached to the item. This makes it easier to recognize a product 30 to which a code symbol is attached.

[0070] (Analysis of factors when product recognition scores drop and notification of product re-release) A cause of a decrease in the recognition score of an item added to the basket and the content of a notification by the tablet terminal 10 when the cause is estimated will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of a cause of a decrease in the recognition score of a recognized item when the item is added to the basket and the content of a notification when the cause is estimated.

[0071] When the tablet terminal 10 recognizes which product is the item put into the basket 23, it calculates the above-mentioned recognition score Sc, which indicates the accuracy of the recognition. Then, the tablet terminal 10 determines whether or not a product whose recognition score Sc exceeds a threshold value TH has been uniquely identified.

[0072] The tablet terminal 10 performs notification based on the notification content table 35 shown in FIG. 7 in accordance with the recognition result of the product.

[0073] The notification content table 35 includes a product identification result 36, an image state 37, and notification content 38.

[0074] The product identification result 36 is an item for classifying the product recognition result. The product identification result 36 indicates whether the product was uniquely identified, whether the product was not uniquely identified, whether the recognition score Sc is equal to or smaller than the threshold value TH, or whether the recognition score Sc exceeds the threshold value TH.

[0075] The image status 37 is an item showing the status of the above-mentioned image Ia, image Ib, and image Ic. The image status 37 includes, for example, (1) multiple products are shown, (2) the characteristics of the products cannot be read, (3) the image is blurred, (4) other, and (5) good.

[0076] The state of "(1) multiple products appear" refers to a state in which the products cannot be uniquely identified, or the products can be uniquely identified but the recognition score Sc is equal to or less than the threshold value TH, and two or more products appear simultaneously in the image captured by the camera 17. In this way, when two or more products appear simultaneously in one image, the accuracy of the product recognition system may be extremely reduced, and therefore it is necessary to have the customer re-insert the products one by one.

[0077] The state of "(2) Unable to read product characteristics" refers to a state in which the product cannot be uniquely identified, or the product can be uniquely identified but the recognition score Sc is equal to or less than the threshold value TH, and the product's characteristic parts, such as the product's label or the outer edge showing the product's shape, are not captured on the item captured in the image captured by the camera 17. In such a case, the customer needs to change the orientation of the product and reinsert it.

[0078] The "(3) image is blurred" state refers to a state in which the product cannot be uniquely identified, or the product can be uniquely identified but the recognition score Sc is equal to or less than the threshold value TH, and the image captured by the camera 17 is blurred. If the captured image is blurred, the product cannot be correctly recognized. In such a case, it is necessary to ask the customer to slowly reinsert the product.

[0079] The state of "(4) Other" refers to a state in which the product cannot be uniquely identified, or the product can be uniquely identified but the recognition score Sc is equal to or less than the threshold value TH, and none of the above applies. In such a case, the customer needs to reinsert the product.

[0080] The "(5) good" state is a state in which the product can be uniquely identified and the recognition score Sc exceeds the threshold value TH. In this case, the product put into the basket 23 is correctly recognized and registered in the product registration file 124, so there is no need to notify the customer.

[0081] The notification content 38 is the content of the notification that the tablet terminal 10 gives to the customer in accordance with the image state 37.

[0082] In the case of the state "(1) multiple products are displayed", the tablet terminal 10 notifies the customer "Please insert the products one by one".

[0083] In the case of the state "(2) the product characteristics cannot be read," the tablet terminal 10 notifies the customer "please change the product and insert it."

[0084] In the case of the "(3) image is blurred" state, the tablet terminal 10 notifies the customer "Please insert the product slowly."

[0085] In the case of the "(4) Other" state, the tablet terminal 10 notifies the customer to "reinsert the product."

[0086] Moreover, in the case of the "(5) good" state, since it has been reliably recognized that the product has been put into the basket 23, the tablet terminal 10 does not notify the customer.

[0087] In this way, the tablet terminal 10 estimates the reason why the product recognition score Sc has fallen to or below the threshold value TH from among a plurality of reasons based on the image state 37 prepared in advance.

[0088] The tablet terminal 10 issues the notification by displaying the above-mentioned message on the display device 15. At this time, the tablet terminal 10 may also display the captured image on the display device 15 together with the notification. This allows the customer to recognize the state of the captured image, and therefore the customer can reinsert the product more carefully. The tablet terminal 10 may also output the above-mentioned message by voice from a speaker (not shown).

[0089] Note that, although the above description has been given of the case where a product is placed in the basket 23, the tablet terminal 10 may also notify the customer in accordance with the state of the image when a product is removed from the basket 23, allowing the customer to remove the product again.

[0090] (Tablet device function configuration) The functional configuration of the tablet terminal 10 will be described with reference to Fig. 8. Fig. 8 is a functional block diagram showing an example of the functional configuration of the tablet terminal of this embodiment.

[0091] The control unit 11 of the tablet terminal 10 reads out the control program 121, deploys it in the RAM 113 provided in the control unit 11, and executes it to realize, as functional units, the imaging control unit 41, the item detection unit 42, the input / removal determination unit 43, the product recognition unit 44, the recognition score decrease cause estimation unit 45, the notification unit 46, the product registration unit 47, and the payment processing unit 48 shown in FIG. 8.

[0092] The imaging control unit 41 controls at least one camera 17 (imaging device) to capture multiple consecutive images in time of an area (imaging range 18) including a basket 23 (storage section) provided in the shopping cart 20 into which the product 30 to be purchased is placed.

[0093] The object detection unit 42 detects whether or not there is a moving object in a plurality of temporally consecutive images captured by the imaging control unit 41.

[0094] The put-in / take-out determination unit 43 determines whether the article detected by the article detection unit 42 is an article that has been put into the basket 23 or an article that has been taken out of the basket 23 .

[0095] The commodity recognition unit 44 recognizes which commodity is the item put into the basket 23 from the multiple images captured by the imaging control unit 41. The commodity recognition unit 44 also recognizes which commodity is the item taken out of the basket 23 from the multiple images captured by the imaging control unit 41. The specific recognition method is as described above. The commodity recognition unit 44 is an example of a recognition unit in the present disclosure.

[0096] When the recognition score Sc, which indicates the accuracy of recognition by the commodity recognition unit 44, is equal to or less than the threshold value TH, the recognition score reduction factor estimation unit 45 estimates the cause of the recognition score Sc becoming equal to or less than the threshold value TH from among a plurality of causes prepared in advance, based on the state of the image captured by the imaging control unit 41. Note that the recognition score reduction factor estimation unit 45 is an example of an estimation unit in the present disclosure.

[0097] The notification unit 46 notifies the customer of a message according to the cause estimated by the recognition score decrease cause estimation unit 45, thereby instructing the customer to re-insert the product.

[0098] The product registration unit 47 registers, as a product to be purchased, the product that the product recognition unit 44 recognizes as having been put into the basket 23. Note that the product registration unit 47 is an example of a registration unit in the present disclosure.

[0099] The payment processing unit 48 receives an instruction from the customer to carry out processing related to the payment of the product registered for sale. Then, the payment processing unit 48 executes processing related to the payment of the product registered for sale in response to the customer's operation. Specifically, the payment processing unit 48 inputs data capable of identifying the product registered for sale into the payment machine 4, thereby realizing a state in which payment can be made at the payment machine 4.

[0100] For example, when the payment processing unit 48 receives an operation from a customer via the operation device 16 to instruct the start of payment, it encodes the sales registration data registered in the product registration file 124 into a code symbol such as a two-dimensional code and displays it on the display device 15. In this case, the payment machine 4 obtains the sales registration data by reading the code symbol displayed on the display device 15 of the tablet terminal 10, and executes the payment process for the products for one transaction that have been registered as sales based on the obtained sales registration data.

[0101] Furthermore, for example, the payment processing unit 48 may transmit the sales registration data registered in the product registration file 124 to the payment machine 4 by short-range wireless communication such as Bluetooth (registered trademark). In this case, the payment machine 4 acquires the sales registration data transmitted from the tablet terminal 10, and executes the payment process for the products for one transaction that have been registered as sales, based on the acquired sales registration data.

[0102] Also, for example, in a configuration in which the product registration file 124 is stored in the server 2, the payment processing unit 48 encodes the transaction ID of the transaction into a code symbol such as a barcode or two-dimensional code and displays it on the display device 15. In this case, the payment machine 4 obtains the transaction ID by reading the code symbol displayed on the display device 15 of the tablet terminal 10. The payment machine 4 then obtains sales registration data related to the obtained transaction ID from the server 2, and executes payment processing for the products for one transaction that have been registered as sales, based on the obtained sales registration data.

[0103] The payment for the products is not limited to the payment method using the cashier 4, but may be made using the tablet terminal 10. For example, when the payment processing unit 48 receives an operation from the customer via the operation device 16 to instruct the customer to start payment, the payment processing unit 48 displays a screen on the display device 15 that allows the customer to select a payment method such as electronic money payment or credit card payment. When the payment method is selected, the payment processing unit 48 receives input of payment information for making the payment using the selected payment method, and executes a payment process to settle the total amount of the products registered for sale based on the input payment information by cooperating with an external payment server. The payment information may be, for example, a credit card number or a two-dimensional code for code payment. In the case of a two-dimensional code for code payment, the input may be received via the camera 17.

[0104] (Processing flow performed by tablet devices) The flow of processing performed by the tablet terminal 10 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the flow of product registration processing performed by the tablet terminal of this embodiment.

[0105] The imaging control unit 41 causes the camera 17 to capture a plurality of images of the imaging range 18 that are consecutive in time (step S11).

[0106] The object detection unit 42 detects whether or not there is a moving object by performing, for example, the image processing described with reference to FIG. 6 (step S12).

[0107] The put-in / take-out determination unit 43 determines whether the item detected by the item detection unit 42 in step S12 is an item put into the basket 23 (step S13). If it is determined that the item detected by the item detection unit 42 is an item put into the basket 23 (step S13: Yes), the process proceeds to step S14. On the other hand, if it is determined that the item detected by the item detection unit 42 is not an item put into the basket 23 (step S13: No), the process returns to step S11 and a new image is captured.

[0108] When it is determined in step S13 that the item detected by the item detection unit 42 is an item that has been put into the basket 23, the item recognition unit 44 recognizes which item the item is (step S14).

[0109] Next, the product recognition unit 44 judges whether the product recognition score Sc is equal to or less than the threshold value TH (step S15). If it is judged that the product recognition score Sc is equal to or less than the threshold value TH (step S15: Yes), the process proceeds to step S16. On the other hand, if it is not judged that the product recognition score Sc is equal to or less than the threshold value TH (step S15: No), the process proceeds to step S18.

[0110] If it is determined in step S15 that the recognition score Sc of the product is equal to or less than the threshold value TH, the recognition score decrease cause estimating unit 45 estimates the cause of the decrease in the recognition score Sc (step S16).

[0111] The notification unit 46 notifies the user of the cause of the decrease in the recognition score Sc estimated in step S16, for example, by displaying the cause on the display device 15 of the tablet terminal 10 (step S17). After that, the process returns to step S11.

[0112] Returning to step S15, if it is determined in step S15 that the product recognition score Sc is not equal to or less than the threshold value TH, the product registration unit 47 registers the product in the product registration file 124 as a product to be purchased by the customer (step S18).

[0113] The payment processing unit 48 determines whether registration of all products has been completed (step S19). If it is determined that registration of all products has been completed (step S19: Yes), the tablet terminal 10 ends the processing of FIG. 9. On the other hand, if it is not determined that registration of all products has been completed (step S19: No), the processing returns to step S11. Note that the payment processing unit 48 may determine whether registration of all products has been completed by the customer pressing a checkout button (not shown) displayed on the display device 15, which instructs the start of payment processing.

[0114] 9, in step S13, the put-in / take-out determination unit 43 also determines whether the item detected by the item detection unit 42 is an item taken out of the basket 23. If it is determined that the item detected by the item detection unit 42 is an item taken out of the basket 23, the item registration unit 47 deletes the registration information of the item from the item registration file 124.

[0115] (Effects of the embodiment) As described above, the tablet terminal 10 (product registration device) of the embodiment includes an imaging control unit 41 that causes at least one camera 17 (imaging device) to capture multiple temporally consecutive images of an imaging range 18 including the basket 23 (storage section) of the shopping cart 20 into which the product 30 to be purchased is placed, a product recognition unit 44 (recognition unit) that recognizes the product 30 placed in the basket 23 from the multiple images captured by the imaging control unit 41, a product registration unit 47 (registration unit) that registers the product 30 recognized by the product recognition unit 44 as a product to be purchased, a recognition score reduction factor estimation unit 45 (estimation unit) that, when the recognition score Sc indicating the accuracy of recognition by the product recognition unit 44 is below a threshold TH, estimates the cause why the recognition score Sc has become below the threshold TH from multiple pre-prepared causes based on the state of the image, and an alarm unit 46 that instructs the customer to re-insert the product 30 by alerting the customer with a message corresponding to the cause estimated by the recognition score reduction factor estimation unit 45. Therefore, if the reliability of recognition decreases when recognizing and registering the product 30 that the customer has put in the basket 23, the customer can be notified of the cause of the decrease in the reliability of recognition and encouraged to re-put the product in. Therefore, the customer can re-put the product 30 in the basket 23 while taking note of the notified content so that the product 30 is reliably recognized.

[0116] Furthermore, in the tablet terminal 10 of the embodiment, the camera 17 (imaging device) is attached to the shopping cart 20. Therefore, the movement of the product put into the basket 23 or the product taken out of the basket 23 is reflected in the temporally consecutive images captured by the camera 17. On the other hand, in the temporally consecutive images captured by the camera 17, the area of ​​the basket 23 in the background does not change over time. Therefore, the product recognition unit 44 can easily and reliably recognize the product that moves over time.

[0117] Furthermore, in the tablet terminal 10 of the embodiment, the product recognition unit 44 (recognition unit) further recognizes the product 30 taken out of the basket 23 (container), and the product registration unit 47 (registration unit) cancels the registration of the product 30 that the product recognition unit 44 recognizes as having been taken out of the basket 23. Therefore, when the purchaser gives up on purchasing the product 30 that has been registered, the registration of the product can be cancelled without performing any special operation.

[0118] In addition, in the tablet terminal 10 of the embodiment, the product recognition unit 44 (recognition unit) recognizes a code symbol attached to the product 30 and in which the product code of the product 30 is registered. Therefore, in the case of a product 30 to which a code symbol is attached, the product 30 can be more easily recognized.

[0119] Although the embodiments of the present invention have been described above, these embodiments are merely examples and are not intended to limit the scope of the invention. This novel embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0120] 1. Product sales data processing system 2 Server 3. Wireless Access Points 4. Cashier 10 Tablet terminal (product registration device) 15 Display Devices 17 Camera (imaging device) 18. Image Range 20 Shopping Cart 21 Handle 22 Caster 23 Cage (storage section) 30 items 32,33 area 35 Notification Content Table 36 Product identification results 37 Image Status 38 Notification Contents 41 Imaging control unit 42 Object detection unit 43 Loading / unloading judgment section 44 Product recognition department (recognition department) 45 Recognition score decline factor estimation unit (estimation unit) 46 Notification Department 47 Product Registration Department (Registration Department) 48 Payment Processing Unit 121 Control Program 122 Dictionary data for product recognition 123 Product Master 124 Product Registration File Ga,Gb Center of gravity position Ia, Ib, Ic, Id, Ie Images Na,Nb network Sc Recognition score TH Threshold [Prior art documents] [Patent documents]

[0121] [Patent Document 1] JP 2014-235530 A

Claims

1. an imaging control unit that controls at least one imaging device to capture a plurality of temporally consecutive images of an area of ​​the shopping cart that includes a storage unit into which products to be purchased are inserted; a recognition unit that recognizes the commodity placed in the container from the plurality of images captured by the imaging control unit; a registration unit that registers the product recognized by the recognition unit as a product to be purchased; an estimation unit that, when a recognition score indicating the accuracy of recognition by the recognition unit is equal to or lower than a threshold, estimates a cause of the recognition score being equal to or lower than the threshold from among a plurality of causes prepared in advance, based on a state of the image; a notification unit that notifies a customer of a message corresponding to the cause estimated by the estimation unit, thereby instructing the customer to re-insert the product. Product registration device.

2. the imaging device is attached to the shopping cart; The product registration device according to claim 1.

3. The recognition unit further recognizes the product taken out of the storage unit, The registration unit cancels the registration of the product that the recognition unit recognizes as having been removed from the storage unit. The product registration device according to claim 1.

4. The recognition unit recognizes a code symbol attached to the product, in which a product code of the product is registered. The product registration device according to any one of claims 1 to 3.

5. A computer that controls the product registration device an imaging control unit that controls at least one imaging device to capture a plurality of temporally consecutive images of an area of ​​the shopping cart that includes a storage unit into which products to be purchased are inserted; a recognition unit that recognizes the commodity placed in the container from the plurality of images captured by the imaging control unit; a registration unit that registers the product recognized by the recognition unit as a product to be purchased; an estimation unit that, when a recognition score indicating the accuracy of recognition by the recognition unit is equal to or lower than a threshold, estimates a cause of the recognition score being equal to or lower than the threshold from among a plurality of causes prepared in advance, based on a state of the image; The estimation unit functions as a notification unit that notifies the customer of a message corresponding to the cause estimated by the estimation unit, thereby instructing the customer to re-insert the product. program.

Images