Article registration device and program

Abstract

To provide an article registration device and a program that can reduce incorrect registration of articles.SOLUTION: An article registration device includes: first acquisition means for acquiring an image captured from an image capturing unit that captures an image of a storage in which an article is stored; second acquisition means for acquiring weight measurement results from a weight measuring unit provided at the bottom of the storage; identification means for comparing a characteristic quantity of the article represented in the captured image with a characteristic quantity of a plurality of articles stored in advance, and identifying the article having a similar characteristic quantity among the plurality of articles; registration means for registering the article identified by the identification means; and determination means for determining whether the difference between the weight of the article identified by the identification means and the change in weight measured before and after the identification is within a threshold value. When determined by the determination means to be outside the threshold value, the determination means identifies the article whose weight corresponds to the amount of change among the articles having the similarity with the characteristic amount of the article represented in the captured image.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] An embodiment of the present invention relates to a product registration device and a program. [Background technology]

[0002] A technology has been proposed in which an imaging device is provided in a shopping cart (hereinafter also referred to as a cart) for carrying products, products are recognized (identified) from captured images of the products put into the cart, and the identified products are registered as purchase targets. In this technology, products with similar features are identified based on feature values ​​that represent the external features of the products extracted from the captured images.

[0003] However, the above-mentioned technology may not be able to distinguish between products with similar appearances. For example, in the case of products with similar appearances but different volumes, similar feature values ​​may be acquired depending on the posture at the time of imaging, and a wrong product with a different volume may be registered. In addition, in the case of a bundle product sold in a form such as a set of six cans, feature values ​​similar to those of a product sold in single can units may be acquired depending on the posture at the time of imaging, and a wrong product with a different quantity may be registered. Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a product registration device and a program capable of reducing erroneous product registration. [Means for solving the problem]

[0005] The product registration device of the embodiment comprises a first acquisition means for acquiring an image from an imaging unit that images a storage unit in which products are stored, a second acquisition means for acquiring weight measurement results from a weight measuring unit provided at the bottom of the storage unit, an identification means for comparing features of the product shown in the image with features of a plurality of products stored in advance and identifying products having similar features from the plurality of products, a registration means for registering the products identified by the identification means, and a determination means for determining whether the difference between the weight of the product identified by the identification means and the change in weight measured before and after identification is within a threshold, and when the determination means determines that the difference is outside the threshold, the identification means identifies a product having a weight corresponding to the change from among products having similarity to the features of the product shown in the image. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of a system according to an embodiment. [Diagram 2] FIG. 2 is a side view showing an example of the configuration of a shopping cart according to an embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a hardware configuration of the tablet terminal according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a data configuration of the product master according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of a data configuration of a sales registration table according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a functional configuration of the tablet terminal according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a product according to the embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of processing performed by the tablet terminal according to the embodiment. [Figure 9] FIG. 9 is a side view showing an example of a configuration of a shopping cart according to the first modification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The product registration device and the program according to the present embodiment will be described in detail below with reference to the accompanying drawings. In the following embodiment, a terminal device (hereinafter also referred to as a cart terminal) attached to a shopping cart (hereinafter also referred to as a cart) for transporting products used in a store will be described as an example of the product registration device. Note that the invention is not limited to this application example.

[0008] Fig. 1 is a diagram showing an example of a store system 1 according to an embodiment. As shown in Fig. 1, the store system 1 includes a cart terminal 2, a server 4, and a payment machine 8. The server 4 and the payment machine 8 are communicatively connected via a network Na.

[0009] The network Na may be the Internet, a virtual private network (VPN), a local area network (LAN), a public communication network, a mobile communication network, or the like, either alone or in appropriate combination. As an example, the network Na may be a combination of the Internet and a mobile communication network.

[0010] In addition, a wireless access point 7 is connected to the network Na. The wireless access point 7 is connected to the cart terminal 2 via a network Nb such as a wireless LAN. Therefore, the cart terminal 2 can communicate with the server 4 and the payment machine 8.

[0011] The cart 3 has a storage section 31 (see FIG. 2) for storing products displayed in the store. The top of the storage section 31 is open, and products are placed into the storage section 31 from the top. A customer borrows the cart 3 at the store, and while moving around the store with the cart 3, stores the products he or she wishes to purchase from the products displayed in the storage section 31.

[0012] The cart terminal 2 is a terminal device operated by a customer using the cart 3, and is provided in the cart 3. The cart terminal 2 recognizes (hereinafter also referred to as "identification") the type of product inserted into the storage unit 31 from an image captured by the imaging unit 34 described later, and executes a sales registration process related to the product. Here, the sales registration process refers to a process of making the product identified by the cart terminal 2 a purchase target. Specifically, the sales registration process refers to a process of registering information about the identified product in the sales registration table 242 described later. For example, in the sales registration process, sales registration data including a product code capable of identifying the product type, etc., and the number of purchases of the product, etc., are registered in the sales registration table 242.

[0013] Furthermore, when the cart terminal 2 accepts an operation to perform a transaction on the payment machine 8, it executes a process to transfer the sales registration data for one transaction to the payment machine 8. For example, the cart terminal 2 encodes and displays the sales registration data as a code symbol such as a two-dimensional code. In this case, the payment machine 8 reads the code symbol displayed on the cart terminal 2 to obtain the sales registration data for one transaction registered on the cart terminal 2, and executes payment processing based on the sales registration data.

[0014] The server 4 uses the cart terminal 2 as a user interface terminal and performs information processing to provide a service for buying and selling products in the store where the user interface terminal is used. The server 4 may be, for example, a store server provided in the store, or a server device provided outside the store. Note that in this embodiment, the server 4 is realized by a single device, but it may also be a cloud server (or cloud system) realized by multiple devices connected to a network.

[0015] The server 4 is an information processing device configured as a computer equipped with a processor, a main storage device, an auxiliary storage device, etc. The server 4 stores a product master in which the names and prices of products sold in the store are registered in a database (not shown) or the like, and provides the product master to the cart terminal 2 and the payment machine 8.

[0016] The accounting machine 8 is an accounting device installed in a store. The accounting machine 8 executes payment processing for products based on sales registration data. The payment processing includes displaying the total price and tax amount of products related to one transaction, calculating and displaying change based on the deposit received from the customer, and issuing a receipt with product information and payment information (total price, deposit amount, change amount, etc.) printed on it.

[0017] Next, the cart 3 will be described. Fig. 2 is a side view showing an example of the configuration of the cart 3. The cart 3 includes a basket-shaped storage section 31 with an open top, rotating casters 32 arranged below the storage section 31, and a handle section 33 connected to the storage section 31 and casters 32 and held by the customer when moving the cart 3.

[0018] The storage section 31 has a generally rectangular side and bottom surrounded by a mesh-like lattice 311 and is open at the top, so that the products stored in the storage section 31 can be visually observed from the outside. The side and bottom may be formed of walls instead of the mesh-like lattice 311.

[0019] The cart 3 also includes a cart terminal 2 near the handle 33. The cart 3 also includes an imaging unit 34 near the edge of the opening of the storage unit 31. The imaging unit 34 is an example of an imaging unit, and is realized by an imaging device such as a digital camera. The imaging unit 34 is installed such that the imaging direction faces the opening of the storage unit 31. In other words, the imaging direction and imaging range of the imaging unit 34 are set so that it can capture images of products being put in and taken out of the storage unit 31.

[0020] The imaging unit 34 captures images of merchandise stored in the storage unit 31 or merchandise taken out of the storage unit 31, and outputs the captured images to the cart terminal 2. Here, the image means a still image or a video image in frame units.

[0021] Furthermore, a weight sensor 35 such as a weighing scale is provided at the bottom of the storage section 31. The weight sensor 35 is an example of a weight measuring section, and measures the weight of the product stored in the storage section 31.

[0022] The configuration of the cart 3 is not limited to the example shown in Fig. 2. For example, the cart 3 may include a battery that supplies power to the cart terminal 2, the imaging unit 34, the weight sensor 35, and the like.

[0023] Next, a description will be given of the hardware configuration of the cart terminal 2. Fig. 3 is a diagram showing an example of the hardware configuration of the cart terminal 2.

[0024] As shown in FIG. 3, the cart terminal 2 includes a CPU (Central Processing Unit) 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a storage unit 24, a display unit 25, an operation unit 26, a communication unit 27, and a connection unit 28.

[0025] The CPU 21 is an example of a processor, and controls the overall operation of the cart terminal 2. The ROM 22 stores various programs. The RAM 23 is a workspace in which programs and various data are developed.

[0026] The CPU 21, the ROM 22, and the RAM 23 are connected via a bus or the like to constitute a computer-configured control unit 200. In the control unit 200, the CPU 21 operates in accordance with a program stored in the ROM 22 or the storage unit 24 and loaded in the RAM 23, thereby executing various types of processing.

[0027] The storage unit 24 has a storage medium such as a HDD or a flash memory, and maintains the stored contents even when the power is cut off. The storage unit 24 stores programs that can be executed by the CPU 21 and various setting information. The storage unit 24 also stores a product master 241, a sales registration table 242, etc.

[0028] The product master 241 is a data table for managing products sold in a store. The sales registration table 242 is a data table for managing products that have been registered as sales using the cart terminal 2 of the store.

[0029] The display unit 25 is a display device such as an LCD (Liquid Crystal Display). The display unit 25 displays various information under the control of the CPU 21. The operation unit 26 is an input device such as a keyboard or a pointing device. The operation unit 26 outputs operation contents input via the input device to the CPU 21. The operation unit 26 may be a touch panel provided on the display unit 25.

[0030] The communication unit 27 is a wired or wireless communication interface connectable to the network Nb. The communication unit 27 is communicatively connected to the wireless access point 7 via the network Nb. The control unit 200 communicates with the server 4 and the payment machine 8 via the communication unit 27.

[0031] The connection unit 28 is an interface for connecting a peripheral device. The connection unit 28 is a wired or wireless interface conforming to standards such as USB (Universal Serial Bus) and Bluetooth (registered trademark). In this embodiment, the connection unit 28 is connected to the imaging unit 34 and the weight sensor 35.

[0032] FIG. 4 is a diagram showing an example of the data configuration of the product master 241. As shown in FIG. 4, the product master 241 stores product codes, product information, and feature amounts in association with each other. The product code is an example of identification information capable of identifying the type of product sold in a store. The product code may be, for example, a JAN code. The product information is information about a product corresponding to the product code. For example, the product name, price, weight, etc. are registered in the product information. Here, the weight is information indicating the weight of the product corresponding to the product code. Hereinafter, the weight registered in the product master 241 is also referred to as the theoretical weight.

[0033] The feature amount is information that quantitatively represents the external characteristics of the product corresponding to the product code. The feature amount is used as information for matching when recognizing the product from an image. Note that the feature amount may be a product image that represents the external appearance of the product.

[0034] It should be noted that the data configuration of the product master 241 is not limited to that shown in Fig. 4. For example, the feature amount may be associated with the product code of the product master 241 and registered in a database other than the product master 241.

[0035] In addition, in this embodiment, the cart terminal 2 is configured to hold the product master 241, but this is not limiting, and the server 4 may be configured to hold the product master 241. In this case, each cart terminal 2 can refer to the product master 241 held by the server 4 and handle the product master 241 in the same way as if it were held locally.

[0036] Fig. 5 is a diagram showing an example of the data configuration of the sales registration table 242. As shown in Fig. 5, the sales registration table 242 stores product codes and points in association with each other. The product code is the product code of the product registered as a sales item, and corresponds to the product code in the product master 241. The points is the registered number of points of the same product registered as a sales item. For example, the point number is registered as 1 by default.

[0037] The data structure of the sales registration table 242 is not limited to that shown in Fig. 5. For example, the sales registration table 242 may be configured to register product information corresponding to a product code in association with the product code. Also, the sales registration table 242 may be configured to register date and time information indicating the date and time of registration in association with the product code.

[0038] In addition, in this embodiment, the cart terminal 2 is configured to hold the sales registration table 242, but this is not limiting, and the server 4 may be configured to hold the sales registration table 242. In this case, the server 4 can manage the sales registration table 242 for each transaction by, for example, issuing a transaction ID that can identify each transaction performed at each cart terminal 2 and holding the sales registration table 242 in association with the transaction ID.

[0039] Specifically, the server 4 notifies the cart terminal 2 of the issued transaction ID, thereby causing the cart terminal 2 to retain the transaction ID while the transaction of that transaction ID is being carried out. The cart terminal 2 then transmits the product code of the product to be registered as a sales item to the server 4 along with the transaction ID, and instructs the server 4 to register the product in the sales registration table 242 corresponding to that transaction ID. This allows the server 4 to centrally manage products to be registered as sales items at each cart terminal 2. Note that the cart terminal 2 is capable of referencing the sales registration table 242 for transactions related to its own terminal.

[0040] Next, a description will be given of the functional configuration of the cart terminal 2. Fig. 6 is a diagram showing an example of the functional configuration of the cart terminal 2.

[0041] As shown in FIG. 6, the cart terminal 2 includes an image acquisition unit 201, a weight acquisition unit 202, a product identification unit 203, a registration processing unit 204, and a payment processing unit 205 as functional components.

[0042] The functional configuration of the cart terminal 2 is a software configuration realized by the processor of the cart terminal 2 operating in accordance with a program stored in a ROM or memory unit, but is not limited to this. For example, some or all of the functional configuration of the cart terminal 2 may be realized by hardware such as a dedicated circuit.

[0043] The image acquiring unit 201 is an example of a first acquiring means. The image acquiring unit 201 acquires an image captured by the imaging unit 34 via the connection unit 28. The weight acquiring unit 202 is an example of a second acquiring means. The weight acquiring unit 202 acquires a weight measured by a weight sensor via the connection unit 28.

[0044] The product identification unit 203 is an example of an identification means, a determination means, and a detection means. The product identification unit 203 recognizes a product captured by the imaging unit 34 based on an image acquired by the image acquisition unit 201. Specifically, the product identification unit 203 extracts a feature amount representing an external feature of the product from the image of the product by using a known object recognition technology or the like. Then, the product identification unit 203 identifies a product code of a product similar to the extracted feature amount from the product master 241 by comparing the extracted feature amount with the feature amount of each product registered in the product master 241. For example, the product identification unit 203 identifies a product code of a product having the highest similarity from the product master 241 among products whose similarity to the feature amount extracted from the image exceeds a threshold value.

[0045] However, the above-mentioned image-based product recognition method may not be able to distinguish between products with similar appearances. For example, as shown in Fig. 7, for products Ga and Gb that have similar appearances but different capacities, similar feature values ​​may be extracted depending on the posture at the time of imaging, and there is a possibility that a wrong product with a different capacity may be registered.

[0046] FIG. 7 is a diagram showing an example of a product. Product Ga and product Gb are examples of products with different volumes, and are products from the same series. Specifically, product Ga is a small-sized product, and product Gb is a large-sized product. In such products, some of the product packages may be similar. For example, product Ga and product Gb are similar in the area Pa indicated by the dashed line of the entire product package.

[0047] In such a case, the correct product can be recognized by capturing an image of the entire product package, but depending on the posture when capturing the image, the entire product package may not be captured. For example, if the customer's hand holding the product covers areas other than area Pa, the product package area Pa will be captured, and similar feature amounts may be extracted, making it impossible to distinguish between product Ga and product Gb. Specifically, product Ga may be recognized as product Gb even though it has been placed in (or removed from) the storage section 31, or product Gb may be erroneously recognized as product Ga even though it has been placed in (or removed from) the storage section 31.

[0048] Also, in the case of a bundled product sold in a form such as a set of six cans, depending on the position at which the image is captured, an image may be captured that closely resembles the product packaging of a product sold individually, which could result in products with different quantities being erroneously registered.

[0049] Therefore, the product identification unit 203 of this embodiment identifies the product put into the storage unit 31 using not only the recognition result based on the image but also the weight measured by the weight sensor 35. Specifically, when the product identification unit 203 recognizes the product put into the storage unit 31 based on the image, it calculates the difference (amount of change) in weight acquired by the weight acquisition unit 202 before and after the recognition, and derives the calculated amount of change as the weight of the product put into the storage unit 31.

[0050] For example, if the value obtained by subtracting the weight before recognition from the weight after recognition is positive, the product identification unit 203 derives this value as the weight of the product put into the storage unit 31. Also, if the value obtained by subtracting the weight before recognition from the weight after recognition is negative, the product identification unit 203 derives the absolute value of this value as the weight of the product taken out of the storage unit 31. Hereinafter, the weight of the product derived based on the measurement result of the weight sensor 35 is also referred to as the actual weight.

[0051] Next, the product identification unit 203 reads out the theoretical weight of the product recognized based on the image from the product master 241, and compares the read out theoretical weight with the derived actual weight of the product. If the weights match or the difference is within a threshold, the product identification unit 203 determines that the recognized product is correct, and identifies the product code of the product as the product imaged by the imaging unit 34.

[0052] On the other hand, if the weight difference falls outside the threshold, the product identification unit 203 determines that the product has been erroneously recognized. In this case, in order to exclude the erroneously recognized product, the product identification unit 203 identifies, from the product master 241, a product that corresponds to the product's actual weight among products that have similarity to the product feature amount extracted from the image. Specifically, the product identification unit 203 identifies, from the product master 241, a product code of a product that has a theoretical weight whose weight difference from the actual weight is within the threshold among products whose similarity to the feature amount extracted from the image exceeds the threshold.

[0053] When it is determined that a product has been removed from the storage unit 31, it is preferable to identify the product corresponding to the actual weight within the range of products registered in the sales registration table 242. Also, for example, the product identification unit 203 may be configured to check whether or not the product code of the product recognized from the image exists in the sales registration table 242, and if not, identify the product based on the actual weight.

[0054] In this way, when the product identification unit 203 recognizes a product based on the image captured by the imaging unit 34, it judges whether the recognized product is correct by comparing the theoretical weight of the product with the actual weight of the product measured by the weight sensor 35. As a result, even in cases where products with similar appearances are sold, such as products with different capacities or bundled products, the product identification unit 203 can more accurately identify which product the product imaged by the imaging unit 34 is, thereby reducing the occurrence of erroneous recognition.

[0055] In the above description, as the customer's operation on the product (hereinafter also referred to as product operation), it is determined whether the product is put into the storage unit 31 or taken out from the storage unit 31 based on the change in weight before and after recognition, but the method of determining the product operation is not limited to this. For example, the product identification unit 203 may detect the movement direction of the product (e.g., a motion vector) from temporally consecutive images to determine whether the identified product is put into the storage unit 31 or taken out from the storage unit 31.

[0056] Specifically, when the product moves from above to below (hereinafter also referred to as the insert direction), the product identification unit 203 determines that the product has been inserted into the storage unit 31. When the product moves from below to above (hereinafter also referred to as the remove direction), the product identification unit 203 determines that the product has been removed from the storage unit 31.

[0057] The registration processing unit 204 is an example of a registration means. The registration processing unit 204 executes a sales registration process based on the product code of the product identified by the product identification unit 203. For example, when the product identification unit 203 identifies a product and determines that the product has been inserted into the storage unit 31, the registration processing unit 204 registers the product code and the number of the identified product in the sales registration table 242.

[0058] Specifically, each time the registration processing unit 204 registers a product with the same product code, it adds 1 to the score of the product. In addition, when the registration processing unit 204 reads out the product name and price of the product corresponding to the registered product code from the product master 241, it displays them on the display unit 25 together with the score, thereby displaying a list of the products registered for sale in the order of registration.

[0059] In addition, for example, when the product identification unit 203 identifies a product and determines that the product has been removed from the storage unit 31, the registration processing unit 204 cancels the registration of the registered product by deleting the product code of the identified product from the sales registration table 242 and / or subtracting one from the registered number of items.

[0060] Specifically, when the registered points become 0 as a result of subtraction, the registration processing unit 204 deletes the product code of the product from the sales registration table 242. In addition, the registration processing unit 204 notifies the deregistered product by erasing the display of the product deleted from the sales registration table 242 from the products displayed in the list, or by subtracting the points.

[0061] In this way, in the store system 1, when a customer puts an item into the storage unit 31, the sales registration of that item is automatically performed. Also, in the store system 1, when a customer takes an item out of the storage unit 31, the registration of that item is automatically canceled.

[0062] The payment processing unit 205 executes processing to settle the sales-registered product in response to the customer's operation. Specifically, the payment processing unit 205 inputs data that can identify the sales-registered product into the payment machine 8, thereby realizing a state in which the payment can be made at the payment machine 8.

[0063] For example, when the payment processing unit 205 receives an operation to instruct the end of sales registration or an operation to instruct the start of payment (hereinafter collectively referred to as a closing operation) via the operation unit 26, it encodes the sales registration data registered in the sales registration table 242 into a code symbol such as a two-dimensional code and displays it on the display unit 25. In this case, the payment machine 8 obtains the sales registration data by reading the code symbol displayed on the display unit 25 of the cart terminal 2, and performs payment processing for the products for one transaction that have been registered as sales based on the obtained sales registration data.

[0064] Also, for example, the payment processing unit 205 may transmit the sales registration data registered in the sales registration table 242 to the payment machine 8 by short-range wireless communication such as Bluetooth (registered trademark). In this case, the payment machine 8 acquires the sales registration data transmitted from the cart terminal 2, and executes the payment process for the products for one transaction that have been registered as sales, based on the acquired sales registration data.

[0065] Also, for example, in a configuration in which the sales registration table 242 is stored in the server 4, the payment processing unit 205 encodes the transaction ID of the transaction into a code symbol such as a barcode or two-dimensional code and displays it on the display unit 25. In this case, the payment machine 8 obtains the transaction ID by reading the code symbol displayed on the display unit 25 of the cart terminal 2. The payment machine 8 then obtains sales registration data related to the obtained transaction ID from the server 4, and performs payment processing for the products for one transaction that have been registered as sales, based on the obtained sales registration data.

[0066] The payment for the products is not limited to the payment method using the cashier 8, but may be made at the cart terminal 2. For example, when the payment processing unit 205 receives an instruction to start payment via the operation unit 26, the payment processing unit 205 displays a screen on the display unit 25 that allows the user to select a payment method such as electronic money payment or credit card payment. When the payment method is selected, the payment processing unit 205 receives input of payment information for making the payment using the selected payment method, and executes a payment process to settle the total amount of the products registered as sales based on the input payment information by cooperating with an external payment server. Such payment information may be, for example, a credit card number or a two-dimensional code for code payment. In the case of a two-dimensional code for code payment, the input may be received via the imaging unit 34.

[0067] An example of the operation of the cart terminal 2 will be described below with reference to Fig. 8. Fig. 8 is a flowchart showing an example of processing performed by the cart terminal 2. Note that this processing is premised on the assumption that imaging by the imaging unit 34 and weight measurement by the weight sensor 35 are performed in the background.

[0068] First, when an instruction to start a transaction is given through an operation via the operation unit 26, the control unit 200 of the cart terminal 2 initializes the sales registration table 242 (step S11).

[0069] In addition, when the sales registration table 242 is configured to be held by the server 4, for example, the control unit 200 requests the server 4 to issue a transaction ID, thereby causing the server 4 to hold the sales registration table 242 related to the transaction ID of the transaction. In addition, the control unit 200 obtains the issued transaction ID from the server 4 and holds it in the RAM 23 or the like.

[0070] Next, the image acquisition unit 201 acquires an image from the imaging unit 34 (step S12). Next, the product identification unit 203 determines whether or not the product has been identified from the image acquired in step S12 (step S13). If the product cannot be identified (step S13; No), the process proceeds to step S23.

[0071] When the product identification unit 203 identifies a product from the image acquired in step S12 (step S13; Yes), it reads out the theoretical weight of the product from the product master 241 (step S14). Next, the product identification unit 203 derives the actual weight of the product based on the measurement result of the weight sensor 35 acquired by the weight acquisition unit 202 (step S15).

[0072] Next, the product identification unit 203 determines the customer's operation on the product identified in step S13 based on the measurement result of the weight sensor 35 acquired by the weight acquisition unit 202 or the movement direction of the product shown in the image (step S16). Specifically, the product identification unit 203 determines whether the product identified in step S13 has been put into the storage unit 31 or taken out from the storage unit 31.

[0073] Next, the product identification unit 203 compares the theoretical weight read in step S14 with the actual weight derived in step S15 (step S17) and determines whether the difference in weight is within the threshold (step S18). If the difference in weight is within the threshold (step S18; Yes), the product identification unit 203 proceeds to step S20 while maintaining the product code of the product identified in step S13.

[0074] Furthermore, if the weight difference is outside the threshold (step S18; No), the product identification unit 203 identifies from the product master 241, among the products having similarity to the feature amount extracted from the image, the product corresponding to the actual weight derived in step S15 (step S19), and proceeds to step S20. Note that, if it is determined in step S16 that the product has been removed from the storage unit 31, the product identification unit 203 may identify the product corresponding to the actual weight derived in step S15 from the products registered in the sales registration table 242.

[0075] Next, the registration processing unit 204 judges whether the product operation determined in step S16 is an insert operation indicating inserting into the storage unit 31 (step S20). If the product operation is an insert operation (step S20; Yes), the registration processing unit 204 registers the product identified by the product identification unit 203 in the sales registration table 242 (step S21) and proceeds to step S23. If the product operation is a removal operation (step S20; No), the registration processing unit 204 deletes the product identified by the product identification unit 203 from the sales registration table 242 (step S22) and proceeds to step S23.

[0076] Next, the registration processing unit 204 (or the payment processing unit 205) determines whether or not a closing operation has been accepted (step S23). If the closing operation has not been accepted (step S23; No), the registration processing unit 204 returns to step S12 and continues the process. On the other hand, if the closing operation has been accepted (step S23; Yes), the payment processing unit 205 executes the payment process based on the product information registered in the sales registration table 242 (step S24), and ends this process.

[0077] As described above, when an image of a product is captured by the imaging unit 34, the cart terminal 2 compares the feature amount of the product shown in the image with the feature amounts of multiple products stored in the product master 241, and identifies products with similar feature amounts from the product master 241. The cart terminal 2 also compares the theoretical weight and actual weight of the identified product, and determines whether the difference is within a threshold, thereby judging whether the product identified from the image is correct. If the weight difference is outside the threshold, the cart terminal 2 determines that there is an error in the product identified from the image, and identifies a product corresponding to the actual weight from among products similar to the product shown in the image, and executes a sales registration process.

[0078] As a result, even if the cart terminal 2 erroneously recognizes a product that has a very similar appearance, it can correct the erroneously recognized product and execute the sales registration process, thereby reducing erroneous registration of products.

[0079] The above-described embodiment can be modified as appropriate by changing a part of the configuration or function of each of the above-described devices. Therefore, some modified examples of the above-described embodiment will be described below as other embodiments. Note that the following mainly describes the differences from the above-described embodiment, and a detailed description of the common points with the contents already described will be omitted. The modified examples described below may be implemented individually or in appropriate combination.

[0080] (Variation 1) In the above embodiment, the number of imaging units 34 attached to the cart 3 is one, but this is not limiting and a configuration in which a plurality of imaging units are provided may be used. For example, as shown in Fig. 9, another imaging unit 341 may be provided at a position different from that of the imaging unit 34. Here, Fig. 9 is a side view showing an example of the overall configuration of the cart 3 according to the first modification.

[0081] The imaging direction of the imaging unit 341 is directed toward the opening of the storage unit 31, similar to the imaging unit 34, and the imaging unit 341 is installed so that at least a part of the imaging range includes the inside of the storage unit 31. The imaging unit 341 is also connected to the cart terminal 2, and an image obtained by imaging is output to the cart terminal 2.

[0082] In this case, the image acquisition unit 201 of the cart terminal 2 acquires images captured by each of the imaging unit 34 and the imaging unit 341. The product identification unit 203 recognizes and identifies the product from the images captured by each of the imaging unit 34 and the imaging unit 341. The registration processing unit 204 registers the sales of the product identified by the product identification unit 203.

[0083] In order to prevent multiple registration of a product, the registration processing unit 204 performs sales registration of the product when a common product is identified from the images of the imaging unit 34 and the imaging unit 341, or when a product is registered from an image captured by either the imaging unit 34 or the imaging unit 341. In addition, when different product codes are identified from the images of the imaging unit 34 and the imaging unit 341, the registration processing unit 204 does not perform sales registration, and causes the display unit 25 to display a message informing of an error.

[0084] (Variation 2) In the above embodiment, the form in which products are stored in the storage section 31 of the cart 3 has been described, but the present invention is not limited to this, and products may be stored in other forms of storage section 31. For example, a shopping basket used to carry products in a store can be an example of the storage section 31. In this case, the imaging section 34 is installed near the edge of the opening of the shopping basket, etc., and is installed so that at least a part of the imaging range includes the inside of the shopping basket. In addition, the weight sensor 35 is installed at the bottom of the shopping basket, or on the outside of the bottom.

[0085] In this case, the terminal device may be a mobile terminal such as a smartphone having the same functions as the cart terminal 2, instead of the cart terminal 2. Furthermore, the shopping basket may be configured to be carried by the customer or moved while placed in the storage section 31 of the cart 3.

[0086] (Variation 3) In the above-described embodiment, the cart terminal 2 recognizes (and identifies) the product from the image captured by the imaging unit 34, but this is not limiting and an external device such as the server 4 may perform this. In this case, for example, the cart terminal 2 may transmit the image acquired by the image acquisition unit 201 and the weight acquired by the weight acquisition unit 202 to the server 4, and the server 4 may include one or both of the product identification unit 203 and the registration processing unit 204. The server 4 may then recognize (and identify) the product and perform sales registration processing based on the image transmitted from the cart terminal 2.

[0087] The programs executed by each device in the above-described embodiments (and the modified examples) are provided in a state where they are pre-installed in a ROM, a storage unit, etc. The programs executed by each device in the above-described embodiments may be provided by being recorded in a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD) in the form of an installable or executable file.

[0088] Furthermore, the programs executed by each device of the above-described embodiments may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the programs executed by each device of the above-described embodiments may be provided or distributed via a network such as the Internet.

[0089] Although the embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. These new embodiments and their modifications can be implemented in various other forms, and various omissions, substitutions, changes, and combinations can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0090] 1 Store System 2. Tablet devices 3. Shopping Cart 4 Server 8. Cashier 200 Control section 201 Image Acquisition Unit 202 Weight acquisition part 203 Product Specification Department 204 Registration processing section 205 Payment Processing Unit 241 Product Master 242 Sales registration table [Prior art documents] [Patent documents]

[0091] [Patent Document 1] JP 2015-35077 A

Claims

1. a first acquisition means for acquiring an image from an imaging unit that images a storage unit in which the products are stored; A second acquisition means for acquiring a weight measurement result from a weight measurement unit provided at a bottom of the storage unit; A specifying means for comparing a feature amount of a product shown in the captured image with feature amounts of a plurality of products stored in advance, and specifying a product having a similar feature amount from among the plurality of products; a registration means for registering the product identified by the identification means; a determination means for determining whether a difference between the weight of the product identified by the identification means and the amount of change in weight measured before and after the identification is within a threshold value; Equipped with The identification means, when the determination means determines that the change is outside the threshold, identifies a product having a weight corresponding to the amount of change from among products having similarity to the features of the product shown in the captured image.

2. the registration means registers the product identified by the identification means when the product is stored in the storage section; The product registration device according to claim 1.

3. the registration means cancels the registration of the product identified by the identification means when it is detected that the product has been removed from the storage unit. The product registration device according to claim 2.

4. The apparatus further includes a determination unit that determines whether a product has been stored in the storage unit or removed from the storage unit based on an increase or decrease in weight measured before and after the identification by the identification unit. The product registration device according to claim 2 or 3.

5. The image capturing apparatus further includes a determination unit that determines whether a product has been stored in the storage unit or removed from the storage unit by detecting a moving direction of the product from the captured images that are successive in time. The product registration device according to claim 2 or 3.

6. The computer of the product registration device, a first acquisition means for acquiring an image from an imaging unit that images a storage unit in which the products are stored; A second acquisition means for acquiring a weight measurement result from a weight measurement unit provided at a bottom of the storage unit; A specifying means for comparing a feature amount of a product shown in the captured image with feature amounts of a plurality of products stored in advance, and specifying a product having a similar feature amount from among the plurality of products; a registration means for registering the product identified by the identification means; a determination means for determining whether a difference between the weight of the product identified by the identification means and the amount of change in weight measured before and after the identification is within a threshold value; Equipped with The identification means is a program that, when the judgment means judges that the change is outside the threshold, identifies a product having a weight corresponding to the amount of change from among products that have similarity to the features of the product shown in the captured image.

Images