Product registration device and program

Abstract

To provide an article registration device and a program that can reduce incorrect registration of articles.SOLUTION: An article registration device includes: first acquisition means for acquiring an image captured from an image capturing unit that captures an image of a storage in which an article is stored; second acquisition means for acquiring weight measurement results from a weight measuring unit provided at the bottom of the storage; identification means for comparing a characteristic quantity of the article represented in the captured image with a characteristic quantity of a plurality of articles stored in advance, and identifying the article having a similar characteristic quantity among the plurality of articles; registration means for registering the article identified by the identification means; and determination means for determining whether the difference between the weight of the article identified by the identification means and the change in weight measured before and after the identification is within a threshold value. When determined by the determination means to be outside the threshold value, the determination means identifies the article whose weight corresponds to the amount of change among the articles having the similarity with the characteristic amount of the article represented in the captured image.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] Embodiments of the present invention relate to a product registration device and a program.

Background Art

[0002] Conventionally, a technique has been proposed in which an imaging device is provided on a shopping cart (hereinafter also referred to as a cart) for carrying products, products are recognized (identified) from captured images of the products put into the cart, and the identified products are registered as purchase targets. In such a technique, products with similar feature amounts are identified based on the feature amounts representing the appearance features of the products extracted from the captured images.

[0003] By the way, in the above-described technique, there is a possibility that products with very similar appearances cannot be distinguished. For example, in the case of products with different capacities but very similar appearances, similar feature amounts may be obtained depending on the posture at the time of imaging, and there is a possibility that incorrect products with different capacities may be registered. Also, for example, in the case of bundled products sold in a form such as a 6-can set, depending on the posture at the time of imaging, feature amounts very similar to those of products sold in a single-can unit may be obtained, and there is a possibility that incorrect products with different quantities may be registered.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to provide a product registration device and a program capable of reducing incorrect registration of products.

Means for Solving the Problems

[0005] The product registration device according to the embodiment is Products stored in the storage compartment a first acquisition means for acquiring a captured image from an imaging unit that captures The weight of the goods stored in the storage compartment is measured. a second acquisition means for acquiring a measurement result of weight from a weight measurement unit provided at the bottom of the storage unit, and comparing the feature amount of the product represented in the captured image with the feature amounts of a plurality of products stored in advance, and selecting from among the plurality of products The similarity between the captured image and the product shown exceeds the threshold, and the similarity is the highest.A means for identifying a product, and a means for registering the product identified by the identification means, Of the weights of the aforementioned multiple products stored in advance The weight of the product identified by the aforementioned identification means, The aforementioned The system includes a determination means for determining whether the difference between the weight change amount measured by the weight measurement unit and the actual weight change amount is within a threshold, and if the determination means determines that the difference is outside the threshold, the identification means determines that the product represented in the captured image is within a threshold. The similarity to the above exceeded the threshold. Identify the products whose weight corresponds to the aforementioned change in weight. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 shows an example of a system according to an embodiment. [Figure 2] Figure 2 is a side view showing an example of the configuration of a shopping cart according to this embodiment. [Figure 3] Figure 3 shows an example of the hardware configuration of a tablet terminal according to the embodiment. [Figure 4] Figure 4 shows an example of the data structure of a product master according to an embodiment. [Figure 5] Figure 5 shows an example of the data structure of the sales registration table according to the embodiment. [Figure 6] Figure 6 shows an example of the functional configuration of a tablet terminal according to the present invention. [Figure 7] Figure 7 shows an example of a product according to this embodiment. [Figure 8] Figure 8 is a flowchart showing an example of the processing performed by the tablet terminal of the embodiment. [Figure 9] Figure 9 is a side view showing an example of the configuration of a shopping cart according to Modification 1. [Modes for carrying out the invention]

[0007] The product registration device and program according to this embodiment will be described in detail below with reference to the attached drawings. In the following embodiment, a terminal device (hereinafter also referred to as a cart terminal) attached to a shopping cart (hereinafter also referred to as a cart) used for transporting goods in a store will be described as an example of a product registration device. However, the invention is not limited by this application example.

[0008] Figure 1 shows an example of a store system 1 according to an embodiment. As shown in Figure 1, the store system 1 comprises a cart terminal 2, a server 4, and a cashier 8. The server 4 and the cashier 8 are connected to each other via a network Na.

[0009] Network Na can be the internet, VPN (virtual private network), LAN (local area network), public communication network, mobile communication network, etc., either individually or in appropriate combinations. As an example, Network Na can be a combination of the internet and a mobile communication network.

[0010] Furthermore, a wireless access point 7 is connected to network Na. The wireless access point 7 is connected to the cart terminal 2 via network Nb, such as a wireless LAN. As a result, the cart terminal 2 can communicate with the server 4 and the accounting machine 8.

[0011] Cart 3 is equipped with a storage compartment 31 (see Figure 2) for storing products displayed in the store. The storage compartment 31 is open at the top, and products are placed into the storage compartment 31 from the top. Customers borrow a cart 3 from the store and, while moving around the store with the cart 3, place the products they wish to purchase from the displayed items into the storage compartment 31.

[0012] Cart terminal 2 is a terminal device operated by customers using cart 3 and is installed on cart 3. Cart terminal 2 recognizes (hereinafter also referred to as "identification") the type of product to be placed in storage unit 31 from the image captured by imaging unit 34 (described later) and executes sales registration processing for that product. Here, sales registration processing means the process of making the product identified by cart terminal 2 a target for purchase. Specifically, sales registration processing refers to the process of registering information about the identified product in sales registration table 242 (described later). For example, in sales registration processing, sales registration data including a product code that can identify the product item, the number of items purchased, etc., is registered in sales registration table 242.

[0013] Furthermore, when the cart terminal 2 receives an operation to perform accounting on the accounting machine 8, it executes a process to transfer the sales registration data for one transaction to the accounting machine 8. For example, the cart terminal 2 encodes the sales registration data into a code symbol such as a two-dimensional code and displays it. In this case, the accounting machine 8 reads the code symbol displayed on the cart terminal 2 to obtain the sales registration data for one transaction registered on the cart terminal 2 and performs accounting processing based on that sales registration data.

[0014] Server 4 uses the cart terminal 2 as a user interface terminal and performs information processing to provide a service for buying and selling goods at the store where the user interface terminal is used. Server 4 may be, for example, a store server installed in the store, or a server device installed outside the store. In this embodiment, Server 4 is implemented by a single device, but it may also be a cloud server (or cloud system) implemented by multiple network-connected devices.

[0015] Server 4 is an information processing device with a computer configuration that includes a processor, main memory, auxiliary memory, etc. Server 4 stores, for example, a product master containing the names and prices of products sold in stores in a database (not shown) and provides it to the shopping cart terminal 2 and the checkout machine 8.

[0016] The accounting machine 8 is an accounting device provided inside the store. The accounting machine 8 executes the settlement process for goods based on the sales registration data. The settlement process refers to processes such as displaying the total amount and tax amount of goods related to one transaction, calculating and displaying the change based on the deposit received from the customer, and issuing a receipt printed with product information and settlement information (total amount, deposit amount, change amount, etc.).

[0017] Next, the cart 3 will be described. FIG. 2 is a side view showing an example of the configuration of the cart 3. The cart 3 includes a basket-shaped storage part 31 with an open upper surface, a rotating caster 32 arranged below the storage part 31, a handle part 33 that is connected to the storage part 31 and the caster 32 and is grasped by the customer when moving the cart 3, and the like.

[0018] The storage part 31 has a configuration where the side surface and the bottom surface are substantially surrounded in a square shape by a mesh-like lattice 311 and the upper part is open, and the goods stored in the storage part 31 can be visually observed from the outside. Note that the side surface and the bottom surface may be constituted by a wall surface instead of the mesh-like lattice 311.

[0019] Also, the cart 3 includes a cart terminal 2 near the handle part 33. Further, the cart 3 includes an imaging part 34 near the edge of the opening of the storage part 31. The imaging part 34 is an example of an imaging part and is realized by an imaging device such as a digital camera. The imaging part 34 is installed with the imaging direction facing the opening of the storage part 31. That is, the imaging direction and the imaging range of the imaging part 34 are set so that the goods put into and taken out of the storage part 31 can be imaged.

[0020] The imaging part 34 images the goods stored in the storage part 31 or the goods taken out from the storage part 31, and outputs the captured image to the cart terminal 2. Here, the image means a still image or a moving image in frame units.

[0021] Also, a weight sensor 35 such as a weighing scale is provided at the bottom of the storage part 31. The weight sensor 35 is an example of a weight measuring part and measures the weight of the goods stored in the storage part 31.

[0022] Note that the configuration of cart 3 is not limited to the example shown in Figure 2. For example, cart 3 may be equipped with a battery that supplies power to the cart terminal 2, imaging unit 34, weight sensor 35, etc.

[0023] Next, we will describe the hardware configuration of cart terminal 2. Figure 3 shows an example of the hardware configuration of cart terminal 2.

[0024] As shown in Figure 3, the cart terminal 2 includes a CPU (Central Processing Unit) 21, ROM (Read Only Memory) 22, RAM (Random Access Memory) 23, storage unit 24, display unit 25, operation unit 26, communication unit 27, and connection unit 28, etc.

[0025] The CPU 21 is an example of a processor that comprehensively controls the operation of the cart terminal 2. The ROM 22 stores various programs. The RAM 23 is a workspace for displaying programs and various data.

[0026] The CPU 21, ROM 22, and RAM 23 are connected via a bus or the like to form a control unit 200 of the computer configuration. In the control unit 200, the CPU 21 operates according to a program stored in the ROM 22 or memory unit 24 and loaded into the RAM 23, thereby executing various processes.

[0027] The storage unit 24 has a storage medium such as an HDD or flash memory and retains its contents even when the power is cut off. The storage unit 24 stores programs that the CPU 21 can execute and various setting information. The storage unit 24 also stores the product master 241, the sales registration table 242, etc.

[0028] The product master table 241 is a data table for managing products sold in stores. The sales registration table 242 is a data table for managing products registered for sale using the user's own shopping cart terminal 2.

[0029] The display unit 25 is a display device such as an LCD (Liquid Crystal Display). The display unit 25 displays various information under the control of the CPU 21. The operation unit 26 is an input device such as a keyboard or a pointing device. The operation unit 26 outputs the operation content entered via the input device to the CPU 21. The operation unit 26 may also be a touch panel provided on the display unit 25.

[0030] The communication unit 27 is a wired or wireless communication interface that can connect to the network Nb. The communication unit 27 is connected to the wireless access point 7 via the network Nb. The control unit 200 communicates with the server 4 and the accounting machine 8 via the communication unit 27.

[0031] The connection unit 28 is an interface for connecting peripheral devices. The connection unit 28 is a wired or wireless interface compliant with standards such as USB (Universal Serial Bus) or Bluetooth (registered trademark). In this embodiment, the connection unit 28 is connected to the imaging unit 34 and the weight sensor 35.

[0032] Figure 4 shows an example of the data structure of the product master 241. As shown in Figure 4, the product master 241 stores product codes, product information, and feature quantities in association with each other. The product code is an example of identification information that can identify the type of product sold in a store. For example, a JAN code can be used for the product code. The product information is information about the product corresponding to the product code. For example, the product name, price, weight, etc. are registered in the product information. Here, the weight is information indicating the weight of the product corresponding to the product code. Hereafter, the weight registered in the product master 241 will also be called the theoretical weight.

[0033] Feature quantities are quantitative information that represents the external characteristics of the product corresponding to the product code. Feature quantities are used as matching information when recognizing a product from an image. Note that feature quantities may also be product images or other representations of the product's appearance.

[0034] Note that the data structure of the product master 241 is not limited to that shown in Figure 4. For example, the features may be associated with the product codes in the product master 241 and registered in a different database from the product master 241.

[0035] Furthermore, in this embodiment, the cart terminal 2 is configured to hold the product master 241, but the configuration is not limited to this, and the server 4 may also hold the product master 241. In this case, each of the cart terminals 2 can handle the product master 241 in the same way as when it is held locally by referring to the product master 241 held by the server 4.

[0036] Figure 5 shows an example of the data structure of the sales registration table 242. As shown in Figure 5, the sales registration table 242 stores the product code and the quantity in association. The product code is the product code of the product that was registered for sale and corresponds to the product code in the product master 241. The quantity is the number of registered items for the same product that was registered for sale. For example, the quantity is registered as 1 by default.

[0037] Note that the data structure of the sales registration table 242 is not limited to that shown in Figure 5. For example, it may be configured to register product information corresponding to a product code, or to register date and time information indicating the registration date and time, associated with the product code.

[0038] Furthermore, in this embodiment, the cart terminal 2 is configured to hold the sales registration table 242, but the configuration is not limited to this, and the server 4 may also hold the sales registration table 242. In this case, the server 4 can manage the sales registration table 242 for each transaction by, for example, issuing a transaction ID that can identify each transaction performed by each cart terminal 2, and holding the sales registration table 242 in association with the transaction ID.

[0039] Specifically, Server 4 notifies Cart Terminal 2 of the issued transaction ID, causing Cart Terminal 2 to retain the transaction ID while the transaction for that transaction ID is being processed. Then, Cart Terminal 2 sends the product code of the product to be registered for sale, along with the transaction ID, to Server 4, instructing Server 4 to register it in the sales registration table 242 corresponding to that transaction ID. This allows Server 4 to centrally manage the products registered for sale at each Cart Terminal 2. It should be noted that Cart Terminal 2 is capable of accessing the sales registration table 242 for transactions related to its own terminal.

[0040] Next, we will explain the functional configuration of the shopping cart terminal 2. Figure 6 shows an example of the functional configuration of the shopping cart terminal 2.

[0041] As shown in Figure 6, the cart terminal 2 is functionally configured to include an image acquisition unit 201, a weight acquisition unit 202, a product identification unit 203, a registration processing unit 204, and a payment processing unit 205.

[0042] The functional configuration of the cart terminal 2 is a software configuration realized by the cart terminal 2's processor operating according to a program stored in ROM or memory, but is not limited to this. For example, some or all of the functional configuration of the cart terminal 2 may be realized by hardware such as dedicated circuits.

[0043] The image acquisition unit 201 is an example of a first acquisition means. The image acquisition unit 201 acquires images captured by the imaging unit 34 via the connection unit 28. The weight acquisition unit 202 is an example of a second acquisition means. The weight acquisition unit 202 acquires the weight measured by the weight sensor via the connection unit 28.

[0044] The product identification unit 203 is an example of an identification means, a determination means, and a detection means. The product identification unit 203 recognizes a product captured by the imaging unit 34 based on the image acquired by the image acquisition unit 201. Specifically, the product identification unit 203 extracts feature quantities representing the external characteristics of the product from the image of the product, using known object recognition technology or the like. The product identification unit 203 then compares these feature quantities with those of each product registered in the product master 241 to identify the product code of the product that is similar to the extracted feature quantities from the product master 241. For example, the product identification unit 203 identifies the product code of the product with the highest similarity among the products whose similarity to the feature quantities extracted from the image exceeds a threshold, from the product master 241.

[0045] However, the image-based product recognition method described above may not be able to distinguish between products that look very similar. For example, as shown in Figure 7, for products Ga and Gb, which look very similar but have different capacities, similar feature quantities may be extracted depending on the orientation during imaging, potentially leading to the registration of incorrect products with different capacities.

[0046] Figure 7 shows an example of a product. Products Ga and Gb are examples of products with different capacities and belong to the same series. Specifically, product Ga is the small size, and product Gb is the large size. In such products, parts of the product packaging may be similar. For example, the area Pa shown by the dashed line is similar in the overall product packaging of products Ga and Gb.

[0047] In such cases, capturing the entire product package allows for the recognition of the correct product. However, depending on the posture during imaging, the entire product package may not be captured. For example, if the customer's hand holding the product covers an area other than area Pa, only the area Pa of the product package will be captured. This may result in the extraction of very similar features, making it impossible to distinguish between product Ga and product Gb. Specifically, product Ga may be mistakenly identified as product Gb even though it has been placed in (or removed from) the storage unit 31, or product Gb may be mistakenly identified as product Ga even though it has been placed in (or removed from) the storage unit 31.

[0048] Furthermore, in the case of bundled products sold in sets of six cans, for example, depending on the camera's orientation during imaging, images very similar to the product packaging of individually sold products may be captured, potentially leading to the incorrect registration of products with different quantities.

[0049] Therefore, the product identification unit 203 of this embodiment identifies the product placed in the storage unit 31 not only based on the recognition result based on the image, but also using the weight measured by the weight sensor 35. Specifically, when the product identification unit 203 recognizes the product placed in the storage unit 31 based on the image, it calculates the difference (change) in weight obtained by the weight acquisition unit 202 before and after the recognition, and derives the calculated change as the weight of the product placed in the storage unit 31.

[0050] For example, if the product identification unit 203 finds that the difference between the weight after recognition and the weight before recognition is positive, the product identification unit 203 derives that value as the weight of the product placed in the storage unit 31. Also, for example, if the difference between the weight after recognition and the weight before recognition is negative, the product identification unit 203 derives the absolute value of that value as the weight of the product removed from the storage unit 31. Hereinafter, the weight of the product derived based on the measurement results of the weight sensor 35 will also be referred to as the actual weight.

[0051] Next, the product identification unit 203 reads the theoretical weight of the product recognized based on the image from the product master 241 and compares the read theoretical weight with the actual weight of the product derived. If the weights match or the difference falls within a threshold, the product identification unit 203 determines that the recognized product is correct and identifies the product code of that product as the product captured by the imaging unit 34.

[0052] On the other hand, if the weight difference falls outside the threshold, the product identification unit 203 determines that the product has been misidentified. In this case, in order to exclude the misidentified product, the product identification unit 203 identifies products from the product master 241 that correspond to the actual weight of the product among products that have similarity to the product features extracted from the image. Specifically, the product identification unit 203 identifies the product codes from the product master 241 of products whose similarity to the features extracted from the image exceeds the threshold, and whose theoretical weight has a weight difference from the actual weight that falls within the threshold.

[0053] Furthermore, when it is determined that a product has been removed from the storage unit 31, it is preferable to identify the product corresponding to the actual weight within the range of products registered in the sales registration table 242. Alternatively, for example, the product identification unit 203 may be configured to check whether the product code of the product recognized from the image exists in the sales registration table 242, and if it does not exist, to identify the product based on the actual weight.

[0054] In this way, when the product identification unit 203 recognizes a product based on the image captured by the imaging unit 34, it compares the theoretical weight of the product with the actual weight of the product measured by the weight sensor 35 to determine whether the recognized product is correct. As a result, even when products with similar appearances, such as products with different capacities or bundled products, are sold, the product identification unit 203 can more accurately identify which product is being photographed by the imaging unit 34, thereby reducing the occurrence of misrecognition.

[0055] In the above explanation, the customer's operation on the product (hereinafter also referred to as product operation) was determined based on the change in weight before and after recognition to determine whether the product was placed in the storage unit 31 or removed from the storage unit 31. However, the method for determining product operation is not limited to this. For example, the product identification unit 203 may determine whether the identified product was placed in the storage unit 31 or removed from the storage unit 31 by detecting the direction of movement of the product (e.g., motion vector) from a series of images in time.

[0056] Specifically, if the direction of movement of the product is from top to bottom (hereinafter also referred to as the input direction), the product identification unit 203 determines that the product has been placed into the storage unit 31. Also, if the direction of movement of the product is from bottom to top (hereinafter also referred to as the retrieval direction), the product identification unit 203 determines that the product has been removed from the storage unit 31.

[0057] The registration processing unit 204 is an example of a registration means. The registration processing unit 204 performs sales registration processing based on the product code of the product identified by the product identification unit 203. For example, if the product identification unit 203 identifies a product and it is determined that the product has been placed in the storage unit 31, the registration processing unit 204 registers the product code and quantity of the identified product in the sales registration table 242.

[0058] Specifically, the registration processing unit 204 increments the score of a product by 1 each time a product with the same product code is registered. Furthermore, when the registration processing unit 204 reads the product name and price of the product corresponding to the registered product code from the product master 241, it displays them along with the score on the display unit 25, thereby displaying a list of the registered products in the order they were registered.

[0059] Furthermore, for example, if the product identification unit 203 identifies a product and it is determined that the product has been removed from the storage unit 31, the registration processing unit 204 will deregister the product by deleting the product code of the identified product from the sales registration table 242 or by subtracting 1 from the registered quantity.

[0060] Specifically, if the registration processing unit 204 determines that the number of registered items becomes 0 due to a reduction in the number of registered items, it deletes the product code of that item from the sales registration table 242. In addition, the registration processing unit 204 notifies the user of the deregistered item by either removing the item from the list of products displayed in the sales registration table 242 or by subtracting the number of items.

[0061] Thus, in store system 1, when a customer places an item into storage unit 31, the sale of that item is automatically registered. Furthermore, in store system 1, when a customer removes an item from storage unit 31, the registration of that item is automatically canceled.

[0062] The payment processing unit 205 executes processing to settle the payment for the registered sale in response to customer operations. Specifically, the payment processing unit 205 inputs data that can identify the registered sale in the accounting machine 8, thereby making the accounting machine 8 ready for payment.

[0063] For example, when the settlement processing unit 205 receives an operation via the operation unit 26 to instruct the end of sales registration or to instruct the start of settlement (hereinafter collectively referred to as a closing operation), it encodes the sales registration data registered in the sales registration table 242 into a code symbol such as a two-dimensional code and displays it on the display unit 25. In this case, the accounting machine 8 reads the code symbol displayed on the display unit 25 of the cart terminal 2 to obtain the sales registration data and performs accounting processing for the goods of one transaction that has been registered for sale based on the obtained sales registration data.

[0064] Alternatively, for example, the payment processing unit 205 may transmit the sales registration data registered in the sales registration table 242 to the accounting machine 8 via short-range wireless communication such as Bluetooth®. In this case, the accounting machine 8 receives the sales registration data transmitted from the cart terminal 2 and performs accounting processing for the goods in one sales transaction based on the acquired sales registration data.

[0065] Furthermore, for example, if the sales registration table 242 is stored on the server 4, the settlement processing unit 205 encodes the transaction ID of the transaction into a code symbol such as a barcode or a two-dimensional code and displays it on the display unit 25. In this case, the accounting machine 8 obtains the transaction ID by reading the code symbol displayed on the display unit 25 of the cart terminal 2. The accounting machine 8 then obtains the sales registration data related to the obtained transaction ID from the server 4 and performs accounting processing for the goods of the registered transaction based on the obtained sales registration data.

[0066] The payment for goods is not limited to the method using the accounting machine 8; it may also be done using the cart terminal 2. For example, when the payment processing unit 205 receives an operation to start payment via the operation unit 26, it displays a screen on the display unit 25 that allows the user to select a payment method such as electronic money payment or credit card payment. Once a payment method is selected, the payment processing unit 205 receives input of payment information for making a payment using the selected payment method and, in cooperation with an external payment server, executes a payment process to settle the total amount of the goods registered for sale based on the input payment information. Such payment information may be, for example, a credit card number or a two-dimensional code for code payment. In the case of a two-dimensional code for code payment, the input may be received via the imaging unit 34.

[0067] The operation example of the cart terminal 2 will be explained below with reference to Figure 8. Figure 8 is a flowchart showing an example of the processing performed by the cart terminal 2. It should be assumed that imaging by the imaging unit 34 and weight measurement by the weight sensor 35 are performed in the background.

[0068] First, when the control unit 200 of the cart terminal 2 is instructed to start a transaction via the operation unit 26, it initializes the sales registration table 242 (step S11).

[0069] Furthermore, if the server 4 is configured to hold the sales registration table 242, for example, the control unit 200 requests the server 4 to issue a transaction ID, causing the server 4 to hold the sales registration table 242 related to the transaction ID of that transaction. The control unit 200 also retrieves the issued transaction ID from the server 4 and stores it in RAM 23 or the like.

[0070] Next, the image acquisition unit 201 acquires an image from the imaging unit 34 (step S12). Then, the product identification unit 203 determines whether or not it was possible to identify the product from the image acquired in step S12 (step S13). If the product cannot be identified (step S13; No), the process proceeds to step S23.

[0071] If the product identification unit 203 identifies the product from the image acquired in step S12 (step S13; Yes), it reads the theoretical weight of the product from the product master 241 (step S14). Subsequently, the product identification unit 203 derives the actual weight of the product based on the measurement result of the weight sensor 35 acquired by the weight acquisition unit 202 (step S15).

[0072] Next, the product identification unit 203 determines the customer's operation on the product identified in step S13 based on the measurement result of the weight sensor 35 acquired by the weight acquisition unit 202, or the direction of movement of the product shown in the image (step S16). Specifically, the product identification unit 203 determines whether the product identified in step S13 was placed in the storage unit 31 or taken out of the storage unit 31.

[0073] Next, the product identification unit 203 compares the theoretical weight read in step S14 with the actual weight derived in step S15 (step S17) and determines whether the weight difference is within a threshold (step S18). If the weight difference is within the threshold (step S18; Yes), the product identification unit 203 proceeds to step S20 while maintaining the product code of the product identified in step S13.

[0074] Furthermore, if the weight difference is outside the threshold (step S18; No), the product identification unit 203 identifies the product corresponding to the actual weight derived in step S15 from the product master 241 among the products that have similarity to the features extracted from the image (step S19), and proceeds to step S20. Note that if it is determined in step S16 that the product has been removed from the storage unit 31, the product identification unit 203 may identify the product corresponding to the actual weight derived in step S15 from the products registered in the sales registration table 242.

[0075] Next, the registration processing unit 204 determines whether the product operation identified in step S16 is an input operation indicating that the product is placed into the storage unit 31 (step S20). If the product operation is an input operation (step S20; Yes), the registration processing unit 204 registers the product identified by the product identification unit 203 in the sales registration table 242 (step S21) and proceeds to step S23. If the product operation is a retrieval operation (step S20; No), the registration processing unit 204 deletes the product identified by the product identification unit 203 from the sales registration table 242 (step S22) and proceeds to step S23.

[0076] Next, the registration processing unit 204 (or the settlement processing unit 205) determines whether or not the closing operation has been accepted (step S23). If the closing operation is not accepted (step S23; No), the registration processing unit 204 returns to step S12 and continues processing. If the closing operation is accepted (step S23; Yes), the settlement processing unit 205 executes the settlement process based on the product information registered in the sales registration table 242 (step S24) and terminates this process.

[0077] As described above, when a product is captured by the imaging unit 34, the cart terminal 2 compares the feature quantities of the product shown in the image with the feature quantities of multiple products stored in the product master 241, and identifies products with similar feature quantities from the product master 241. The cart terminal 2 also compares the theoretical weight and actual weight of the identified product and determines whether the difference is within a threshold, thereby determining whether the product identified from the image is correct. If the weight difference is outside the threshold, the cart terminal 2 determines that there is an error in the product identified from the image, identifies a product with the same actual weight from among products similar to the product shown in the image, and executes the sales registration process.

[0078] This allows the cart terminal 2 to correct misidentified products and execute the sales registration process even if it mistakenly identifies products that look very similar, thereby reducing the likelihood of incorrect product registration.

[0079] The embodiments described above can also be modified and implemented as appropriate by changing some of the configurations or functions of each of the devices described above. Therefore, several modifications of the embodiments described above will be described below as other embodiments. In the following, we will mainly describe the differences from the embodiments described above, and will omit detailed explanations of points that are common with what has already been described. Furthermore, the modifications described below may be implemented individually or in combination as appropriate.

[0080] (Variation 1) In the above embodiment, the number of imaging units 34 attached to the cart 3 was set to one, but this is not limited to this configuration, and multiple units may be provided. For example, as shown in Figure 9, another imaging unit 341 may be provided at a different position from the imaging unit 34. Here, Figure 9 is a side view showing an example of the overall configuration of the cart 3 according to Modification 1.

[0081] The imaging direction of the imaging unit 341 is directed toward the opening of the storage unit 31, similar to the imaging unit 34, and is positioned so that at least a portion of the imaging range includes the inside of the storage unit 31. The imaging unit 341 is also connected to the cart terminal 2, and the image obtained by imaging is output to the cart terminal 2.

[0082] In this case, the image acquisition unit 201 of the cart terminal 2 acquires images captured by the imaging unit 34 and the imaging unit 341, respectively. The product identification unit 203 recognizes and identifies products from the images captured by the imaging unit 34 and the imaging unit 341, respectively. The registration processing unit 204 registers the sales of the products identified by the product identification unit 203.

[0083] Furthermore, in order to prevent duplicate registration of products, the registration processing unit 204 will register the sale of a product only when a common product is identified from the images of the imaging unit 34 and the imaging unit 341, or when a product is registered from an image captured by either the imaging unit 34 or the imaging unit 341. In addition, if different product codes are identified from the images of the imaging unit 34 and the imaging unit 341, the registration processing unit 204 will not register the sale and will display an error message on the display unit 25.

[0084] (Modification 2) In the above-described embodiment, the product is stored in the storage compartment 31 of the cart 3. However, the configuration is not limited to this, and the product may be stored in a storage compartment 31 of other forms. For example, a shopping basket used for transporting goods within a store can be used as an example of a storage compartment 31. In this case, the imaging unit 34 is installed near the edge of the opening of the shopping basket, and is installed so that at least a part of the imaging range includes the inside of the shopping basket. The weight sensor 35 is installed at the bottom of the shopping basket, or on the outside of the bottom.

[0085] In this case, the terminal device may be a mobile device such as a smartphone having the same functions as the cart terminal 2, instead of the cart terminal 2. Furthermore, the shopping basket may be configured to be carried by the customer directly, or it may be configured to be carried while placed in the storage compartment 31 of the cart 3.

[0086] (Variation 3) In the above embodiment, the cart terminal 2 recognizes (and identifies) products from images captured by the imaging unit 34. However, the configuration is not limited to this, and this may be done by an external device such as a server 4. In this case, for example, the cart terminal 2 transmits the image acquired by the image acquisition unit 201 and the weight acquired by the weight acquisition unit 202 to the server 4, and the server 4 may be configured to include one or both of the product identification unit 203 and the registration processing unit 204. The server 4 then performs product recognition (and identification) and sales registration processing based on the image transmitted from the cart terminal 2.

[0087] The programs executed in each of the above-described embodiments (and modifications) are provided pre-installed in ROM, storage, etc. The programs executed in each of the above-described embodiments may also be provided as files in an installable or executable format, recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk).

[0088] Furthermore, the programs executed by each of the above-described embodiments may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Alternatively, the programs executed by each of the above-described embodiments may be provided or distributed via a network such as the Internet.

[0089] Although embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments and their variations can be implemented in a variety of other forms, and various omissions, substitutions, changes, and combinations can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0090] 1 Store System 2. Tablet devices 3 Shopping cart 4 servers 8. Cash register 200 Control Unit 201 Image Acquisition Unit 202 Weight acquisition part 203 Product Specification Department 204 Registration Processing Unit 205 Payment Processing Unit 241 Product Master 242 Sales Registration Table [Prior art documents] [Patent Documents]

[0091] [Patent Document 1] Japanese Patent Publication No. 2015-35077

Claims

1. A first acquisition means for acquiring an image from an imaging unit that images products stored in a storage unit, A second acquisition means is provided at the bottom of the storage compartment and acquires the weight measurement result from a weight measuring unit that measures the weight of the goods stored in the storage compartment, A means for identifying a product from among the multiple products that has a similarity to the product shown in the captured image and has the highest similarity, by comparing the feature quantities of the product shown in the captured image with the feature quantities of multiple products that have been stored in advance, and from among the multiple products, the product whose similarity to the product shown in the captured image exceeds a threshold, A registration means for registering the product identified by the aforementioned identification means, A determination means for determining whether the difference between the weight of a product identified by the identification means from among the weights of the plurality of products stored in advance and the amount of change in weight measured by the weight measuring unit is within a threshold, Equipped with, The identification means, when determined by the determination means to be outside the threshold, identifies a product whose weight corresponds to the amount of change among products whose similarity to the product shown in the captured image exceeds the threshold.

2. The system further includes a determination means for determining whether the product has been stored in the storage compartment or removed from the storage compartment, based on the increase or decrease in weight measured before and after the specified method. The registration means registers the product identified by the identification means when it is detected that the product has been stored in the storage compartment, and deregisters the product identified by the identification means when it is detected that the product has been removed from the storage compartment. The product registration device according to claim 1.

3. The imaging unit images the products stored in the storage unit and the products taken out of the storage unit, The system further includes a determination means for determining whether an item has been stored in the storage compartment or removed from the storage compartment by detecting the direction of movement of the item from the temporally continuous captured images. The registration means registers the product identified by the identification means when it is detected that the product has been stored in the storage compartment, and deregisters the product identified by the identification means when it is detected that the product has been removed from the storage compartment. The product registration device according to claim 1.

4. The computer of the product registration device A first acquisition means for acquiring an image from an imaging unit that images products stored in a storage compartment, A second acquisition means is provided at the bottom of the storage compartment and acquires the weight measurement result from a weight measuring unit that measures the weight of the goods stored in the storage compartment, A means for identifying a product from among the multiple products that has a similarity to the product shown in the captured image and has the highest similarity, by comparing the feature quantities of the product shown in the captured image with the feature quantities of multiple products that have been stored in advance, and from among the multiple products, the product whose similarity to the product shown in the captured image exceeds a threshold, A registration means for registering the product identified by the aforementioned identification means, A determination means for determining whether the difference between the weight of a product identified by the identification means from among the weights of the plurality of products stored in advance and the amount of change in weight measured by the weight measuring unit is within a threshold, and make it work The identification means is a program that, when the determination means determines that the weight is outside the threshold, identifies products whose similarity to the products shown in the captured image exceeds the threshold, and whose weight corresponds to the amount of change.

Images