Product registration device and program

Abstract

To provide an article registration device and a program capable of detecting articles with changed packages.SOLUTION: An article registration device includes: first acquisition means for acquiring an identifier that can identify an article from a code symbol attached to the article; second acquisition means for acquiring, from an image taken of the article, a characteristic quantity representing an external appearance feature of the article; third acquisition means for acquiring a characteristic quantity of the article corresponding to the identifier acquired by the first acquisition means from dictionary data associating, for each of a plurality of articles, an identifier of the article with the characteristic quantity representing the external appearance feature of the article; and detection means for detecting the article whose external appearance has been changed based on the characteristic quantity acquired by the first acquisition means and the characteristic quantity acquired by the third acquisition means.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] Embodiments of the present invention relate to a product registration device and a program.

Background Art

[0002] With the decline in the working population, in stores such as supermarkets, as part of labor-saving measures, self-checkout methods such as full-self POS and semi-self POS, where customers themselves register purchased products (hereinafter also referred to as product registration) and settle the purchased products, have been introduced. Also, as a new form, the introduction of a checkout method of the type that allows customers to register products while moving around the store has started.

[0003] In the checkout method of the type that allows customers to move around the store, customers themselves register products by using a camera or a code reader of a terminal device such as their own smartphone or a tablet terminal attached to a shopping cart to read a code symbol such as a barcode attached to the product to be purchased. Recently, a technology for registering products using image recognition technology that recognizes (hereinafter also referred to as specifying) the type of a product from an image of the product captured by a camera has also been proposed. In such image recognition technology, feature amounts on the appearance of the product captured by the camera are extracted and compared with the feature amounts of a plurality of types of products registered in dictionary data for verification, thereby recognizing the type of a product having similar features.

[0004] By the way, the package of a product sold in a store may be changed at the timing of the change of seasons or the implementation of a campaign. When the package of a product is changed, the features on the appearance also change, so there is a possibility that the product cannot be correctly recognized. Therefore, when the package of a product is changed, it is necessary to re-register the feature amount of the product in the dictionary data.

[0005] However, there are cases where stores fail to notice changes in packaging, and the dictionary continues to be used without being updated. In particular, with the aforementioned store-moving checkout system, where customers register products themselves, it is difficult for store staff to notice changes in packaging. [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] The problem that this invention aims to solve is to provide a product registration device and program capable of detecting products whose packaging has been changed. [Means for solving the problem]

[0007] The product registration device of the embodiment is A determination means for determining the direction of movement of a product entering or leaving a predetermined area based on an image of the predetermined area in which the product enters or leaves, and the Code symbol attached to the product Read from the above image, A first acquisition means for obtaining an identifier that can identify the product, If the direction of movement determined by the determination means is the direction of insertion into the predetermined area, the registration means switches between and executes a registration process that registers the product corresponding to the identifier acquired by the first acquisition means as a purchase target based on the identifier acquired by the first acquisition means, and if the direction of movement determined by the determination means is the direction of removal from the predetermined area, the registration means switches between and executes a cancellation process that removes the product corresponding to the identifier acquired by the first acquisition means as a purchase target based on the identifier acquired by the first acquisition means, and if the identifier is acquired by the first acquisition means, regardless of the determination result of the determination means, the image A second acquisition means for obtaining feature quantities representing the external characteristics of the product, and a third acquisition means for obtaining feature quantities corresponding to the identifier obtained by the first acquisition means from dictionary data that associates the identifier of the product with the feature quantities representing the external characteristics of the product for each of the multiple products, and the aforementioned 2 The system includes a detection means that detects products whose appearance has been altered based on the feature quantities acquired by the acquisition means and the feature quantities acquired by the third acquisition means. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of a checkout system according to an embodiment. [Figure 2] Figure 2 is a side view showing an example of the configuration of a shopping cart according to this embodiment. [Figure 3] Figure 3 shows an example of the hardware configuration of a terminal device according to this embodiment. [Figure 4] Figure 4 shows an example of the data structure of a product master according to an embodiment. [Figure 5]Figure 5 shows an example of the data structure of the sales registration table according to the embodiment. [Figure 6] Figure 6 shows an example of the server hardware configuration according to the present invention. [Figure 7] Figure 7 shows an example of the functional configuration of a terminal device and server according to the embodiment. [Figure 8] Figure 8 is a flowchart showing an example of the sales registration process performed by the terminal device of the embodiment. [Figure 9] Figure 9 is a sequence diagram showing an example of a data update process performed between the terminal device and the server in the embodiment. [Modes for carrying out the invention]

[0009] The product registration device and program according to this embodiment will be described in detail below with reference to the attached drawings. In the following embodiment, a terminal device attached to a shopping cart (hereinafter also referred to as a cart) used for transporting goods in a store will be described as an example of a product registration device. However, the invention is not limited by this application example.

[0010] Figure 1 shows an example of a checkout system 1 according to an embodiment. As shown in Figure 1, the checkout system 1 comprises a terminal device 2, a server 4, and an accounting device 8. The server 4 and the accounting device 8 are connected to each other via a network Na.

[0011] Network Na can be the internet, VPN (virtual private network), LAN (local area network), public communication network, mobile communication network, etc., either individually or in appropriate combinations. As an example, Network Na can be a combination of the internet and a mobile communication network.

[0012] Furthermore, a wireless access point 7 is connected to network Na. The wireless access point 7 is connected to terminal device 2 via network Nb, such as a wireless LAN. Therefore, terminal device 2 can communicate with server 4 and accounting device 8.

[0013] Cart 3 is equipped with a storage compartment 31 (see Figure 2) for storing products displayed in the store. The storage compartment 31 is open at the top, and products are placed into the storage compartment 31 from the top. Consumers using the store (hereinafter also referred to as customers) borrow a cart 3 from the store and, while moving around the store with the cart 3, place the products they wish to purchase from the displayed products into the storage compartment 31.

[0014] Terminal device 2 is a terminal device such as a tablet terminal that is operated by the customer. In this embodiment, terminal device 2 is attached to cart 3 and is operated by the customer using cart 3.

[0015] Terminal device 2 recognizes (hereinafter also referred to as "identification") the type of product to be placed in storage unit 31 from the image captured by the imaging unit 34, which will be described later, and executes sales registration processing for that product. Here, sales registration processing is also called product registration, and means the process of making the product identified by terminal device 2 a target for purchase. Specifically, sales registration processing refers to the process of registering information about the identified product in the sales registration table 243, which will be described later. Hereinafter, the information about the product registered in the sales registration table 243 will also be referred to as "sales registration data".

[0016] Furthermore, when terminal device 2 receives an operation to perform accounting (hereinafter also referred to as payment) for goods, it executes a process to transfer the sales registration data for one transaction registered in the sales registration process to accounting device 8. For example, terminal device 2 encodes the sales registration data into a code symbol such as a two-dimensional code and displays it. In this case, accounting device 8 reads the code symbol displayed on terminal device 2 to obtain the sales registration data for one transaction registered by terminal device 2 and executes accounting processing based on that sales registration data.

[0017] Server 4 is an example of a server device. Server 4 may be, for example, a store server provided in a store, or may be a server device provided outside the store. Server 4 in the present embodiment manages dictionary data and the like used in the package recognition process described later.

[0018] In the present embodiment, it is assumed that Server 4 is realized by a single device, but it may be a cloud server (or cloud system) realized by a plurality of network-connected devices.

[0019] The accounting device 8 is an accounting device provided in the store. The accounting device 8 executes settlement processing for products based on sales registration data. The settlement processing refers to processing such as displaying the total amount and tax amount of products related to one transaction, calculating and displaying change based on the deposit received from the customer, and issuing a receipt printed with product information and settlement information (total amount, deposit amount, change amount, etc.). In the present embodiment, the accounting device 8 is described as a self-service type accounting device that allows a customer who has performed sales registration using the terminal device 2 to perform settlement processing by their own operation, but it is not limited to this, and it may be an accounting device operated by an employee such as a store clerk.

[0020] Next, the cart 3 will be described. FIG. 2 is a side view showing an example of the configuration of the cart 3. The cart 3 includes a basket-shaped storage part 31 with an open upper surface, a rotating caster 32 disposed below the storage part 31, a handle part 33 connected to the storage part 31 and the caster 32 and gripped by a customer when moving the cart 3, and the like.

[0021] The storage part 31 has a configuration in which the side surface and the bottom surface are substantially surrounded in a square shape by a mesh-like lattice 311 and the upper part is open, and the products stored in the storage part 31 can be visually observed from the outside. Note that the side surface and the bottom surface may be constituted by wall surfaces instead of the mesh-like lattice 311.

[0022] Furthermore, the cart 3 is equipped with a terminal device 2 near the handle portion 33. The cart 3 is also equipped with an imaging unit 34 near the edge of the opening of the storage portion 31. The imaging unit 34 is implemented by an imaging device such as a digital camera. The imaging direction of the imaging unit 34 is set toward the opening of the storage portion 31. In other words, the imaging direction and imaging range of the imaging unit 34 are set so that it can image products being put in and taken out of the storage portion 31. The imaging unit 34 may also be configured to be included in the terminal device 2.

[0023] The imaging unit 34 captures images of products stored in the storage unit 31 or products taken out of the storage unit 31, and outputs the captured images to the terminal device 2. Here, "image" refers to a still image or a moving image in frame units.

[0024] Note that the configuration of cart 3 is not limited to the example shown in Figure 2. For example, the number of imaging units 34 is not limited to one, but may be multiple units. Also, the terminal device 2 may be configured to be detachable from cart 3. In this case, the terminal device 2 may be a mobile terminal (smartphone, tablet, etc.) owned by the customer or rented by the store and attached to cart 3.

[0025] From here, we will describe an example of the configuration of the main devices of the checkout system 1. First, we will describe the hardware configuration of terminal device 2. Figure 3 is a diagram showing an example of the hardware configuration of terminal device 2.

[0026] As shown in Figure 3, the terminal device 2 includes a CPU (Central Processing Unit) 21, ROM (Read Only Memory) 22, RAM (Random Access Memory) 23, storage unit 24, display unit 25, operation unit 26, communication unit 27, and connection unit 28, etc.

[0027] The CPU 21 is an example of a processor that comprehensively controls the operation of terminal device 2. The ROM 22 stores various programs. The RAM 23 is a workspace for displaying programs and various data.

[0028] The CPU 21, ROM 22, and RAM 23 are connected via a bus or the like to form a control unit 200 of the computer configuration. In the control unit 200, the CPU 21 operates according to a program stored in the ROM 22 or memory unit 24 and loaded into the RAM 23, thereby executing various processes.

[0029] The memory unit 24 has a storage medium such as an HDD (Hard Disk Drive) or flash memory, and retains its contents even when the power is cut off. The memory unit 24 stores programs that the CPU 21 can execute and various setting information. For example, the memory unit 24 has application programs for cooperating with the checkout system 1 pre-stored (installed).

[0030] Furthermore, the storage unit 24 stores the product master 241 and product recognition data 242, etc. The product master 241 is a data table for storing and managing product information of products sold in stores. Figure 4 is a diagram showing an example of the data structure of the product master 241. As shown in Figure 4, the product master 241 stores product codes in association with product information.

[0031] A product code is an example of identification information that can identify the type of product sold in a store. Product information is information about the product corresponding to the product code. Product information includes, for example, the product name and price.

[0032] Product recognition data 242 is a dataset for recognizing and identifying the type of product based on its external characteristics. Here, external characteristics of a product include, for example, the design, pattern, color, texture, and shape of the product or its outer packaging. Hereafter, the process of identifying a product code that can identify the type of product based on its external characteristics will also be called the "package recognition process."

[0033] The product recognition data 242 includes, for example, a feature extraction model 2421 and a matching dictionary 2422.

[0034] Feature extraction model 2421 is a pre-trained model for extracting feature quantities (e.g., feature vectors) that represent the external characteristics of a product. When an image is input to feature extraction model 2421, it extracts and outputs feature quantities that represent the external characteristics of the item (product) shown in the image. Feature extraction can be performed using known techniques such as CNN (Convolutional Neural Network).

[0035] The matching dictionary 2422 is dictionary data used to identify the type of product corresponding to a feature quantity extracted from an image. For example, the matching dictionary 2422 stores, for multiple products sold in a store, the product code corresponding to each product type and the feature quantity representing the external characteristics of that product (hereinafter also referred to as the matching feature quantity) in association with it.

[0036] In the package recognition process, the feature quantities of products extracted from the image using the feature extraction model 2421 are compared with the matching feature quantities of each product registered in the matching dictionary 2422 to calculate the similarity (also called distance or likelihood) between the features. Then, in the package recognition process, among the products registered in the matching dictionary 2422, product codes whose feature similarity exceeds a threshold (e.g., 90%) are identified as the type of product that was photographed.

[0037] Furthermore, the storage unit 24 stores the sales registration table 243. The sales registration table 243 is a data table for storing and managing products that have been registered for sale on its own terminal device 2. Figure 5 shows an example of the data structure of the sales registration table 243.

[0038] As shown in Figure 5, the sales registration table 243 stores product codes, product information, and quantities in association. In this table, for example, the combination of product code, product name, product information, and quantity corresponds to the sales registration data.

[0039] The product code is a product code that can identify a product that has been registered for sale, and it corresponds to the product code registered in product master 241. The product information stores the product information (product name, price, etc.) from product master 241 that corresponds to the product code of the product that has been registered for sale. The quantity refers to the number of products of the same type (same product code) that have been registered for sale.

[0040] Note that the data structure of the sales registration table 243 is not limited to that shown in Figure 5. For example, the sales registration table 243 may have a data structure that does not hold product information.

[0041] Returning to Figure 3, the display unit 25 is a display device such as an LCD (Liquid Crystal Display). The display unit 25 displays various information under the control of the CPU 21. The operation unit 26 is an input device such as a keyboard or a pointing device. The operation unit 26 outputs the operation content entered via the input device to the CPU 21. The operation unit 26 may also be a touch panel provided on the display unit 25.

[0042] The communication unit 27 is a wired or wireless communication interface that can connect to the network Nb. The communication unit 27 is connected to the wireless access point 7 via the network Nb. The control unit 200 communicates with external devices such as the server 4 and the accounting device 8 via the communication unit 27.

[0043] The connection unit 28 is an interface for connecting peripheral devices. The connection unit 28 is a wired or wireless interface compliant with standards for short-range wireless communication such as USB (Universal Serial Bus) or Bluetooth (registered trademark). In this embodiment, the connection unit 28 is connected to the imaging unit 34.

[0044] Next, we will describe the hardware configuration of Server 4. Figure 6 shows an example of the hardware configuration of Server 4.

[0045] As shown in Figure 6, the server 4 includes a CPU 41, ROM 42, RAM 43, storage unit 44, and communication unit 45, etc.

[0046] CPU41 is an example of a processor that provides overall control over the operation of server 4. ROM42 stores various programs. RAM43 is a workspace for displaying programs and various data.

[0047] The CPU 41, ROM 42, and RAM 43 are connected via a bus or the like to form a control unit 400 of the computer configuration. In the control unit 400, the CPU 41 operates according to a program stored in the ROM 42 or memory unit 44 and loaded into the RAM 43, thereby executing various processes.

[0048] The storage unit 44 has a storage medium such as an HDD or flash memory and maintains its contents even when the power is cut off. The storage unit 44 stores programs that the CPU 41 can execute and various setting information. For example, the storage unit 44 stores various programs for realizing server functions in the checkout system 1, such as database programs and web application programs. The storage unit 44 may also store various programs for generating and updating product recognition data 242.

[0049] Furthermore, the storage unit 44 stores the product master 441 and product recognition data 442. The product master 441 corresponds to the product master 241 described above. For example, the server 4 provides the product master 441 to each of the terminal devices 2, so that the product master 441 is stored as the product master 241 in each of the terminal devices 2.

[0050] Furthermore, the product recognition data 442 corresponds to the product recognition data 242 described above and includes a feature extraction model 4421 and a matching dictionary 4422. For example, the server 4 provides the product recognition data 442 to each of the terminal devices 2, so that the product recognition data 442 is stored as product recognition data 242 in each of the terminal devices 2.

[0051] Furthermore, the storage unit 44 has a data storage area 443. The data storage area 443 is a data area for storing package update information, which will be described later, transmitted from the terminal device 2.

[0052] The communication unit 45 is a wired or wireless communication interface that can be connected to network Na. The communication unit is connected to external devices such as terminal devices 2 and accounting devices 8 via network Na. The control unit 400 communicates with external devices such as terminal devices 2 and accounting devices 8 via the communication unit 45.

[0053] Next, the functional configuration of terminal device 2 and server 4 will be described. Figure 7 shows an example of the functional configuration of terminal device 2 and server 4.

[0054] As shown in Figure 7, the terminal device 2 is functionally configured to include an image acquisition unit 201, a product area detection unit 202, a code reading unit 203, a package recognition unit 204, a registration processing unit 205, and a payment processing unit 206.

[0055] The functional configuration of terminal device 2 is a software configuration realized by the processor of terminal device 2 operating according to a program stored in ROM or memory, but is not limited to this. For example, some or all of the functional configuration of terminal device 2 may be realized by hardware such as dedicated circuits.

[0056] The image acquisition unit 201 acquires images captured by the imaging unit 34 via the connection unit 28. The images acquired by the image acquisition unit 201 (hereinafter also referred to as captured images) are temporarily stored in the RAM 23 for processing by the code reading unit 203 and the registration processing unit 205.

[0057] The product area detection unit 202 detects areas in the captured image where objects are represented. Specifically, the product area detection unit 202 detects areas where products are represented (hereinafter also referred to as product areas) from the image acquired by the image acquisition unit 201, and outputs the product area or information representing the product area. The method for detecting the product area is not particularly limited, and known object detection techniques such as SSD (Single Shot Multibox Detector) can be used.

[0058] Furthermore, the product area detection unit 202 determines the direction of movement of the product by detecting the position and movement (e.g., motion vector) of the product area within the image detected from each of the images acquired sequentially by the image acquisition unit 201. Specifically, the product area detection unit 202 determines whether the product has moved downwards or upwards.

[0059] For example, when an item is placed in the storage unit 31, the item area detection unit 202 detects that the item has moved downwards (hereinafter also referred to as the loading direction). Also, when an item is removed from the storage unit 31, the item area detection unit 202 detects that the item has moved upwards (hereinafter also referred to as the removal direction).

[0060] The code reading unit 203 is an example of the first acquisition means. The code reading unit 203 reads information from code symbols such as barcodes and two-dimensional codes contained in the captured image. Specifically, the code reading unit 203 performs a code reading process to read code symbols from the product area detected by the product area detection unit 202.

[0061] In the code reading process, the code reading unit 203 extracts an image representing a code symbol from the product area and decodes the extracted code symbol to obtain the information held in that code symbol. For example, the code reading unit 203 obtains the product code of the subject product by reading the code symbol attached to the product.

[0062] The package recognition unit 204 is an example of a second acquisition means, a third acquisition means, a fourth acquisition means, a detection means, and an output means. The package recognition unit 204 uses product recognition data 242 to perform package recognition processing on the captured image, thereby recognizing the type of product represented in the captured image.

[0063] Specifically, the package recognition unit 204 performs package recognition processing on the product area detected by the product area detection unit 202, thereby extracting (acquiring) the product features represented in that product area. Next, the package recognition unit 204 compares the extracted features with the matching features of each product registered in the matching dictionary 2422, and retrieves the product code of the product whose similarity exceeds the threshold from the matching dictionary 2422, outputting it as the recognition result.

[0064] The registration processing unit 205 is an example of a registration means. Based on the direction of movement of the product determined by the product area detection unit 202 and the product code read by the code reading unit 203 or the product code recognized by the package recognition unit 204, the registration processing unit 205 executes a sales registration process or a registration cancellation process for the product.

[0065] When a product code is obtained by the code reading unit 203 or the package recognition unit 204, and movement in the input direction is detected by the product area detection unit 202, the registration processing unit 205 executes a sales registration process to register the product corresponding to the product code in the sales registration table 243. Specifically, the registration processing unit 205 reads product information corresponding to the product code obtained by the code reading unit 203 or the package recognition unit 204 from the product master 241, associates the product code with the quantity (initial value is 1), and registers it in the sales registration table 243. In addition, when a product code already registered in the sales registration table 243 is obtained by the code reading unit 203 or the package recognition unit 204, and movement in the input direction is detected by the product area detection unit 202, the registration processing unit 205 adds 1 to the quantity of the corresponding sales registration data.

[0066] Furthermore, when the code reading unit 203 or package recognition unit 204 acquires a product code and the product area detection unit 202 detects movement in the input direction, the registration processing unit 205 executes a process to cancel the sales registration of the product corresponding to the product code. Specifically, the registration processing unit 205 searches the sales registration table 243 for sales registration data corresponding to the acquired product code and subtracts 1 from the score of the retrieved sales registration data. Also, if the score after the subtraction becomes 0, the registration processing unit 205 deletes the sales registration data from the sales registration table 243.

[0067] Furthermore, each time the registration processing unit 205 updates the sales registration table 243, it displays a screen on the display unit 25 based on the sales registration data registered in the sales registration table 243. For example, the registration processing unit 205 displays a screen on the display unit 25 that lists the products that have been registered for sale.

[0068] As described above, in the checkout system 1 of this embodiment, when a customer places an item into the storage compartment 31 of the cart 3, the sale of that item is automatically registered. Furthermore, in the checkout system 1, when a customer removes an item from the storage compartment 31 of the cart 3, the registration of that item is automatically canceled. In other words, the checkout system 1 is a mobile checkout system that allows customers to register items themselves while moving around the store.

[0069] The settlement processing unit 206 executes processing to settle the payment for the registered sale items in response to customer operations. Specifically, the settlement processing unit 206 performs processing to transfer the sales registration data of the registered sale items to the accounting device 8, thereby making the transaction ready for settlement at the accounting device 8.

[0070] For example, when the settlement processing unit 206 receives an operation via the operation unit 26 to instruct the end of sales registration or to instruct the start of settlement (hereinafter collectively referred to as a closing operation), it transmits the sales registration data registered in the sales registration table 243 to the accounting device 8 via short-range wireless communication such as Bluetooth®.

[0071] Alternatively, for example, the settlement processing unit 206 may transmit the sales registration data registered in the sales registration table 243 to the accounting device 8 via the server 4. In this case, the settlement processing unit 206 transmits the sales registration data registered in the sales registration table 243 to the server 4 along with an identification ID that can identify its own terminal device 2 or an identification ID that can identify the transaction. The settlement processing unit 206 also transmits the identification ID to the accounting device 8, the payment destination, via short-range wireless communication. The accounting device 8 then queries the server 4 for the identification ID transmitted from the settlement processing unit 206 and obtains the sales registration data corresponding to that identification ID.

[0072] Furthermore, the method of notifying the accounting device 8 of the identification ID is not limited to short-range wireless communication. For example, the settlement processing unit 206 may encode the identification ID into a code symbol such as a two-dimensional code and display it on the display unit 25, thereby notifying the accounting device 8 of the identification ID via the code symbol. In this case, the accounting device 8 can obtain the identification ID from the terminal device 2 by reading the code symbol displayed on the display unit 25.

[0073] Alternatively, for example, the payment processing unit 206 may transmit its terminal device 2 identification ID to the accounting device 8 via short-range wireless communication such as Bluetooth®. In this case, the accounting device 8 obtains sales registration data corresponding to the identification ID transmitted from the terminal device 2 from the server 4, and performs accounting processing for the goods for one transaction that has been registered for sale based on the obtained sales registration data.

[0074] Furthermore, payment for goods is not limited to the method using the accounting device 8, but may also be performed using the terminal device 2. For example, when the payment processing unit 206 receives an operation to start payment via the operation unit 26, it displays a screen on the display unit 25 that allows the user to select a payment method such as electronic money payment or credit card payment. Once a payment method is selected, the payment processing unit 206 receives input of payment information for making a payment using the selected payment method and, in cooperation with an external payment server, executes a payment process to settle the total amount of the goods registered for sale based on the input payment information. Such payment information may be, for example, a credit card number or a two-dimensional code for code payment. The payment processing unit 206 may also receive input of a two-dimensional code for code payment via the imaging unit 34.

[0075] Furthermore, payment for goods is not limited to the method using the accounting device 8, but may also be performed using the terminal device 2. For example, when the payment processing unit 206 receives an operation to start payment via the operation unit 26, it displays a screen on the display unit 25 that allows the user to select a payment method such as electronic money payment or credit card payment. Once a payment method is selected, the payment processing unit 206 receives input of payment information for making a payment using the selected payment method and, in cooperation with an external payment server, executes a payment process to settle the total amount of the goods registered for sale based on the input payment information. Such payment information may be, for example, a credit card number or a two-dimensional code for code payment. The payment processing unit 206 may also receive input of a two-dimensional code for code payment via the imaging unit 34.

[0076] Incidentally, the packaging of products sold in stores may change with the seasons or during promotional campaigns. When product packaging changes, the external characteristics also change, which may make it difficult to correctly identify the product. Therefore, when product packaging changes, an update process is required to reflect the product's characteristics in the matching dictionary 2422.

[0077] However, there are cases where the store does not notice the change in packaging, and the matching dictionary 2422 continues to be used without being updated. In particular, in a store-moving checkout system like this embodiment, where the customer registers the product themselves, there is a problem in that it is difficult for store staff to notice changes in packaging.

[0078] Therefore, in the terminal device 2 of this embodiment, the package recognition unit 204 performs processing to detect products whose packaging has been changed, based on the product code read by the code reading unit 203.

[0079] Specifically, before recognizing the product code in package recognition, the package recognition unit 204 reads (acquires) the matching feature quantities for the product corresponding to the product code from the matching dictionary 2422 once the product code has been read by the code reading unit 203. Next, the package recognition unit 204 detects products whose packaging has been changed by comparing the product feature quantities extracted from the image in package recognition with the matching feature quantities read from the matching dictionary 2422.

[0080] Here, if the similarity between the product features extracted from the image during package recognition and the matching features read from the matching dictionary 2422 is above a threshold, the package recognition unit 204 determines that the package has not been changed. Conversely, if the similarity between the product features extracted from the image during package recognition and the matching features read from the matching dictionary 2422 falls below the threshold, the package recognition unit 204 determines that it has detected a product whose package has been changed.

[0081] When the package recognition unit 204 detects a product whose packaging has been changed, it executes a process to notify or alert the user about that product.

[0082] For example, the package recognition unit 204 outputs information including the product code read by the code reading unit 203 (hereinafter also referred to as package change information). Here, outputting package change information means sending it to an external device, but it is not limited to this and is a concept that includes other output forms such as display output, print output, and file output. In this embodiment, an example of sending package change information to a server 4 will be described.

[0083] Furthermore, for example, the package recognition unit 204 may include the image of the image region detected by the product region detection unit 202 (hereinafter also simply referred to as the image region) along with the product code read by the code reading unit 203 in the package change information and send it to the server 4. Alternatively, for example, the package recognition unit 204 may include the feature quantities extracted from the image region along with the product code read by the code reading unit 203 in the package change information and send it to the server 4. In this case, the product code, image region, and / or feature quantities included in the package change information can be used as update data to update the dictionary data of the product whose package has been changed. This embodiment will describe the form in which the update data is included in the package change information and sent.

[0084] The timing of outputting package change information is not particularly restricted. For example, the package recognition unit 204 may output package change information each time it detects a product whose packaging has been changed. Alternatively, the package recognition unit 204 may store the package change information related to a product in the RAM 23 or storage unit 24 each time it detects a product whose packaging has been changed, and send the stored package change information to the server 4 at a predetermined timing. For example, the package recognition unit 204 may send the package change information at a predetermined time, such as every hour, or when it has detected a predetermined number of package changes (e.g., 10 times).

[0085] Furthermore, the destination for the package change information is not limited to server 4; it may be other devices. For example, the package recognition unit 204 may transmit the package change information to a terminal device operated by a store employee (hereinafter also referred to as the employee terminal). The package recognition unit 204 may also include some or all of the product information corresponding to the product code (e.g., product name) in the package change information before transmitting it. When the employee terminal receives the package change information, it will display the product code, image area, and product name of the product corresponding to the product code included in the package change information on its display device to notify the employee of the product whose packaging has been changed.

[0086] On the other hand, as shown in Figure 7, server 4 is functionally configured to include an update notification receiving unit 401, an update processing unit 402, and a data provision unit 403.

[0087] The functional configuration of Server 4 is a software configuration realized by the Server 4's processor operating according to programs stored in ROM or memory, but is not limited to this. For example, some or all of the functional configuration of Server 4 may be realized by hardware such as dedicated circuits.

[0088] The update notification receiving unit 401 receives package update information transmitted from the terminal device 2 via the communication unit 45. Upon receiving the package update information, the update notification receiving unit 401 stores the update data contained in the package update information in the data storage area 443.

[0089] Furthermore, when the update notification receiving unit 401 receives package update information, it may send a message to the store clerk's terminal indicating that the package has been updated for the product corresponding to the product code included in the package update information. For example, the update notification receiving unit 401 may send the received package update information to the store clerk's terminal.

[0090] The update processing unit 402 updates the matching dictionary 2422 related to package recognition of products whose packages have been updated. Specifically, the update processing unit 402 performs processing to update the correspondence between product features and product codes based on the update data stored in the data storage area 443.

[0091] For example, if the update data includes an image region and a product code, the update processing unit 402 extracts features from the image region using the feature extraction model 4421 and updates the features of the matching dictionary 4422 corresponding to the product code specified in the update data based on the extracted features. Alternatively, if the update data includes features and a product code, the update processing unit 402 updates the features of the matching dictionary 4422 corresponding to the product code specified in the update data based on the features specified in the update data. The method for updating the matching dictionary 4422 is not particularly limited, and known techniques such as fine-tuning in the field of machine learning can be used.

[0092] The update process for the matching dictionary 4422 shall be performed for each product code, but the timing of the process is not particularly restricted. For example, the update processing unit 402 may perform the update process each time update data is stored in the data storage area 443, or it may perform it at a predetermined timing. In the latter case, for example, the update processing unit 402 may perform the update process when a predetermined number (e.g., 10) of update data for the same product code has been stored in the data storage area 443. Alternatively, for example, the update processing unit 402 may perform the update process in response to instructions from a store terminal.

[0093] The data provision unit 403 provides various types of data to each of the terminal devices 2. Specifically, when the matching dictionary 4422 is updated by the update processing unit 402, the data provision unit 403 provides the updated matching dictionary 4422 to each of the terminal devices 2. Alternatively, the data provision unit 403 may provide each of the terminal devices 2 with the data difference between the existing matching dictionary 4422 and the updated matching dictionary 4422 as a new matching dictionary 4422. When the control unit 200 of the terminal device 2 receives the matching dictionary 4422 from the data provision unit 403, it updates the matching dictionary 2422 in the storage unit 24 based on the matching dictionary 4422.

[0094] Furthermore, even if the matching dictionary 4422 has not been updated, the data provision unit 403 may provide the matching dictionary 4422 to each of the terminal devices 2 at a predetermined timing.

[0095] Furthermore, the timing at which the data provision unit 403 provides the matching dictionary 4422 is not particularly limited; for example, it may provide the data at a predetermined time. The matching dictionary 4422 may also provide data in response to a request from the terminal device 2. The data provision unit 403 may also provide data when it is updated. Furthermore, the data provision unit 403 may provide data in response to instructions from the store clerk terminal.

[0096] Furthermore, the data provision unit 403 may also provide data other than the matching dictionary 4422 (for example, the product master 441 or the feature extraction model 4421) to each of the terminal devices 2 at predetermined timings. In this case, the control unit 200 of the terminal device 2 updates the corresponding data stored in the storage unit 24 based on the data provided by the data provision unit 403.

[0097] The following describes an example of the operation of the checkout system 1. First, an example of the sales registration process performed by terminal device 2 will be explained with reference to Figure 8. Figure 8 is a flowchart showing an example of the sales registration process performed by terminal device 2.

[0098] First, the image acquisition unit 201 of the terminal device 2 sequentially acquires images captured by the imaging unit 34 (step S11). Next, the product area detection unit 202 determines whether or not a product area has been detected from the images captured in step S11 (step S12). If no product area is detected (step S12; No), the process proceeds to step S26. If a product area is detected (step S12; Yes), the product area detection unit 202 determines the direction of movement of the product area (product) from the images acquired sequentially over time (step S13).

[0099] Next, the code reading unit 203 performs a code reading process to read a code symbol (product code) from the product area detected by the product area detection unit 202 (step S14). The package recognition unit 204 also performs a feature extraction process to extract feature quantities from the product area detected by the product area detection unit 202 (step S15).

[0100] Next, the package recognition unit 204 determines whether or not a product code was read in the code reading process of step S14 (step S16). If the product code was not read (step S16; No), the package recognition unit 204 performs a matching process that compares the feature quantities extracted in the feature extraction process of step S17 with the feature quantities of each product registered in the matching dictionary 2422 (step S17). If the package recognition unit 204 identifies the product codes of products whose similarity exceeds a threshold as a result of the matching process (step S18), it proceeds to step S23.

[0101] On the other hand, if it is determined in step S16 that a product code has been read (step S16; Yes), the package recognition unit 204 reads and obtains the matching feature quantity for the product corresponding to that product code from the matching dictionary 2422 (step S19). Next, the package recognition unit 204 calculates the similarity by comparing the feature quantity extracted in the feature extraction process in step S15 with the matching feature quantity read in step S19 (step S20). Next, the package recognition unit 204 determines whether the calculated similarity exceeds a threshold (step S21).

[0102] If the similarity score exceeds the threshold (step S21; Yes), the package recognition unit 204 determines that there is no change to the package and proceeds to step S23.

[0103] Furthermore, if the similarity falls below a threshold (step S21; No), the package recognition unit 204 determines that it has detected a product whose packaging has been changed. In this case, the package recognition unit 204 sends package change information, including the product code read in step S14 and the image of the product area detected in step S12, to the server 4 (step S22), and proceeds to step S23. When the update notification receiving unit 401 of the server 4 receives package change information from the terminal device 2, it saves the received package change information to the data storage area 443.

[0104] In the following step S23, the registration processing unit 205 determines whether the direction of movement determined in step S13 was the input direction or the output direction (step S23).

[0105] If the direction of movement is the input direction (step S23; Yes), the registration processing unit 205 executes a sales registration process to register the product corresponding to the product code obtained in step S14 or step S18 in the sales registration table 243 (step S24). If the direction of movement is the retrieval direction (step S23; No), the registration processing unit 205 executes a registration cancellation process to delete the product corresponding to the product code obtained in step S14 or step S18 from the sales registration table 243 (step S25).

[0106] Next, the registration processing unit 205 (or the settlement processing unit 206) determines whether or not the closing operation has been accepted (step S26). If the closing operation is not accepted (step S26; No), the process returns to step S11. If the closing operation is accepted (step S26; Yes), the settlement processing unit 206 executes the process to settle the payment for the products registered in the sales registration table 243 (step S27).

[0107] This allows terminal device 2 to perform sales registration and cancellation processing for products using either the product code read from the code symbol or the product code identified from the product's external characteristics. Furthermore, terminal device 2 can perform detection processing for products whose packaging has been changed during the sales registration and cancellation processing process.

[0108] Next, an example of data update processing between terminal device 2 and server 4 will be described with reference to Figure 9. Figure 9 is a sequence diagram showing an example of data update processing between terminal device 2 and server 4. It should be assumed that, as a prerequisite for this process, package change information is stored in the data storage area 443 of server 4. Furthermore, the package change information is assumed to include the product area and product code.

[0109] First, the update processing unit 402 of server 4 classifies the package change information stored in the data storage area 443 according to the type of product code included in the package information (step S31). Next, the update processing unit 402 extracts feature quantities related to the product code of the classification from the product area included in the classified package change information (step S32). Then, based on the combination of the feature quantities extracted in step S32 and the classified product code, the update processing unit 402 updates the matching feature quantities of the matching dictionary 4422 registered in association with the product code (step S33).

[0110] Next, the data provision unit 403 provides the updated matching dictionary 4422 to each of the terminal devices 2 (step S34). Meanwhile, when the control unit 200 of the terminal device 2 receives the matching dictionary 4422 from the server 4, it updates the matching dictionary 2422 based on the matching dictionary 4422 (step S35).

[0111] This allows each terminal device 2 to perform package recognition processing using the updated matching dictionary 2422, which reflects the feature quantities of the updated product package.

[0112] As described above, the terminal device 2 of this embodiment obtains a product code from a code symbol attached to a product and obtains feature quantities from an image of the product. Furthermore, for each of multiple products, the terminal device 2 obtains the matching feature quantity of the product corresponding to the product code read from the code symbol from a matching dictionary 2422 that associates the product code of the product with the matching feature quantity of that product, and performs processing to detect products whose appearance has been changed by comparing it with the feature quantity obtained from the image. In this way, the terminal device 2 can efficiently detect products whose packaging has been changed by comparing the feature quantity derived from the image of products related to the same product code with the matching feature quantity in the matching dictionary 2422, based on the similarity of the feature quantities.

[0113] Furthermore, terminal device 2 detects a product as having had its appearance changed if the similarity between the image-derived features of a product with the same product code and the matching features in the matching dictionary 2422 falls below a threshold. This allows terminal device 2 to identify products whose appearance has changed, thus enabling efficient detection of products with altered packaging.

[0114] Furthermore, if terminal device 2 detects a product whose packaging has been changed, it outputs package change information including the product code of that product. This allows terminal device 2 to notify the destination device of the product whose packaging has been changed, enabling it to provide support for countermeasures such as updating dictionary data.

[0115] Furthermore, when terminal device 2 detects a product whose packaging has been changed, it outputs package update information including the product code of the product and an image of the product area containing the product. This allows terminal device 2 to provide data that can be used to update the dictionary data for products whose packaging has been detected, thereby supporting the updating of the dictionary data.

[0116] Furthermore, terminal device 2 detects products with changed packaging during the customer's sales registration (or cancellation) process, thus eliminating the need for store staff to verify packaging. Additionally, terminal device 2 can acquire (collect) data usable for updating dictionary data at each terminal device, enabling efficient support for updating dictionary data.

[0117] The embodiments described above can also be modified and implemented as appropriate by changing some of the configurations or functions of each of the devices described above. Therefore, several modifications of the embodiments described above will be described below as other embodiments. In the following, we will mainly describe the differences from the embodiments described above, and will omit detailed explanations of points that are common with what has already been described. Furthermore, the modifications described below may be implemented individually or in combination as appropriate.

[0118] (Variation 1) In the above-described embodiment, a terminal device 2 that is attached to the cart 3 was explained, but the form of the terminal device 2 is not limited to this. For example, the terminal device 2 may be a mobile device such as a smartphone owned by the customer, or a mobile device such as a smartphone or tablet that the store lends to the customer.

[0119] In this case, the terminal device 2 may be configured to cooperate with the cart 3 by communicating with the imaging unit 34 via wired or wireless connection, or it may be configured independently of the cart 3. In the latter case, the customer can register the sale of a product by taking an image of the product with the camera provided by the terminal device 2, in the same manner as in the embodiment described above. Furthermore, when canceling a registration, the customer can cancel the registration in the same manner as in the embodiment described above by selecting the product to be canceled from the list of registered sales displayed on the screen of the terminal device 2.

[0120] (Modification 2) In the above embodiment, the terminal device 2 performs the package recognition processing, but the configuration is not limited to this, and the processing may be performed by an external device for package recognition.

[0121] In this case, the package recognition unit 204 of the terminal device 2 works in cooperation with an external device for package recognition processing to achieve the same functionality as in the above-described embodiment. Specifically, the package recognition unit 204 transmits an image of the image region detected by the product area detection unit 202 to the external device for package recognition processing, causing the external device to perform package recognition processing, and obtains the processing result of the package recognition processing, i.e., the product recognition result, from the external device.

[0122] Furthermore, if a product code is read by the code reading unit 203 before the package recognition unit 204 obtains the recognition result from the external device, it notifies the external device of the product code, instructing the external device to compare the feature quantities extracted from the image area with the matching feature quantities related to the notified product code. In this case, the package recognition unit 204 obtains the comparison result, i.e., the similarity score, from the external device and determines whether the similarity score exceeds a threshold. If the similarity score is below the threshold, the package recognition unit 204 generates package update information including the image area and the product code and sends it to the server 4 or the store clerk terminal.

[0123] In this modified configuration, the external device will have product recognition data 242, and therefore the matching dictionary 2422 of the external device will be updated using the matching dictionary 4422 updated on the server 4.

[0124] As a result, in the checkout system 1 according to this modified example, an external device performs the package recognition processing, which reduces the processing load on the terminal device 2 and achieves the same effects as the embodiment described above.

[0125] Furthermore, in the modified checkout system 1, an external device can centrally execute package recognition processing for multiple terminal devices 2. Therefore, by updating the matching dictionary 2422 of the external device, improvements in the accuracy of package recognition processing can be reflected in multiple terminal devices 2. Consequently, the management of the matching dictionary 2422 can be made more efficient in the modified checkout system 1.

[0126] (Variation 3) In the above-described embodiment, the code symbol (product code) attached to the product is read from the image captured by the imaging unit 34. However, the configuration is not limited to this, and for example, a code reading device specifically designed for reading code symbols may be used.

[0127] In this case, the code reader may be built into the terminal device 2, or it may be connected to the terminal device 2 by wire or wireless connection. The code reading unit 203 of the terminal device 2 works in cooperation with the code reader to acquire information about the code symbol (for example, the product code) read by the code reader.

[0128] As a result, the checkout system 1 according to this modified example can achieve the same effects as the embodiment described above, even if it is configured to read code symbols using a code reader.

[0129] The programs executed in each of the above-described embodiments (and modifications) are provided pre-installed in ROM, storage, etc. The programs executed in each of the above-described embodiments may also be provided as files in an installable or executable format, recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk).

[0130] Furthermore, the programs executed by each of the above-described embodiments may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Alternatively, the programs executed by each of the above-described embodiments may be provided or distributed via a network such as the Internet.

[0131] Although embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments and their variations can be implemented in a variety of other forms, and various omissions, substitutions, changes, and combinations can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of symbols]

[0132] 1. Checkout System 2 Terminal devices 3. Shopping cart 4 servers 8. Accounting device 200 Control Unit 201 Image Acquisition Unit 202 Product Area Detection Unit 203 Code reading unit 204 Package Recognition Unit 2421 Feature Extraction Model 2422 Dictionary for comparison 400 Control Unit 401 Update Notification Reception Department 402 Update Processing Unit 403 Data Provision Department [Prior art documents] [Patent Documents]

[0133] [Patent Document 1] Japanese Patent Publication No. 2022-29170

Claims

1. A determination means for determining the direction of movement of a product entering or leaving a predetermined area based on an image of the predetermined area in which the product enters or leaves, A first acquisition means reads a code symbol attached to the product from the image and obtains an identifier that can identify the product, A registration means that switches between and executes the following: when the direction of movement determined by the determination means is the direction of insertion into the predetermined area, it registers the product corresponding to the identifier acquired by the first acquisition means as a purchase target; and when the direction of movement determined by the determination means is the direction of removal from the predetermined area, it executes the following cancellation process, which removes the product corresponding to the identifier acquired by the first acquisition means from the purchase target. If the identifier is obtained by the first acquisition means, a second acquisition means obtains a feature quantity representing the external characteristics of the product from the image, regardless of the determination result of the determination means. A third acquisition means obtains the feature quantity of the product corresponding to the identifier obtained by the first acquisition means from a dictionary data that associates the identifier of the product with the feature quantity representing the external characteristics of the product for each of the multiple products, A detection means for detecting products whose appearance has been changed based on the feature quantities acquired by the second acquisition means and the feature quantities acquired by the third acquisition means, A product registration device equipped with the following features.

2. The detection means detects a product as having had its appearance altered if the similarity between the feature quantity acquired by the second acquisition means and the feature quantity acquired by the third acquisition means falls below a threshold. The product registration device according to claim 1.

3. When the detection means detects a product whose appearance has been altered, the output means outputs information including the identifier of the product. The product registration device according to claim 1 or 2, further comprising:

4. The output means outputs the information including the identifier of the product and the image containing the product. The product registration device according to claim 3.

5. The system further includes a fourth acquisition means that compares the feature quantities acquired by the second acquisition means with the feature quantities of each product registered in the dictionary data, thereby acquiring identifiers of products from the dictionary data whose feature similarity exceeds the threshold. The registration means switches between executing the registration process and the cancellation process based on the direction of movement determined by the determination means and the identifier acquired by the first acquisition means or the identifier acquired by the fourth acquisition means. The product registration device according to claim 2.

6. The computer of the product registration device A determination means for determining the direction of movement of products entering and exiting a predetermined area based on an image of the predetermined area in which the products enter and exit, A first acquisition means reads a code symbol attached to the product from the image and obtains an identifier that can identify the product, A registration means that switches between and executes the following: when the direction of movement determined by the determination means is the direction of insertion into the predetermined area, it registers the product corresponding to the identifier acquired by the first acquisition means as a purchase target; and when the direction of movement determined by the determination means is the direction of removal from the predetermined area, it executes the following cancellation process, which removes the product corresponding to the identifier acquired by the first acquisition means from the purchase target. If the identifier is obtained by the first acquisition means, a second acquisition means obtains a feature quantity representing the external characteristics of the product from the image, regardless of the determination result of the determination means. A third acquisition means obtains the feature quantity of the product corresponding to the identifier obtained by the first acquisition means from a dictionary data that associates the identifier of the product with the feature quantity representing the external characteristics of the product for each of the multiple products, A detection means for detecting products whose appearance has been changed based on the feature quantities acquired by the second acquisition means and the feature quantities acquired by the third acquisition means, A program to make it work.

Images