Article registration device and program

Abstract

To provide an article registration device and a program capable of detecting articles with changed packages.SOLUTION: An article registration device includes: first acquisition means for acquiring an identifier that can identify an article from a code symbol attached to the article; second acquisition means for acquiring, from an image taken of the article, a characteristic quantity representing an external appearance feature of the article; third acquisition means for acquiring a characteristic quantity of the article corresponding to the identifier acquired by the first acquisition means from dictionary data associating, for each of a plurality of articles, an identifier of the article with the characteristic quantity representing the external appearance feature of the article; and detection means for detecting the article whose external appearance has been changed based on the characteristic quantity acquired by the first acquisition means and the characteristic quantity acquired by the third acquisition means.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] An embodiment of the present invention relates to a product registration device and a program. [Background technology]

[0002] As the working population declines, supermarkets and other stores have introduced self-service checkout systems, known as full-self POS or semi-self POS, in which customers themselves register the products they purchase (hereinafter also referred to as product registration) and pay for them, as part of their labor-saving measures. Also, a new type of checkout system has begun to be introduced: a mobile sales floor checkout system, in which customers themselves register products while moving around the store.

[0003] In a mobile sales floor checkout system, a customer registers a product by reading a code symbol such as a barcode attached to the product to be purchased using a camera or a code reader of a terminal device such as a smartphone or a tablet terminal attached to a shopping cart. Recently, a technology for registering products has been proposed that uses an image recognition technology that recognizes (hereinafter also referred to as identification) the type of the product from an image of the product captured by a camera. With such an image recognition technology, the external features of the product are extracted from the image of the product captured by the camera, and the type of product with similar features is recognized by comparing it with the features of multiple types of products registered in dictionary data for comparison.

[0004] Incidentally, the packaging of products sold in stores may change due to seasonal changes, campaigns, etc. When the packaging of a product changes, the external characteristics also change, which may result in the product not being correctly recognized. Therefore, when the packaging of a product is changed, it is necessary to re-register the feature quantities of that product in the dictionary data.

[0005] However, there are cases where the store does not notice the change in the package, and the dictionary continues to be used without being updated. In particular, in the above-mentioned mobile checkout method, since the customer registers the product himself, there is a problem that it is difficult for the store clerk to notice the change in the package. Summary of the Invention [Problem to be solved by the invention]

[0006] An object of the present invention is to provide a product registration device and a program capable of detecting products whose packaging has been changed. [Means for solving the problem]

[0007] The product registration device of the embodiment includes a first acquisition means for acquiring an identifier capable of identifying the product from a code symbol attached to the product, a second acquisition means for acquiring features representative of the product's external characteristics from an image of the product, a third acquisition means for acquiring, for each of a plurality of products, product features corresponding to the identifier acquired by the first acquisition means from dictionary data that associates the product's identifier with the features representative of the product's external characteristics, and a detection means for detecting products whose appearance has been changed based on the features acquired by the first acquisition means and the features acquired by the third acquisition means. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a checkout system according to an embodiment. [Diagram 2] FIG. 2 is a side view showing an example of the configuration of a shopping cart according to an embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a hardware configuration of the terminal device according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a data configuration of the product master according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of a data configuration of a sales registration table according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a hardware configuration of a server according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a functional configuration of a terminal device and a server according to the embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of a sales registration process performed by the terminal device of the embodiment. [Figure 9] FIG. 9 is a sequence diagram illustrating an example of a data update process performed between the terminal device and the server according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, a product registration device and a program according to the present embodiment will be described in detail with reference to the accompanying drawings. In the following embodiment, a terminal device attached to a shopping cart (hereinafter, also referred to as a cart) for transporting products used in a store will be described as an example of the product registration device. Note that the invention is not limited to this application example.

[0010] Fig. 1 is a diagram showing an example of a checkout system 1 according to an embodiment. As shown in Fig. 1, the checkout system 1 includes a terminal device 2, a server 4, and an accounting device 8. The server 4 and the accounting device 8 are communicatively connected via a network Na.

[0011] The network Na may be the Internet, a virtual private network (VPN), a local area network (LAN), a public communication network, a mobile communication network, or the like, either alone or in appropriate combination. As an example, the network Na may be a combination of the Internet and a mobile communication network.

[0012] In addition, a wireless access point 7 is connected to the network Na. The wireless access point 7 is connected to the terminal device 2 via a network Nb such as a wireless LAN. Therefore, the terminal device 2 can communicate with the server 4 and the accounting device 8.

[0013] The cart 3 includes a storage section 31 (see FIG. 2) for storing products displayed in the store. The top of the storage section 31 is open, and products are placed into the storage section 31 from the top. A consumer (hereinafter also referred to as a customer) using the store borrows the cart 3 at the store, and while moving around the store with the cart 3, stores the products he or she wishes to purchase from the products displayed in the storage section 31.

[0014] The terminal device 2 is a terminal device such as a tablet terminal operated by a customer. In this embodiment, the terminal device 2 is attached to the cart 3 and is operated by a customer who uses the cart 3.

[0015] The terminal device 2 recognizes (hereinafter also referred to as "identification") the type of product inserted into the storage unit 31 from an image captured by the imaging unit 34 described below, and executes a sales registration process related to the product. Here, the sales registration process is also called product registration or the like, and refers to a process in which the product identified by the terminal device 2 is made the target for purchase. Specifically, the sales registration process refers to a process in which information related to the identified product is registered in the sales registration table 243 described below. Hereinafter, the information related to the product registered in the sales registration table 243 is also referred to as "sales registration data."

[0016] Furthermore, when the terminal device 2 accepts an operation to perform accounting for an item (hereinafter also referred to as payment), it executes a process to transfer the sales registration data for one transaction registered in the sales registration process to the accounting device 8. For example, the terminal device 2 encodes the sales registration data into a code symbol such as a two-dimensional code and displays it. In this case, the accounting device 8 reads the code symbol displayed on the terminal device 2 to obtain the sales registration data for one transaction registered in the terminal device 2, and executes accounting based on the sales registration data.

[0017] The server 4 is an example of a server device. The server 4 may be, for example, a store server installed in a store, or may be a server device installed outside the store. The server 4 of this embodiment manages dictionary data and the like used in the package recognition process described later.

[0018] In this embodiment, the server 4 is realized by a single device, but may be a cloud server (or a cloud system) realized by a plurality of devices connected via a network.

[0019] The accounting device 8 is an accounting device installed in a store. The accounting device 8 executes payment processing for products based on sales registration data. The payment processing includes displaying the total price and tax amount of products related to one transaction, calculating and displaying change based on the deposit received from the customer, and issuing a receipt on which product information and payment information (total price, deposit amount, change amount, etc.) are printed. Note that in this embodiment, the accounting device 8 is described as a self-service accounting device that allows a customer who has registered sales using the terminal device 2 to perform payment processing by themselves, but the accounting device is not limited to this and may be an accounting device operated by an employee such as a store clerk.

[0020] Next, the cart 3 will be described. Fig. 2 is a side view showing an example of the configuration of the cart 3. The cart 3 includes a basket-shaped storage section 31 with an open top, rotating casters 32 arranged below the storage section 31, and a handle section 33 connected to the storage section 31 and casters 32 and held by the customer when moving the cart 3.

[0021] The storage section 31 has a generally rectangular side and bottom surrounded by a mesh-like lattice 311 and is open at the top, so that the products stored in the storage section 31 can be visually observed from the outside. The side and bottom may be formed of walls instead of the mesh-like lattice 311.

[0022] The cart 3 also includes a terminal device 2 near the handle 33. The cart 3 also includes an imaging unit 34 near the edge of the opening of the storage unit 31. The imaging unit 34 is realized by an imaging device such as a digital camera. The imaging unit 34 is installed so that the imaging direction faces the opening of the storage unit 31. In other words, the imaging direction and imaging range of the imaging unit 34 are set so that it can capture images of products being put in and taken out of the storage unit 31. The imaging unit 34 may be configured to be included in the terminal device 2.

[0023] The imaging unit 34 captures images of merchandise stored in the storage unit 31 or merchandise taken out of the storage unit 31, and outputs the captured images to the terminal device 2. Here, the image means a still image or a video image in frame units.

[0024] The configuration of the cart 3 is not limited to the example shown in Fig. 2. For example, the number of imaging units 34 is not limited to one, and multiple imaging units may be provided. Furthermore, the terminal device 2 may be configured to be detachable from the cart 3. In this case, the terminal device 2 may be a mobile terminal (smartphone, tablet terminal, etc.) owned by the customer or rented by the store, which is attached to the cart 3.

[0025] From here, an example of the configuration of the main devices in the checkout system 1 will be described. First, the hardware configuration of the terminal device 2 will be described. FIG.

[0026] As shown in FIG. 3, the terminal device 2 includes a CPU (Central Processing Unit) 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a storage unit 24, a display unit 25, an operation unit 26, a communication unit 27, and a connection unit 28.

[0027] The CPU 21 is an example of a processor, and controls the overall operation of the terminal device 2. The ROM 22 stores various programs. The RAM 23 is a workspace in which programs and various data are expanded.

[0028] The CPU 21, the ROM 22, and the RAM 23 are connected via a bus or the like to constitute a computer-configured control unit 200. In the control unit 200, the CPU 21 operates in accordance with a program stored in the ROM 22 or the storage unit 24 and loaded in the RAM 23, thereby executing various types of processing.

[0029] The storage unit 24 has a storage medium such as a hard disk drive (HDD) or a flash memory, and maintains the stored contents even when the power is cut off. The storage unit 24 stores programs that can be executed by the CPU 21 and various setting information. For example, an application program for cooperating with the checkout system 1 is pre-stored (installed) in the storage unit 24.

[0030] The storage unit 24 also stores a product master 241, product recognition data 242, etc. The product master 241 is a data table for storing and managing product information of products sold in a store. Fig. 4 is a diagram showing an example of the data configuration of the product master 241. As shown in Fig. 4, the product master 241 stores product codes and product information in association with each other.

[0031] The product code is an example of identification information that can identify the type of product sold in a store. The product information is information about the product corresponding to the product code. The product information includes, for example, the product name, price, etc.

[0032] The product recognition data 242 is a data set for recognizing and identifying the type of a product based on the external characteristics of the product. Examples of the external characteristics of a product include the design, pattern, color, texture, and shape of the product or its exterior package. Hereinafter, the process of identifying a product code that can identify the type of product based on the external characteristics of the product is also referred to as a "package recognition process."

[0033] The product recognition data 242 includes, for example, a feature extraction model 2421 and a matching dictionary 2422 .

[0034] The feature extraction model 2421 is a trained model for extracting feature quantities (e.g., feature vectors) that represent the external characteristics of a commodity. When an image is input, the feature extraction model 2421 extracts and outputs feature quantities that represent the external characteristics of the item (commodity) depicted in the image. For example, a known technique such as a Convolutional Neural Network (CNN) can be used to extract the feature quantities.

[0035] The matching dictionary 2422 is dictionary data for identifying the type of product corresponding to the feature amount based on the feature amount of the product extracted from the image. For example, the matching dictionary 2422 stores product codes corresponding to the type of each product for multiple products sold in a store in association with feature amounts (hereinafter also referred to as matching feature amounts) that represent the external characteristics of the product.

[0036] In the package recognition process, the similarity (also called distance or likelihood) between the features is calculated by comparing the feature of the product extracted from the image using the feature extraction model 2421 with the matching feature of each product registered in the matching dictionary 2422. Then, in the package recognition process, the product code of the product registered in the matching dictionary 2422 whose feature similarity exceeds a threshold value (e.g., 90%) is identified as the type of the product that is the subject.

[0037] The storage unit 24 also stores a sales registration table 243. The sales registration table 243 is a data table for storing and managing products that have been sales-registered on the terminal device 2. FIG. 5 is a diagram showing an example of the data configuration of the sales registration table 243.

[0038] 5, the sales registration table 243 stores product codes, product information, points, etc. in association with each other. Here, among the data configuration of the sales registration table 243, for example, a set of product code, product name, product information, and point number corresponds to the sales registration data.

[0039] The product code is a product code that can identify the product registered for sales, and corresponds to the product code registered in the product master 241. The product information stores product information (product name, price, etc.) in the product master 241 that corresponds to the product code of the product registered for sales. The number of items refers to the number of products of the same type (same product code) that are registered for sales.

[0040] The data configuration of the sales registration table 243 is not limited to that shown in Fig. 5. For example, the sales registration table 243 may have a data configuration that does not hold product information.

[0041] Returning to FIG. 3, the display unit 25 is a display device such as an LCD (Liquid Crystal Display). The display unit 25 displays various information under the control of the CPU 21. The operation unit 26 is an input device such as a keyboard or a pointing device. The operation unit 26 outputs operation contents input via the input device to the CPU 21. The operation unit 26 may be a touch panel provided on the display unit 25.

[0042] The communication unit 27 is a wired or wireless communication interface connectable to the network Nb. The communication unit 27 is communicatively connected to the wireless access point 7 via the network Nb. The control unit 200 communicates with external devices such as the server 4 and the accounting device 8 via the communication unit 27.

[0043] The connection unit 28 is an interface for connecting a peripheral device. The connection unit 28 is a wired or wireless interface that complies with a short-range wireless communication standard such as USB (Universal Serial Bus) or Bluetooth (registered trademark). In this embodiment, the connection unit 28 is connected to the imaging unit 34.

[0044] Next, a description will be given of a hardware configuration of the server 4. FIG 6 is a diagram showing an example of a hardware configuration of the server 4.

[0045] As shown in FIG. 6, the server 4 includes a CPU 41, a ROM 42, a RAM 43, a storage unit 44, a communication unit 45, and the like.

[0046] The CPU 41 is an example of a processor, and controls the overall operation of the server 4. The ROM 42 stores various programs. The RAM 43 is a workspace in which programs and various data are expanded.

[0047] The CPU 41, the ROM 42, and the RAM 43 are connected via a bus or the like to constitute a computer-configured control unit 400. In the control unit 400, the CPU 41 operates in accordance with a program stored in the ROM 42 or the storage unit 44 and loaded in the RAM 43, thereby executing various types of processing.

[0048] The storage unit 44 has a storage medium such as a HDD or a flash memory, and maintains the stored contents even when the power is cut off. The storage unit 44 stores programs that can be executed by the CPU 41 and various setting information. For example, the storage unit 44 stores various programs for implementing server functions in the checkout system 1, such as a database program and a Web application program. The storage unit 44 may also store various programs for generating and updating the product recognition data 242.

[0049] Furthermore, the storage unit 44 stores a product master 441 and product recognition data 442. The product master 441 corresponds to the above-mentioned product master 241. For example, the server 4 provides the product master 441 to each of the terminal devices 2, and the product master 441 is stored in each of the terminal devices 2 as the product master 241.

[0050] Moreover, the commodity recognition data 442 corresponds to the above-mentioned commodity recognition data 242, and includes a feature extraction model 4421 and a matching dictionary 4422. For example, the server 4 provides the commodity recognition data 442 to each of the terminal devices 2, whereby the commodity recognition data 442 is stored in each of the terminal devices 2 as the commodity recognition data 242.

[0051] The storage unit 44 also has a data storage area 443. The data storage area 443 is a data area for storing package update information transmitted from the terminal device 2, which will be described later.

[0052] The communication unit 45 is a wired or wireless communication interface that can be connected to the network Na. The communication unit is communicatively connected to external devices such as the terminal device 2 and the accounting device 8 via the network Na. The control unit 400 communicates with external devices such as the terminal device 2 and the accounting device 8 via the communication unit 45.

[0053] Next, a description will be given of the functional configuration of the terminal device 2 and the server 4. Fig. 7 is a diagram showing an example of the functional configuration of the terminal device 2 and the server 4.

[0054] As shown in FIG. 7, the terminal device 2 includes an image acquisition unit 201, a product area detection unit 202, a code reading unit 203, a package recognition unit 204, a registration processing unit 205, and a payment processing unit 206 as functional components.

[0055] The functional configuration of the terminal device 2 is a software configuration realized by the processor of the terminal device 2 operating in accordance with a program stored in a ROM or memory unit, but is not limited to this. For example, some or all of the functional configuration of the terminal device 2 may be realized by hardware such as a dedicated circuit.

[0056] The image acquiring unit 201 acquires an image captured by the imaging unit 34 via the connection unit 28. The image acquired by the image acquiring unit 201 (hereinafter also referred to as a captured image) is temporarily stored in the RAM 23 for processing by the code reading unit 203 and the registration processing unit 205.

[0057] The product area detection unit 202 detects an area in which an object is represented from the captured image. Specifically, the product area detection unit 202 detects an area in which a product is represented (hereinafter also referred to as a product area) from the image acquired by the image acquisition unit 201, and outputs the product area or information representing the product area. Note that there is no particular limit to the method of detecting the product area, and a known object detection technology such as SSD (Single Shot multibox Detector) can be used.

[0058] Furthermore, the product area detection unit 202 determines the moving direction of the product by detecting the position and movement (e.g., motion vector) within the image of the product area detected from each of the images acquired continuously in time series by the image acquisition unit 201. Specifically, the product area detection unit 202 determines whether the product has been moved downward or upward.

[0059] For example, when a product is inserted into the storage unit 31, the product area detection unit 202 detects that the product has moved downward (hereinafter also referred to as the inserting direction). When a product is removed from the storage unit 31, the product area detection unit 202 detects that the product has moved upward (hereinafter also referred to as the removing direction).

[0060] The code reading unit 203 is an example of a first acquisition unit. The code reading unit 203 reads information from a code symbol such as a barcode or a two-dimensional code included in a captured image. Specifically, the code reading unit 203 executes a code reading process for reading a code symbol from the product area detected by the product area detection unit 202.

[0061] In the code reading process, the code reading unit 203 extracts an image representing a code symbol from the product area, and obtains information stored in the code symbol by decoding the extracted code symbol. For example, the code reading unit 203 obtains the product code of the product that is the subject by reading a code symbol attached to the product.

[0062] The package recognition unit 204 is an example of a second acquisition means, a third acquisition means, a fourth acquisition means, a detection means, and an output means. The package recognition unit 204 uses the product recognition data 242 to execute package recognition processing on the captured image, thereby recognizing the type of product depicted in the captured image.

[0063] Specifically, the package recognition unit 204 extracts (acquires) features of the product depicted in the product area from the product area by executing package recognition processing on the product area detected by the product area detection unit 202. Next, the package recognition unit 204 compares the extracted features with matching features of each product registered in the matching dictionary 2422 to obtain product codes of products whose similarity exceeds a threshold value from the matching dictionary 2422, and outputs the product codes as recognition results.

[0064] The registration processing unit 205 is an example of a registration means. The registration processing unit 205 executes a sales registration process or a registration cancellation process for a product based on the movement direction of the product determined by the product area detection unit 202 and the product code read by the code reading unit 203 or the product code recognized by the package recognition unit 204.

[0065] When a product code is acquired by the code reading unit 203 or the package recognition unit 204 and movement in the insertion direction is detected by the product area detection unit 202, the registration processing unit 205 executes a sales registration process for registering a product corresponding to the product code in the sales registration table 243. Specifically, the registration processing unit 205 reads product information corresponding to the product code acquired by the code reading unit 203 or the package recognition unit 204 from the product master 241, associates it with the product code and a score (initial value is 1), and registers it in the sales registration table 243. In addition, when a product code already registered in the sales registration table 243 is acquired by the code reading unit 203 or the package recognition unit 204 and movement in the insertion direction is detected by the product area detection unit 202, the registration processing unit 205 adds 1 to the score of the corresponding sales registration data.

[0066] Furthermore, when a product code is acquired by the code reading unit 203 or the package recognition unit 204 and movement in the insertion direction is detected by the product area detection unit 202, the registration processing unit 205 executes processing to cancel the sales registration of the product corresponding to the product code. Specifically, the registration processing unit 205 searches the sales registration data corresponding to the acquired product code from the sales registration table 243, and subtracts 1 from the score of the searched sales registration data. Furthermore, when the score becomes 0 as a result of the subtraction, the registration processing unit 205 deletes the sales registration data from the sales registration table 243.

[0067] Furthermore, every time the registration processing unit 205 updates the sales registration table 243, it causes the display unit 25 to display a screen based on the sales registration data registered in the sales registration table 243. For example, the registration processing unit 205 causes the display unit 25 to display a screen showing a list of registered sales products.

[0068] In this way, in the checkout system 1 of this embodiment, when a customer puts an item into the storage section 31 of the cart 3, the sales registration of that item is automatically performed. Also, in the checkout system 1, when a customer removes an item from the storage section 31 of the cart 3, the registration of that item is automatically canceled. In other words, the checkout system 1 is a mobile sales floor check system that allows customers to register items themselves while moving around the store.

[0069] The payment processor 206 executes processing to settle the sales of registered products in response to customer operations. Specifically, the payment processor 206 executes processing to transfer sales registration data of registered products to the accounting device 8, thereby realizing a state in which the accounting device 8 can settle the sales.

[0070] For example, when the payment processing unit 206 receives an operation to end sales registration or an operation to start payment (collectively referred to as a closing operation) via the operation unit 26, it transmits the sales registration data registered in the sales registration table 243 to the accounting device 8 via short-range wireless communication such as Bluetooth (registered trademark).

[0071] Also, for example, the payment processing unit 206 may send the sales registration data registered in the sales registration table 243 to the accounting device 8 via the server 4. In this case, the payment processing unit 206 sends the sales registration data registered in the sales registration table 243 to the server 4 together with an identification ID capable of identifying its own terminal device 2 or an identification ID capable of identifying the transaction. The payment processing unit 206 also sends the identification ID to the accounting device 8 of the transaction destination via short-range wireless communication. The accounting device 8 then queries the server 4 about the identification ID sent from the payment processing unit 206, thereby obtaining the sales registration data corresponding to the identification ID.

[0072] It should be noted that the method of notifying the accounting device 8 of the identification ID is not limited to short-range wireless communication. For example, the payment processing unit 206 may encode the identification ID into a code symbol such as a two-dimensional code and display it on the display unit 25, thereby notifying the accounting device 8 of the identification ID via the code symbol. In this case, the accounting device 8 can obtain the identification ID from the terminal device 2 by reading the code symbol displayed on the display unit 25.

[0073] Furthermore, for example, the payment processing unit 206 may transmit the identification ID of its own terminal device 2 to the accounting device 8 by short-range wireless communication such as Bluetooth (registered trademark). In this case, the accounting device 8 obtains from the server 4 the sales registration data corresponding to the identification ID transmitted from the terminal device 2, and performs accounting for the products for one transaction that have been registered as sales, based on the obtained sales registration data.

[0074] The payment for the products is not limited to the method using the accounting device 8, and may be made by the terminal device 2. For example, when the payment processing unit 206 receives an operation to instruct the start of payment via the operation unit 26, the payment processing unit 206 displays a screen on the display unit 25 that allows the user to select a payment method such as electronic money payment or credit card payment. When the payment method is selected, the payment processing unit 206 receives input of payment information for making the payment using the selected payment method, and executes a payment process to settle the total amount of the products registered for sale based on the input payment information by cooperating with an external payment server. Such payment information may be, for example, a credit card number or a two-dimensional code for code payment. The payment processing unit 206 may also receive input of a two-dimensional code for code payment via the imaging unit 34.

[0075] The payment for the products is not limited to the method using the accounting device 8, and may be made by the terminal device 2. For example, when the payment processing unit 206 receives an operation to instruct the start of payment via the operation unit 26, the payment processing unit 206 displays a screen on the display unit 25 that allows the user to select a payment method such as electronic money payment or credit card payment. When the payment method is selected, the payment processing unit 206 receives input of payment information for making the payment using the selected payment method, and executes a payment process to settle the total amount of the products registered for sale based on the input payment information by cooperating with an external payment server. Such payment information may be, for example, a credit card number or a two-dimensional code for code payment. The payment processing unit 206 may also receive input of a two-dimensional code for code payment via the imaging unit 34.

[0076] Incidentally, the packaging of products sold in stores may change at the timing of seasonal changes, campaigns, etc. When the packaging of a product changes, the external characteristics also change, and there is a possibility that the product may not be correctly recognized. Therefore, when the packaging of a product is changed, an update process is required to reflect the feature amount of the product in the matching dictionary 2422.

[0077] However, there are cases where the store does not notice the change in the package, and the system continues to be used without updating the comparison dictionary 2422. In particular, in the case of the present embodiment, a mobile sales floor checkout system, in which the customer registers the product himself, has a problem that it is difficult for the store clerk to notice the change in the package.

[0078] Therefore, in the terminal device 2 of this embodiment, the package recognition unit 204 executes a process for detecting a product whose package has been changed, based on the product code read by the code reading unit 203.

[0079] Specifically, before recognizing a product code by package recognition, when a product code is read by the code reading unit 203, the package recognition unit 204 reads (acquires) a matching feature of the product corresponding to the product code from the matching dictionary 2422. Next, the package recognition unit 204 compares the product feature extracted from the image by package recognition with the matching feature read from the matching dictionary 2422 to detect products whose packaging has been changed.

[0080] Here, if the similarity between the product feature extracted from the image by package recognition and the matching feature read from the matching dictionary 2422 is equal to or greater than a threshold, the package recognition unit 204 determines that the package has not been changed. If the similarity between the product feature extracted from the image by package recognition and the matching feature read from the matching dictionary 2422 is less than the threshold, the package recognition unit 204 determines that a product with changed packaging has been detected.

[0081] When the package recognition unit 204 detects a product whose package has been changed, it executes a process for notifying or reporting the product.

[0082] For example, the package recognition unit 204 outputs information including the product code read by the code reading unit 203 (hereinafter also referred to as package change information). Here, output of package change information means transmission to an external device in this embodiment, but is not limited to this and is a concept that includes other output forms such as display output, print output, file output, etc. In this embodiment, an example will be described in which the package change information is sent to the server 4 as the output destination.

[0083] Also, for example, the package recognition unit 204 may include an image of the image area detected by the product area detection unit 202 (hereinafter also simply referred to as an image area) together with the product code read by the code reading unit 203 in the package change information and transmit it to the server 4. Also, for example, the package recognition unit 204 may include a feature extracted from the image area together with the product code read by the code reading unit 203 in the package change information and transmit it to the server 4. In this case, the product code, image area and / or feature included in the package change information can be used as update data for updating dictionary data for a product whose package has been changed. In this embodiment, a form in which update data is included in and transmitted in package change information will be described.

[0084] It should be noted that there is no particular limit to the timing for outputting the package change information. For example, the package recognition unit 204 may output the package change information every time it detects a product whose package has been changed. Furthermore, the package recognition unit 204 may store the package change information relating to the product in the RAM 23 or the storage unit 24 every time it detects a product whose package has been changed, and transmit the stored package change information to the server 4 at a predetermined timing. For example, the package recognition unit 204 may transmit the package change information at a predetermined time such as every hour, or when it has detected a package change a predetermined number of times (for example, 10 times).

[0085] Furthermore, the destination of the package change information is not limited to the server 4 and may be another device. For example, the package recognition unit 204 may transmit the package change information to a terminal device operated by a store clerk (hereinafter also referred to as a store clerk terminal). Furthermore, the package recognition unit 204 may transmit the package change information including all or a part of the product information (e.g., product name) corresponding to the product code. When the store clerk terminal receives the package change information, it notifies the store clerk of the product whose package has been changed by displaying on a display device the product code and image area included in the package change information, the product name of the product corresponding to the product code, etc.

[0086] On the other hand, as shown in FIG. 7, the server 4 includes an update notification receiving unit 401, an update processing unit 402, and a data providing unit 403 as functional components.

[0087] The functional configuration of server 4 is a software configuration realized by the processor of server 4 operating in accordance with a program stored in a ROM or memory unit, but is not limited to this. For example, some or all of the functional configuration of server 4 may be realized by hardware such as a dedicated circuit.

[0088] The update notification receiving unit 401 receives package update information transmitted from the terminal device 2 via the communication unit 45. Upon receiving the package update information, the update notification receiving unit 401 stores the update data included in the package update information in the data storage area 443.

[0089] When the update notification receiving unit 401 receives package update information, the update notification receiving unit 401 may transmit to the store clerk terminal a notification that the package of the product corresponding to the product code included in the package update information has been updated. For example, the update notification receiving unit 401 may transmit the received package update information to the store clerk terminal.

[0090] The update processing unit 402 updates the matching dictionary 2422 related to package recognition of the product whose package has been updated. Specifically, the update processing unit 402 performs processing for updating the correspondence between the feature amount of the product and the product code based on the update data stored in the data storage area 443.

[0091] For example, when the update data includes an image region and a product code, the update processing unit 402 extracts a feature from the image region using the feature extraction model 4421, and updates the feature of the matching dictionary 4422 corresponding to the product code specified in the update data based on the extracted feature. Also, for example, when the update data includes a feature and a product code, the update processing unit 402 updates the feature of the matching dictionary 4422 corresponding to the product code specified in the update data based on the feature specified in the update data. There is no particular limit to the method of updating the matching dictionary 4422, and for example, a known technique such as fine tuning in the field of machine learning can be used.

[0092] The update process of the matching dictionary 4422 is performed for each product code, but the timing of the process is not limited. For example, the update processing unit 402 may perform the update process each time update data is stored in the data storage area 443, or may perform the update process at a predetermined timing. In the latter case, for example, the update processing unit 402 may perform the update process when a predetermined number (e.g., 10 pieces) of update data for the same product code are stored in the data storage area 443. Also, for example, the update processing unit 402 may perform the update process in response to an instruction from a store terminal.

[0093] The data providing unit 403 provides various data to each of the terminal devices 2. Specifically, when the matching dictionary 4422 is updated by the update processing unit 402, the data providing unit 403 provides the updated matching dictionary 4422 to each of the terminal devices 2. Note that the data providing unit 403 may provide each of the terminal devices 2 with data of the difference between the existing matching dictionary 4422 and the updated matching dictionary 4422 as a new matching dictionary 4422. When the matching dictionary 4422 is provided from the data providing unit 403, the control unit 200 of the terminal device 2 updates the matching dictionary 2422 in the storage unit 24 based on the matching dictionary 4422.

[0094] Even if the matching dictionary 4422 has not been updated, the data providing unit 403 may provide the matching dictionary 4422 to each of the terminal devices 2 at a predetermined timing.

[0095] The timing when the data providing unit 403 provides the matching dictionary 4422 is not particularly limited, and the data may be provided at a predetermined time, for example. The matching dictionary 4422 may provide data in response to a request from the terminal device 2. The data providing unit 403 may provide data when the data is updated. The data providing unit 403 may provide data in response to an instruction from a store clerk terminal.

[0096] The data providing unit 403 may also provide data other than the matching dictionary 4422 (for example, the product master 441 and the feature extraction model 4421) to each of the terminal devices 2 at a predetermined timing. In this case, the control unit 200 of the terminal device 2 updates the corresponding data stored in the storage unit 24 based on the data provided from the data providing unit 403.

[0097] The following describes an example of the operation of the checkout system 1. First, an example of sales registration processing performed by the terminal device 2 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of sales registration processing performed by the terminal device 2.

[0098] First, the image acquisition unit 201 of the terminal device 2 sequentially acquires images captured by the imaging unit 34 (step S11). Next, the product area detection unit 202 determines whether or not a product area has been detected from the image captured in step S11 (step S12). If a product area is not detected (step S12; No), the process proceeds to step S26. If a product area is detected (step S12; Yes), the product area detection unit 202 determines the moving direction of the product area (product) from the images acquired consecutively in time series (step S13).

[0099] Next, the code reading unit 203 executes a code reading process to read a code symbol (product code) from the product area detected by the product area detection unit 202 (step S14). In addition, the package recognition unit 204 executes a feature extraction process to extract features from the product area detected by the product area detection unit 202 (step S15).

[0100] Next, the package recognition unit 204 determines whether or not the product code has been read in the code reading process of step S14 (step S16). If the product code has not been read (step S16; No), the package recognition unit 204 executes a matching process to match the feature amount extracted in the feature extraction process of step S17 with the feature amount of each product registered in the matching dictionary 2422 (step S17). When the package recognition unit 204 identifies the product code of the product whose similarity exceeds a threshold as a result of the matching process (step S18), the process proceeds to step S23.

[0101] On the other hand, if it is determined in step S16 that the product code has been read (step S16; Yes), the package recognition unit 204 reads and acquires the matching feature of the product corresponding to the product code from the matching dictionary 2422 (step S19). Next, the package recognition unit 204 calculates the similarity by comparing the feature extracted in the feature extraction process in step S15 with the matching feature read in step S19 (step S20). Next, the package recognition unit 204 determines whether the calculated similarity exceeds a threshold value (step S21).

[0102] Here, if the similarity exceeds the threshold value (step S21; Yes), the package recognition unit 204 determines that the package has not been changed, and proceeds to step S23.

[0103] Furthermore, if the similarity is equal to or lower than the threshold value (step S21; No), the package recognition unit 204 determines that a product whose package has been changed has been detected. In this case, the package recognition unit 204 transmits package change information including the product code read in step S14 and the image of the product area detected in step S12 to the server 4 (step S22), and proceeds to step S23. When the update notification receiving unit 401 of the server 4 receives the package change information from the terminal device 2, it stores the received package change information in the data storage area 443.

[0104] In the next step S23, the registration processing unit 205 determines whether the movement direction determined in step S13 was the inserting direction or the removing direction (step S23).

[0105] Here, if the movement direction is the insert direction (step S23; Yes), the registration processing unit 205 executes a sales registration process (step S24) to register the product corresponding to the product code acquired in step S14 or step S18 in the sales registration table 243 based on the product code. If the movement direction is the remove direction (step S23; No), the registration processing unit 205 executes a registration cancellation process (step S25) to delete the product corresponding to the product code acquired in step S14 or step S18 from the sales registration table 243.

[0106] Next, the registration processing unit 205 (or the payment processing unit 206) determines whether or not a closing operation has been accepted (step S26). If the closing operation has not been accepted (step S26; No), the process returns to step S11. If the closing operation has been accepted (step S26; Yes), the payment processing unit 206 executes a process to make a payment for the products registered in the sales registration table 243 (step S27).

[0107] This allows the terminal device 2 to perform sales registration or cancellation of a product by using either the product code read from the code symbol or the product code identified from the external characteristics of the product. Also, the terminal device 2 can perform detection processing of products whose packaging has been changed during the process of sales registration or cancellation of registration of a product.

[0108] Next, an example of data update processing performed between the terminal device 2 and the server 4 will be described with reference to Fig. 9. Fig. 9 is a sequence diagram showing an example of data update processing performed between the terminal device 2 and the server 4. Note that, as a premise of this processing, it is assumed that package change information is stored in the data storage area 443 of the server 4. In addition, it is assumed that the package change information includes a product area and a product code.

[0109] First, the update processing unit 402 of the server 4 classifies the package change information stored in the data storage area 443 by the type of product code included in the package information (step S31). Next, the update processing unit 402 extracts features related to the product code of the classification from the product area included in the classified package change information (step S32). Next, the update processing unit 402 updates the matching features in the matching dictionary 4422 registered in association with the product code based on a pair of the features extracted in step S32 and the classified product code (step S33).

[0110] Next, the data providing unit 403 provides the updated matching dictionary 4422 to each of the terminal devices 2 (step S34). On the other hand, upon receiving the matching dictionary 4422 from the server 4, the control unit 200 of the terminal device 2 updates the matching dictionary 2422 based on the matching dictionary 4422 (step S35).

[0111] This enables each of the terminal devices 2 to perform package recognition processing using the updated comparison dictionary 2422 that reflects the feature amounts of the product whose package has been updated.

[0112] As described above, the terminal device 2 of this embodiment acquires a product code from a code symbol attached to a product, and acquires features from an image of the product. The terminal device 2 also acquires, for each of a plurality of products, a matching feature of the product corresponding to the product code read from the code symbol from the matching dictionary 2422 that associates the product code of the product with a matching feature, and compares it with the feature acquired from the image, thereby executing a process for detecting a product whose appearance has been changed. In this way, the terminal device 2 can efficiently detect a product whose packaging has been changed by comparing the feature derived from the image of a product having the same product code with the matching feature in the matching dictionary 2422, based on the similarity of the feature.

[0113] Furthermore, when the similarity between the image-derived feature of a product related to the same product code and the matching feature of the matching dictionary 2422 is equal to or less than a threshold, the terminal device 2 detects the product as having a changed appearance. This allows the terminal device 2 to identify products whose appearance features have changed, and therefore efficiently detect products whose packaging has been changed.

[0114] Furthermore, when the terminal device 2 detects a product whose packaging has been changed, it outputs package change information including the product code of the product. This allows the terminal device 2 to notify the output destination device of the product whose packaging has been changed, and therefore to provide support for measures such as updating dictionary data.

[0115] Furthermore, when the terminal device 2 detects a product whose packaging has been changed, it outputs package update information including the product code of the product and an image of the product area including the product. This allows the terminal device 2 to provide the output destination device with data that can be used to update dictionary data for the product whose packaging has been changed, thereby providing dictionary data update support.

[0116] In addition, the terminal device 2 detects products whose packaging has been changed during the process of a customer's product sales registration (or registration cancellation), eliminating the need for a store clerk to check the packaging. In addition, the terminal device 2 can acquire (collect) data that can be used to update the dictionary data in each terminal device 2, thereby efficiently supporting the updating of the dictionary data.

[0117] The above-described embodiment can be modified as appropriate by changing a part of the configuration or function of each of the above-described devices. Therefore, some modified examples of the above-described embodiment will be described below as other embodiments. Note that the following mainly describes the differences from the above-described embodiment, and a detailed description of the common points with the contents already described will be omitted. The modified examples described below may be implemented individually or in appropriate combination.

[0118] (Variation 1) In the above embodiment, the terminal device 2 is described as being attached to the cart 3, but the form of the terminal device 2 is not limited to this. For example, the terminal device 2 may be a mobile terminal such as a smartphone carried by a customer, or a mobile terminal such as a smartphone or tablet terminal lent to a customer by a store.

[0119] In this case, the terminal device 2 may be configured to cooperate with the cart 3 by communicating with the imaging unit 34 via wired or wireless communication, or may be configured to be independent of the cart 3. In the latter case, the customer can register the sale of a product as in the above-mentioned embodiment by capturing an image of the product with the camera provided in the terminal device 2. When canceling the registration, the customer can select the product to be canceled from the list of registered products displayed on the screen of the terminal device 2, thereby canceling the registration as in the above-mentioned embodiment.

[0120] (Variation 2) In the above embodiment, the terminal device 2 is configured to perform the package recognition process, but the present invention is not limited to this, and an external device for package recognition process may perform the process.

[0121] In this case, the package recognition unit 204 of the terminal device 2 realizes the same functions as those of the above-described embodiment by working in cooperation with an external device for package recognition processing. Specifically, the package recognition unit 204 transmits an image of the image area detected by the product area detection unit 202 to the external device for package recognition processing, thereby causing the external device to execute package recognition processing, and obtains the processing result of the package recognition processing, i.e., the product recognition result, from the external device.

[0122] Furthermore, if a product code is read by the code reading unit 203 before obtaining a recognition result from an external device, the package recognition unit 204 notifies the external device of the product code, thereby instructing the external device to compare the feature extracted from the image area with the matching feature related to the notified product code. In this case, the package recognition unit 204 obtains the comparison result, i.e., the similarity, from the external device and determines whether the similarity exceeds a threshold. If the similarity is equal to or less than the threshold, the package recognition unit 204 generates package update information including the image area and the product code, and transmits it to the server 4 or a store clerk terminal.

[0123] In the configuration of this modified example, since the external device has the commodity recognition data 242, the matching dictionary 2422 of the external device is updated using the matching dictionary 4422 updated by the server 4.

[0124] As a result, in the checkout system 1 according to this modified example, the external device performs the package recognition process, which reduces the processing load on the terminal device 2 and provides the same effects as those of the above-described embodiment.

[0125] Furthermore, in checkout system 1 according to this modification, an external device can centrally execute package recognition processing for multiple terminal devices 2, and therefore by updating matching dictionary 2422 of the external device, improved accuracy of package recognition processing can be reflected in multiple terminal devices 2. Therefore, in checkout system 1 according to this modification, it is possible to improve the efficiency of management of matching dictionary 2422.

[0126] (Variation 3) In the above-described embodiment, the code symbol (product code) attached to the product is read from the image captured by the imaging unit 34, but the present invention is not limited to this and may be configured to use, for example, a code reading device specialized in reading code symbols.

[0127] In this case, the code reading device may be built into the terminal device 2, or may be connected to the terminal device 2 by wire or wirelessly. The code reading unit 203 of the terminal device 2 cooperates with the code reading device to obtain information on the code symbol read by the code reading device (e.g., a product code).

[0128] As a result, even if the checkout system 1 according to this modified example is configured to read the code symbol using a code reader, it can achieve the same effects as the above embodiment.

[0129] The programs executed by each device in the above-described embodiments (and the modified examples) are provided in a state where they are pre-installed in a ROM, a storage unit, etc. The programs executed by each device in the above-described embodiments may be provided by being recorded in a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD) in the form of an installable or executable file.

[0130] Furthermore, the programs executed by each device of the above-described embodiments may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the programs executed by each device of the above-described embodiments may be provided or distributed via a network such as the Internet.

[0131] Although the embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. These new embodiments and their modifications can be implemented in various other forms, and various omissions, substitutions, changes, and combinations can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0132] 1. Checkout System 2 Terminal Equipment 3 Cart (Shopping Cart) 4 Server 8. Accounting Equipment 200 Control section 201 Image Acquisition Unit 202 Product area detection unit 203 Code reader 204 Package Recognition Unit 2421 Feature Extraction Model 2422 Matching Dictionary 400 Control section 401 Update Notification Reception Department 402 Update processing section 403 Data Provision Department [Prior art documents] [Patent documents]

[0133] [Patent Document 1] Patent Publication No. 2022-29170

Claims

1. a first acquiring means for acquiring an identifier capable of identifying a product from a code symbol attached to the product; A second acquisition means for acquiring a feature quantity representing an external feature of the commodity from an image of the commodity; a third acquiring means for acquiring, for each of a plurality of products, a feature amount of the product corresponding to the identifier acquired by the first acquiring means from dictionary data in which the identifier of the product is associated with a feature amount representing an appearance feature of the product; a detection means for detecting a product whose appearance has been changed based on the feature amount acquired by the first acquisition means and the feature amount acquired by the third acquisition means; A product registration device comprising:

2. The detection means detects the product as having an appearance changed when a similarity between the feature amount acquired by the first acquisition means and the feature amount acquired by the third acquisition means is equal to or less than a threshold. The product registration device according to claim 1.

3. an output means for outputting information including an identifier of a product when the detection means detects a product whose appearance has been changed; The product registration device according to claim 1 or 2, further comprising:

4. The output means outputs the information including an identifier of the product and the image including the product. The product registration device according to claim 3.

5. a fourth acquiring means for acquiring, from the dictionary data, an identifier of a product whose similarity reaches a threshold value by comparing the feature amount acquired by the second acquiring means with the feature amount of each product registered in the dictionary data; a registration means for registering a product corresponding to the identifier acquired by the first acquisition means or the identifier acquired by the fourth acquisition means as a purchase target, based on the identifier; The product registration device according to claim 1 , further comprising:

6. The computer of the product registration device, a first acquiring means for acquiring an identifier capable of identifying a product from a code symbol attached to the product; A second acquisition means for acquiring a feature quantity representing an external feature of the commodity from an image of the commodity; a third acquiring means for acquiring, for each of a plurality of products, a feature amount of the product corresponding to the identifier acquired by the first acquiring means from dictionary data in which the identifier of the product is associated with a feature amount representing an appearance feature of the product; a detection means for detecting a product whose appearance has been changed based on the feature amount acquired by the first acquisition means and the feature amount acquired by the third acquisition means; A program to make it function as such.

Images