Information processing device and program

Abstract

To provide practical reliability by expanding the range of a product to be identified by image recognition, and to improve practicality in the operation of a system for providing technique.SOLUTION: An information processing apparatus includes: a code information acquisition section that acquires code information which is obtained by reading a code symbol displayed on a package of a product and enables the product to be identified; an image data acquisition section that acquires image data on the product imaged when reading the code symbol; a feature information acquisition section that acquires feature information which is information indicating a feature of an appearance of the product from the image data; a dictionary output section that outputs a dictionary for collation in which one or more pieces of the feature information are stored in association with one piece of the code information; and a master image generation section that generates a product image to be stored in association with the code information on the basis of the image data.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an information processing device and a program. [Background technology]

[0002] In sales data processing, a technology is used to identify products using information indicating the external characteristics (shape, color, pattern, etc.) of an object obtained by image recognition (for example, Patent Document 1). Currently, this technology is mainly used to identify products that do not have code symbols such as barcodes, such as fruits and vegetables. For other products that have code symbols on their packaging, products are identified using code information obtained by reading the code symbols, as in the past.

[0003] In recent years, attempts have been made to expand the scope of product identification using image recognition to include products with code symbols, and to identify products based on the external characteristics of the package, rather than relying on reading the code symbol.Identifying packaged products using image recognition poses a challenge in terms of increasing reliability (accuracy), and efforts are being made to make this technology practical. Summary of the Invention [Problem to be solved by the invention]

[0004] The problem that the present invention aims to solve is to expand the range of products that can be identified by image recognition, to enable practical reliability to be provided, and to increase the practicality of the operation of the system that provides the technology. [Means for solving the problem]

[0005] An information processing device according to an embodiment includes a code information acquisition unit that acquires code information that enables a product to be identified by reading a code symbol displayed on a product package; an image data acquisition unit that acquires image data of the product captured when the code symbol is read; a feature information acquisition unit that acquires feature information that indicates external features of the product from the image data; a dictionary output unit that outputs a comparison dictionary that associates one or more pieces of feature information with one piece of code information and stores the associated information; and a master image creation unit that creates a product image to be stored in association with the code information, based on the image data. The master image creation unit selects, as the product image, from the image data, an area that includes a product name. do. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a product registration system according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of a POS terminal. [Figure 3] FIG. 3 is a diagram showing examples of items included in the product master. [Figure 4] FIG. 4 is a diagram showing examples of items included in the registered product information. [Figure 5] FIG. 5 is a block diagram showing an example of the configuration of a code scanner. [Figure 6] FIG. 6 is a block diagram showing an example of the configuration of the verification terminal. [Figure 7] FIG. 7 is a diagram showing examples of items included in the matching dictionary. [Figure 8] FIG. 8 is a diagram showing examples of items included in the image information. [Figure 9] FIG. 9 is a block diagram showing an example of the configuration of the store server. [Figure 10] FIG. 10 is a block diagram illustrating an example of the configuration of the server. [Figure 11] FIG. 11 is a diagram showing examples of items included in the product master. [Figure 12] FIG. 12 is a diagram showing examples of items included in the transaction information. [Figure 13] FIG. 13 is a diagram showing an example of various functional units included in each control unit and the correspondence between them. [Figure 14] FIG. 14 is a diagram showing the operation timing and relationship of each functional unit. [Figure 15] FIG. 15 is a diagram showing the operation timing and relationship of each functional unit. [Figure 16] FIG. 16 is a diagram showing the process by the master image creation unit in more detail. DETAILED DESCRIPTION OF THE INVENTION

[0007] (First embodiment) An embodiment will be described with reference to the drawings. Fig. 1 is a diagram showing an example of the configuration of a product registration system 100 according to the embodiment. The product registration system 100 includes a server 1, a store server 3, a POS terminal 5, a code scanner 6, and a verification terminal 7. The server 1, the store server 3, and the POS terminal 5 are examples of information processing devices.

[0008] In order for the product registration system 100 to correctly identify products from captured images, it is necessary to enrich the dictionary for inferring products from images. To do this, it is necessary to collect many images related to the products. Furthermore, for example, when a new product is released or when the package design is changed for a limited time even if the product remains unchanged, the feature information obtained from the captured image will no longer match the matching data. For this reason, it is necessary to continuously update the dictionary to accommodate new packages.

[0009] Since new packages appear every day, relying on manual updating of the dictionary is not desirable in terms of preventing omissions in the dictionary, and the number of entries would be impossible for a large-scale store. Therefore, in this embodiment, the dictionary is configured to be updated automatically and learning-based, automating the process of responding to new packages.

[0010] In the process of identifying a product by comparing feature information with matching data, the user may be asked to confirm whether the product identification is correct, or several candidates may be presented and the user may be asked to select one. In such cases, to make it easier for the user to make a decision, it is preferable to display an image of a product that is estimated to match the feature information rather than displaying only text information. Therefore, this embodiment proposes a configuration for obtaining an image suitable for presentation to the user during user confirmation, in parallel with the above-mentioned automatic dictionary learning.

[0011] The server 1 can communicate with a store server 3 via a network 2 to send and receive information. The store server 3 is communicably connected to one or more POS terminals 5 via a network 9 such as a LAN (Local Area Network), to send and receive information.

[0012] One code scanner 6 and one verification terminal 7 are provided for each POS terminal 5. The verification terminal 7 is connected to the POS terminal 5 via the code scanner 6. The code scanner 6 also relays communication between the verification terminal 7 and the POS terminal 5. The verification terminal 7 is connected to the store server 3 via a network 9 and is capable of communicating with the store server 3. The code scanner 6 and the POS terminal 5 are connected via USB, and the code scanner 6 and the verification terminal 7 are connected via a LAN.

[0013] The POS terminal 5 receives input from the code scanner 6, registers the product, and makes payment for the registered product. The code scanner 6 captures an image of the product's exterior, and outputs the read result (code information) of a code symbol such as a barcode or two-dimensional code, as well as information (code information) input from the verification terminal 7, to the POS terminal 5.

[0014] The verification terminal 7 obtains information (feature information) indicating the external characteristics (shape, color, pattern, etc.) of the product from the data of the image captured by the code scanner 6, and returns information (code information) that can identify a product that matches this information to the code scanner 6. The verification terminal 7 also associates data (image data) of the image captured by the code scanner 6 (captured image) with the read result of the code symbol output by the code scanner 6 (code information), and provides this to the store server 3.

[0015] The store server 3 is an information processing device such as a PC (Personal Computer), and is installed in the back room of the store, and is used for tasks such as monitoring the operation of the POS terminals 5 and other devices installed in the store, and managing information stored in the in-store devices such as the POS terminals 5. The store server 3 also provides store staff with a UI (User Interface) for performing the above tasks.

[0016] The store server 3 accumulates information on the paid product (transaction information) sent from the POS terminal 5 and transfers it to the server 1. The store server 3 also transfers a set of image data and code information sent from the verification terminal 7 to the server 1. The store server 3 also receives product master data and a verification dictionary suitable for its own store from the server 1 and deploys them to each POS terminal 5 and verification terminal 7 in its own store.

[0017] The server 1 accumulates pairs of image data and code information acquired from each store via the store server 3, and generates verification data to be associated with the code information based on the pairs.

[0018] 2 is a block diagram showing an example of the configuration of the POS terminal 5. The POS terminal 5 includes a CPU (Central Processing Unit) 51, a ROM (Read Only Memory) 52, a RAM (Random Access Memory) 53, a communication unit 54, a display unit 55, an operation unit 56, an audio output unit 57, a cash deposit / withdrawal unit 58, a storage unit 59, etc.

[0019] The CPU 51 is an example of a processor and controls all the components of the POS terminal 5. The ROM 52 stores various programs. The RAM 53 is a workspace for storing programs and various data. The CPU 51, ROM 52, and RAM 53 are connected via a bus or the like to form a control unit 50 having a computer configuration.

[0020] The communication unit 54 is a communication interface that connects the control unit 50 and an external device (for example, the store server 3) so that they can communicate with each other via the network 9. The communication unit 54 also accepts input of code information output by the USB-connected code scanner 6.

[0021] The display unit 55 has a display device such as an LCD (Liquid Crystal Display), and displays various information under the control of the CPU 51.

[0022] The operation unit 56 has input devices such as a touch panel provided over the surface of the display unit 55, a keyboard, and a pointing device, and outputs operation contents input via the input devices to the CPU 51.

[0023] The audio output unit 57 has a speaker and a buzzer, and outputs audio under the control of the CPU 51.

[0024] The deposit / withdrawal unit 58 is, for example, a change machine that stores cash (coins and bills) and receives deposits and dispenses change during settlement.

[0025] The storage unit 59 has a storage medium such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory, and maintains its stored contents even when power is cut off. The storage unit 59 stores various setting information such as a program 591 executable by the CPU 51 and a product master 592. The storage unit 59 (or RAM 53) also stores registered product information 593, which is information on registered products (registered products), from the start to the completion of at least one transaction.

[0026] 3 is a diagram showing examples of items included in the product master 592. The product master 592 is an example of product information, and is information about products handled by a store compiled, for example, in the form of a table. Items of product information include, for example, "product code," "product name," "unit price," "color," and "product image."

[0027] The product code is an example of information that can identify a product, such as a JAN code. Other information (unit price, verification data, color, compatible products, etc.) is stored in association with the product code.

[0028] The "product name" field records the name of the product. The "unit price" field records the price for one product. The "color" field records information indicating the color and pattern of the product.

[0029] "Product image" is image data that is added as information to explain the product when the product of the product code is displayed as an option.

[0030] Fig. 4 is a diagram showing examples of items included in registered product information 593. Registered product information 593 is information on products registered for purchase by customers (i.e., products subject to transaction) compiled, for example, in the form of a table. Product information items include, for example, "product code," "unit price," and "quantity." "Quantity" is the number of products purchased.

[0031] The program 591 is application software for causing the POS terminal 5 to function. POS is an abbreviation for "Point Of Sale" and means "point of sale information management." The POS terminal is a device that performs point of sale information management.

[0032] The control unit 50 of the POS terminal 5 executes various processes by the CPU 51 operating in accordance with programs stored in the ROM 52 or the storage unit 59 and loaded into the RAM 53. The control unit 50 also provides various functional units (described later, see FIG. 13) by executing the program 591.

[0033] 5 is a block diagram showing an example of the configuration of the code scanner 6. The code scanner 6 includes a CPU 61, a ROM 62, a RAM 63, a communication unit 64, an imaging unit 65, and the like.

[0034] The CPU 61 is an example of a processor and controls all the components of the code scanner 6. The ROM 62 stores various programs. The RAM 63 is a workspace for storing programs and various data. The CPU 61, ROM 62, and RAM 63 are connected via a bus or the like to form a control unit 60 having a computer configuration.

[0035] The communication unit 64 is a communication interface that connects the control unit 60 and an external device (for example, the verification terminal 7) so that they can communicate with each other via the network 9. The communication unit 64 also communicates with the POS terminal 5 via a USB cable.

[0036] The imaging unit 65 has an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary MOS), captures an image of an item held over by the operator, and acquires image data. The imaging unit 65 captures an image of an item each time the operator holds it over for scanning, and acquires multiple pieces of image data.

[0037] The control unit 60 of the code scanner 6 executes various processes by the CPU 61 operating in accordance with a program stored in the ROM 62 and loaded into the RAM 63, and provides the following functions.

[0038] The code scanner 6 extracts a code symbol such as a barcode or two-dimensional code from an image (captured image) made up of image data, and decodes the code symbol to obtain code information. The code information is information that can identify a product, such as a product code such as a JAN code. In other words, the code symbol in this case is an encoded JAN code.

[0039] The code scanner 6 also outputs image data to the collating terminal 7 and receives a product code from the collating terminal 7. The code scanner 6 outputs the product code it obtains by decoding the code symbol itself or the product code it receives from the collating terminal 7, whichever is obtained first, to the POS terminal 5. If the code scanner 6 finishes decoding and obtains a product code before receiving the product code from the collating terminal 7, it transmits the product code to the collating terminal 7.

[0040] 6 is a block diagram showing an example of the configuration of the verification terminal 7. The verification terminal 7 includes a CPU 71, a ROM 72, a RAM 73, a communication unit 74, a storage unit 79, and the like.

[0041] The CPU 71 is an example of a processor and controls all the components of the verification terminal 7. The ROM 72 stores various programs. The RAM 73 is a workspace for storing programs and various data. The CPU 71, ROM 72, and RAM 73 are connected via a bus or the like to form a control unit 70 having a computer configuration.

[0042] The communication unit 74 is a communication interface that connects the control unit 70 and external devices (for example, the store server 3 and the code scanner 6) via the network 9 so that they can communicate with each other.

[0043] The storage unit 79 has a storage medium such as an HDD, SSD, or flash memory, and maintains its stored contents even when power is cut off. The storage unit 79 stores programs executable by the CPU 71 and various setting information. The storage unit 79 also stores a comparison dictionary 794 and image information 795.

[0044] 7 is a diagram showing an example of items included in the matching dictionary 794. The matching dictionary 794 stores product codes and matching data in association with each other in the form of a table, for example.

[0045] The matching data is a representative value of the feature information. The feature information is information that indicates the characteristics of the appearance of an item. The feature information is a value that is output (obtained) by performing a predetermined process on image data obtained by capturing an image of the appearance of a product, and is, for example, a quantity that has a magnitude and direction, such as a vector. The matching data is a standard value of the feature information that varies depending on the capturing conditions, etc.

[0046] To identify a product from its external appearance (i.e., its packaging), it is necessary to be able to view the product from various angles, so it is preferable to have matching data corresponding to various angles. Therefore, multiple matching data are registered for one product code.

[0047] 8 is a diagram showing examples of items included in the image information 795. The image information 795 stores product codes and image data in association with each other, for example, in the form of a table. The image data and product codes are received from the code scanner 6.

[0048] The control unit 70 of the verification terminal 7 executes various processes and provides various functional units (described later, see FIG. 13) by the CPU 71 operating in accordance with a program stored in the ROM 72 and loaded into the RAM 73.

[0049] 9 is a block diagram showing an example of the configuration of the store server 3. The store server 3 includes a CPU 31, a ROM 32, a RAM 33, a communication unit 34, a display unit 35, an operation unit 36, a storage unit 39, and the like.

[0050] The CPU 31 is an example of a processor and controls all the components of the store server 3. The ROM 32 stores various programs. The RAM 33 is a workspace for storing programs and various data. The CPU 31, ROM 32, and RAM 33 are connected via a bus or the like to form a control unit 30 having a computer configuration.

[0051] The communication unit 34 is a communication interface that communicatively connects the control unit 30 and external devices (for example, the POS terminal 5 and the verification terminal 7) via the network 9. The communication unit 34 is also communicatively connected to the server 1 via the network 2.

[0052] The display unit 35 has a display device such as an LCD, and displays various information under the control of the CPU 31.

[0053] The operation unit 36 ​​has input devices such as a touch panel provided over the surface of the display unit 35, a keyboard, and a pointing device, and outputs operation contents input via the input devices to the CPU 31.

[0054] The storage unit 39 has a storage medium such as an HDD, SSD, or flash memory, and maintains its stored contents even when the power is turned off. The storage unit 39 stores various setting information such as a program 391 executable by the CPU 31 and a product master 392. The storage unit 39 also stores transaction information 393 and a matching dictionary 394.

[0055] The items included in the product master 392 are the same as those included in the product master 592 shown in Fig. 3. The items included in the transaction information 393 are the same as those included in the registered product information 593 shown in Fig. 4. The items included in the matching dictionary 394 are the same as those included in the matching dictionary 794 shown in Fig. 7.

[0056] The control unit 30 of the store server 3 executes various processes by the CPU 31 operating in accordance with programs stored in the ROM 32 or the storage unit 39 and loaded into the RAM 33. The control unit 30 also provides various functional units (described later, see FIG. 13) by executing the program 391.

[0057] 10 is a block diagram showing an example of the configuration of the server 1. The server 1 includes a CPU 11, a ROM 12, a RAM 13, a communication unit 14, a storage unit 19, and the like.

[0058] The CPU 11 is an example of a processor and controls all parts of the server 1. The ROM 12 stores various programs. The RAM 13 is a workspace for storing programs and various data. The CPU 11, ROM 12, and RAM 13 are connected via a bus or the like to form a control unit 10 of a computer configuration.

[0059] The communication unit 14 is a communication interface that connects the control unit 10 and an external device (for example, the store server 3) via the network 2 so that they can communicate with each other.

[0060] The memory unit 19 has a storage medium such as an HDD, SSD, or flash memory, and maintains its stored contents even when the power is turned off. The memory unit 19 stores various setting information such as a program 191 executable by the CPU 11 and a product master 192. The memory unit 19 also stores transaction information 193, a matching dictionary 194, image information 195, etc.

[0061] 11 is a diagram showing examples of items included in the product master 192. The product master 192 is an example of product information, and is information about products handled by a store compiled, for example, in the form of a table. Items of product information include, for example, "store code," "product code," "product name," "unit price," "color," "product image," and "not available."

[0062] The store code is information that can identify a store, and is, for example, a unique number assigned to each store. In other words, the store code is not duplicated among stores of the same chain that are managed by the server 1.

[0063] The "Not Available" item is, for example, a flag; if the flag is OFF, it indicates that the store is currently selling the product with the product code in the same record; if the flag is ON, it indicates that the store is not selling the product.

[0064] The product master 192 is the product master 592 (392) shown in Fig. 3 with the addition of the fields "store code" and "not available." The server 1 is connected to the store servers 3 of one or more stores. The server 1 extracts and transmits, from the records contained in the product master 192, records to which the store code of each store is associated and whose "not available" flag is OFF, as the product master 592 to be used by the POS terminal 5 of each store. Upon receiving this, each store server 3 updates the product master 592 of the POS terminal 5 of its own store.

[0065] FIG. 12 is a diagram showing examples of items included in transaction information 193. Transaction information 193 is information about products purchased by a customer, compiled, for example, in the form of a table. Items of product information include, for example, "store code," "product code," "unit price," and "quantity." The items included in transaction information 193 are the same as those in registered product information 593 shown in FIG. 4, with the addition of an item for "store code."

[0066] The items included in the matching dictionary 194 are the same as those included in the matching dictionary 794 shown in Fig. 7. The items included in the image information 195 are the same as those included in the image information 795 shown in Fig. 8.

[0067] The control unit 10 of the server 1 executes various processes by the CPU 11 operating in accordance with programs stored in the ROM 12 or the storage unit 19 and loaded into the RAM 13. The control unit 10 also provides various functional units (described later, see FIG. 13) by executing the program 191.

[0068] Fig. 13 is a diagram showing an example of various functional units and their corresponding relationships provided in each of the control units 10, 30, 50, 60, and 70. Figs. 14 and 15 are diagrams showing the operation timing and relationships of the above-mentioned functional units.

[0069] The control unit 10 of the server 1 includes a transaction information update unit 101, a master image update unit 102, an image information storage unit 103, a matching dictionary update unit 104, a master distribution unit 105, etc. The control unit 30 of the store server 3 includes a transaction information update unit 301, a master image creation unit 302, an image information transfer unit 303, a master update unit 305, a master distribution unit 306, etc. The control unit 50 of the POS terminal 5 includes a product registration unit 501, a payment processing unit 502, a master update unit 505, etc. The control unit 70 of the matching terminal 7 includes a matching unit 701, an image information storage unit 702, an image information transfer unit 703, a matching dictionary update unit 704, etc. The operation of each unit will be described below with reference to FIGS. 14 and 15.

[0070] 14 shows the process performed during store hours. First, when the POS terminal 5 starts a transaction, the control unit 60 of the code scanner 6 connected to it captures an image of the product (step S61). The control unit 60 transfers data of the captured image (image data) to the verification terminal 7 (step S62).

[0071] Furthermore, the control unit 60 cuts out a code symbol from the captured image (step S63), and decodes the cut-out code symbol (step S64) to obtain a product code.

[0072] Meanwhile, in the collation terminal 7 that has received the image data, the collation unit 701 performs a predetermined process on the image data to obtain information indicating the external characteristics of the product (characteristic information) (step S71).

[0073] Next, the matching unit 701 compares the acquired characteristic information with the matching data stored in the matching dictionary 794 (step S72).Then, the matching unit 701 transmits the product code associated with the matching data that matches the characteristic information to the code scanner 6 (step S73).

[0074] If the code scanner 6 completes the decoding in step S64 earlier than step S73, the control unit 60 transmits the decoded product code to the verification terminal 7 (step S66). In this case, the verification terminal 7 does not transmit the product code to the code scanner 6. Also, in this case, the image information storage unit 702 of the verification terminal 7 stores the image data (step S74). In step S74, the image information storage unit 702 associates the image data transferred in step S62 with the product code received in step S66, and stores the image data in the storage unit 79 as image information 795.

[0075] The capturing of an image by the code scanner 6 (step S61) and the transfer of image data (step S62) are repeated until decoding is completed (step S64) or a product code is received from the verifying terminal 7 (step S73). Therefore, there is a high possibility that multiple image data will be transferred from the code scanner 6 to the verifying terminal 7. In step S74, the same product code is associated with all of the image data and stored.

[0076] Furthermore, if the verification (step S72) is successful in the verification terminal 7, the image data received by the verification terminal 7 through transfer in step S62 is discarded.

[0077] Then, the control unit 60 of the code scanner 6 transmits the product code acquired in step S64 or step S73 to the POS terminal 5 (step S65).

[0078] When a product code is received by the POS terminal 5, the product registration unit 501 obtains information about the product (such as product name and unit price) that matches the received product code from the product master 592, specifies the quantity, and records it as registered product information 593 (step S51).

[0079] In the product registration from the start of the transaction until the transition to the payment, a series of processes from step S61 to step S51 is repeated. When a key for instructing the transition to the payment provided on the operation unit 56 of the POS terminal 5 is operated, the control unit 50 ends the product registration and proceeds to the payment as the payment processing unit 502.

[0080] The payment processing unit 502 calculates the price of the product recorded in the registered product information 593 and performs payment processing, such as transferring cash via the deposit / withdrawal unit 58 (step S52). Note that payment may not be made in cash, but may be made by cashless payment such as credit card payment or electronic money payment. When the payment processing is completed, the payment processing unit 502 transfers the registered product information 593 to the store server 3 (step S53).

[0081] When the store server 3 receives the registered product information 593, the control unit 30 functions as the transaction information update unit 301 and updates the transaction information 393 with the registered product information 593 (step S31). In other words, the transaction information update unit 301 stores the registered product information 593 received from the store server 3 in the memory unit 19 as the transaction information 393.

[0082] The transaction information update unit 301 further transmits the transaction information 393 to the server 1 (step S32). The transaction information 393 transmitted at this time may be the increment from the previous transmission (that is, the amount equivalent to the registered product information 593).

[0083] When the server 1 receives the transaction information 393, the control unit 10 functions as the transaction information update unit 101 and updates the transaction information 193 with the transaction information 393 (step S11). At this time, the transaction information update unit 101 associates the transaction information 393 with a store code that identifies the store that sent the information, and stores the associated store code in the transaction information 193.

[0084] Next, a process performed at an arbitrary timing will be described with reference to Fig. 15. The arbitrary timing does not have to be during the business hours of the store, but may be outside the business hours (after closing to before opening), or may be when there is room for traffic in the product registration system 100.

[0085] First, the creation of a "product image" by the master image creation unit 302 and the update of the product master 192 by the master image update unit 102 are as follows: The processing by the master image creation unit 302 is executed, for example, by an operator (a store clerk) operating the store server 3 starting up a predetermined application software.

[0086] The master image creation unit 302 creates a “product image” that is stored in association with a product code in the product master 192 based on the image data stored in the image information 795 .

[0087] The master image creation unit 302 accesses the storage unit 79 of the verification terminal 7 via the network 9 and acquires image information 795 (step S33). Next, the master image creation unit 302 picks up one product code included in the acquired image information 795 and creates a "product image" from the image data associated with the product code (step S34). The master image creation unit 302 repeats the above process for all product codes included in the image information 795. Then, the master image creation unit 302 associates the created product image with the product code and transmits the image to the server 1 (step S35).

[0088] The control unit 10 of the server 1, which receives the pair of product image and product code, functions as the master image update unit 102 and updates the product image in the record of the product master 192 associated with the received product code with the received product image (step S12).

[0089] Here, the creation of product images in step S34 will be explained in more detail. As mentioned above, a "product image" is image data that is attached as information explaining a product when a product with a product code is displayed as an option. For this reason, the "product image" must be suitable for being provided together with information explaining the product. More specifically, it is preferable that the "product image" satisfies, for example, the following conditions (1) to (4): (1) The image includes the product name. (2) The image does not contain code symbols such as barcodes or two-dimensional codes. (3) The image is a frontal view of the product name. (4) The image does not include the outline of the product.

[0090] It should be noted that conditions other than those described above may be added. Furthermore, if it is difficult to satisfy all the conditions, it may be configured to determine that the application is suitable if some of the conditions are satisfied. In this case, priorities may be set for the conditions, and if none of the conditions are satisfied, it may be possible to determine that the conditions with the lowest priorities do not need to be satisfied. The order of the above conditions (1) to (4) indicates the priority order of the conditions in this embodiment.

[0091] Furthermore, as long as the image data satisfies the above-mentioned conditions for being suitable as a "product image," it is acceptable for the product to appear tilted in the image displayed by the image data. For such image data, the master image creation unit 302 generates appropriate image data by cutting out the product portion from the image displayed by the image data and rotating it to an upright position.

[0092] For example, segmentation technology can be used for the process of rotating the image to an upright position and for determining whether the product is facing forward (1) above. Segmentation technology makes it possible to determine the range, position, and arrangement of pixels in the image data that contain the product, and to extract the area surrounded by pixels that includes the outline of the product.

[0093] Fig. 16 is a diagram showing in more detail the processing by the master image creation unit 302. The processing in this figure corresponds to the processing in steps S33 to S34 in Fig. 15. The processing in steps S341 to S345 shown in Fig. 16 is included in the processing in step S34 in Fig. 15.

[0094] First, the master image creation unit 302 acquires image information 795 from the verification terminal 7 (step S33). Next, the master image creation unit 302 picks out one product code included in the acquired image information 795, and selects one or more product image candidates from one or more image data associated with the product code (step S341). The candidates selected here are image data that meet the above-mentioned conditions.

[0095] The master image creation unit 302 displays the candidates selected in step S341 as options on the display unit 35, and accepts an operation to select an option via the operation unit 36 ​​(step S342).

[0096] When the operation unit 36 ​​accepts an operation to select one of the options (step S343; Yes), the master image creation unit 302 determines the image data selected as the option as the "product image" (step S344).

[0097] Next, the master image creation unit 302 determines whether the process of determining product images for all product codes included in the image information 795 acquired in step S33 has been completed (step S345), and if completed (step S345; Yes), ends this process. On the other hand, if the process of determining product images for all product codes included in the image information 795 acquired in step S33 has not been completed (step S345; No), the master image creation unit 302 returns the process to step S341.

[0098] Here, in step S343, the operator (store clerk) of the store server 3 visually checks the options and selects the one that he / she judges to be the most suitable. If the operator judges that "the options do not have appropriate image data," he / she can operate the operation unit 36 ​​to reject all of the options. Rejection of all options can be accepted, for example, by displaying an operator for accepting rejection together with the options on the display unit 35 and accepting the rejection on the operation unit 36.

[0099] In step S343, if all of the options are rejected (step S343; No), the master image creation unit 302 advances the process to step S345.

[0100] If only one candidate is selected in step S341, only one option will be displayed in step S342. In this case, the selection received in the process of step S343 is a selection to accept or reject the option.

[0101] 15, the following describes the updating of the verification dictionaries 194, 394, 794 and the product masters 392, 592. The control unit 30 of the verification terminal 7 functions as the image information transfer unit 703 and transfers the image information 795 to the store server 3 (step S75). Note that the image information 795 may be deleted after being transferred in step S75.

[0102] When the store server 3 receives the image information 795, the control unit 30 functions as the image information transfer unit 303 and transfers it to the server 1 (step S36).

[0103] When the server 1 receives the image information 795, the control unit 10 functions as the image information storage unit 103 and updates the image information 195 (step S13). That is, the image information storage unit 103 adds the image information 795 received from the store server 3 to the storage unit 19 as the image information 195.

[0104] Here, the image information storage unit 103 functions as an example of a code information acquisition unit that acquires code information that enables a product to be identified by reading a code symbol displayed on the product's packaging, and an image data acquisition unit that acquires image data of the product captured when the code symbol is read.

[0105] Next, a process performed outside business hours will be described. The control unit 10 of the server 1 functions as the matching dictionary update unit 104 and updates the matching dictionary 194 (step S14). For this update, the matching dictionary update unit 104 uses the record that is the most recent increment (the record added in the previous step S13) from the image information 195. The matching dictionary update unit 104 performs a predetermined process on the image data of each record to obtain feature information, and adds the product data of that record associated with the feature information to the matching dictionary 194. In this step, automatic learning (registration, updating) of new package information in the matching dictionary 194 is realized.

[0106] Here, the check dictionary update unit 104 functions as an example of a feature information acquisition unit that acquires feature information, which is information indicating the external features of a product, from image data.

[0107] Next, the control unit 10, functioning as the master distribution unit 105, distributes the product master 192 (step S15) and the matching dictionary 194 (step S16) to the store server 3 of each store.

[0108] First, in step S15, the master distribution unit 105 transmits records that have been changed since the previous transfer among those that match the store code of the destination store in the product master 192. Furthermore, for records that have been changed from available to unavailable since the previous transfer, the master distribution unit 105 transmits an instruction to the store server 3 to delete the record of the corresponding product code.

[0109] In step S16, the master distribution unit 105 extracts and distributes suitable records when distributing the matching dictionary 194. That is, the master distribution unit 105 extracts product codes that have been traded with the specified store code (of the store to which the data is to be distributed) from the product master 192, makes a list of them, extracts records associated with the product codes included in the list from the matching dictionary 194, and transmits them to the store server 3.

[0110] When the store server 3 receives the product master 192 and the matching dictionary 194 from the server 1, the control unit 30 functions as the master update unit 305 to update the product master 392 (step S37) and the matching dictionary 394 (step S38).

[0111] When the product master 392 and the matching dictionary 394 are updated, the control unit 30 of the store server 3, as the master distribution unit 306, distributes the product master 392 to the POS terminals 5 in the store (step S39), and also distributes the matching dictionary 394 to the matching terminal 7 (step S40).

[0112] In the POS terminal 5 that has received the product master 392, the control unit 50 functions as the master update unit 505 and updates the product master 592 (step S54).

[0113] In the verification terminal 7 that has received the verification dictionary 394, the control unit 70 functions as the verification dictionary update unit 704 to update the verification dictionary 794 (step S76).

[0114] In the product registration system 100 configured as described above, the code scanner 6 decodes the code symbol while the verification terminal 7 recognizes the image, and product registration and payment are performed based on the code information obtained earlier.

[0115] The image data output by the code scanner 6 is input to the verification terminal 7. The verification terminal 7 generates information (feature information) indicating the external features (shape, color, pattern, etc.) of the product captured in the captured image indicated by the image data, and compares the feature information with the verification data by referring to the verification dictionary 794.

[0116] The verification terminal 7 identifies the product by image recognition as described above, and returns the result to the code scanner 6. The recognition result output by the verification terminal 7 is output to the POS terminal 5 via the code scanner 6.

[0117] The POS terminal 5 registers the product and performs payment for the registered product based on the code information output by the code scanner 6. The POS terminal 5 automatically registers the information of the identified product as an item to be purchased (transaction item).

[0118] Of the image data obtained by the code scanner 6 and transmitted to the verification terminal 7, if there is no verification data matching the characteristic information in the verification dictionary 793, the image data is sent to the server 1 together with the product code identified by the code scanner 6. The server 1 generates verification data from the newly added (updated) image data and updates the verification dictionary 194. The additions (updated) to the verification dictionary 194 are sent to the verification terminal 7. This allows the verification terminal 7 to obtain verification data for products that could not be verified last time.

[0119] The above-described embodiment can be modified as needed by changing part of the configuration or function of each of the above-described devices.

[0120] For example, in implementation, a functional unit equivalent to the master image creation unit 302 may be provided in another device in the store, such as the POS terminal 5, instead of the store server 3. Also, in implementation, a functional unit equivalent to the master image creation unit 302 may be provided in the server 1.

[0121] Furthermore, in implementing the present invention, a functional unit equivalent to the master image creation unit 302 provided in the server 1 may be configured to be called and used from another device in the store, such as the store server 3. In this case, if the functional unit equivalent to the master image creation unit 302 provided in the server 1 is provided as an API (Application Programming Interface), it will be easy to use from an external device.

[0122] In addition, in the above embodiment, it was explained that the "product image" was determined by visual confirmation by a store clerk, but in practice, visual confirmation by a store clerk may be omitted and the image that is mechanically determined to be the most suitable may be determined.

[0123] In the above embodiment, the image data and product code acquired during product registration are used to create the product images to be stored in the product master 192, but this is not a limitation. For example, an available POS terminal 5, code scanner 6, and verification terminal 7 may be used at a time separate from product registration, with the code scanner 6 acquiring image data and product codes, selecting product image candidates from the image data, and accepting the selection of the candidates via the display unit 55 and operation unit 56 of the POS terminal 5 to determine the product image. Furthermore, in this case, instead of the code scanner 6, another imaging device that handles digital images, such as a web camera, may be used. This allows for processing equivalent to that performed by the master image creation unit 302.

[0124] Furthermore, in this embodiment, the POS terminal 5 may be operated by a store clerk or a customer. Furthermore, in this embodiment, the server 1 may be composed of multiple devices rather than a single device. Furthermore, in this embodiment, the functional units described as being possessed by the server 1 may be performed by any of the devices in the store. Furthermore, in this embodiment, a device equivalent to the server 1 may be installed in the store. Furthermore, in this embodiment, the store server 3 may not be required. Furthermore, in this embodiment, the verification terminal 7 may be shared by multiple code scanners 6. Furthermore, the memory unit 19 (39, 59, 79) is an example of a device that provides a memory area capable of storing various information. The information described as being stored in the same memory unit in the above embodiment does not necessarily have to be stored in the same memory unit, but may be stored separately in multiple memory devices.

[0125] The programs executed by each device in the above-described embodiments are provided in advance in a ROM, etc. The programs executed by each device in the above-described embodiments may be provided by being recorded in an installable or executable file format on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD).

[0126] Furthermore, the programs executed by each device in the above-described embodiments may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the programs executed by each device in the above-described embodiments may be provided or distributed via a network such as the Internet.

[0127] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, modifications, and combinations can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0128] 100...Product registration system, 1...server, 10...control unit, 101...transaction information update unit, 102...master image update unit, 103...image information storage unit, 104...check dictionary update unit, 105...master distribution unit, 11...CPU, 12...ROM, 13...RAM, 14...Communication section, 19...Storage section, 191...program, 192...product master, 193...transaction information, 194... Matching dictionary, 195... Image information, 2...Network, 3...store server, 30...control unit, 301: transaction information update unit; 302: master image creation unit; 303: image information transfer unit; 305...master update unit, 306...master distribution unit, 31...CPU, 32...ROM, 33...RAM, 34...Communication section, 35...Display section, 36...Operation section, 39...Storage section, 391...program, 392...product master, 393...transaction information, 394...matching dictionary, 5...POS terminals, 50...control unit, 501...product registration unit, 502...payment processing unit, 505...master update unit, 51...CPU, 52...ROM, 53...RAM, 54...Communication section, 55...display unit, 56...operation unit, 57...audio output unit, 58...deposit / withdrawal unit, 59...storage unit, 591...program, 592...product master, 593...registered product information, 6...code scanner, 60...control unit, 61...CPU, 62...ROM, 63...RAM, 64...communication unit, 65...imaging unit, 7 ... collating terminal, 70...control unit, 701: Collation unit, 702: Image information storage unit, 703...image information transfer unit, 704...matching dictionary update unit, 71...CPU, 72...ROM, 73...RAM, 74...Communication section, 79...Memory unit, 793...Comparison dictionary, 794...Comparison dictionary, 795...Image information, 9...Network. [Prior art documents] [Patent documents]

[0129] [Patent Document 1] Japanese Patent Publication No. 2019-021105

Claims

1. a code information acquiring unit that acquires code information that enables a product to be identified by reading a code symbol displayed on a product package; an image data acquisition unit that acquires image data of the product captured when the code symbol is read; a characteristic information acquisition unit that acquires characteristic information indicating characteristics of the appearance of the product from the image data; a dictionary output unit that outputs a matching dictionary that stores one piece of code information in association with one or more pieces of feature information; a master image creation unit that creates a product image to be stored in association with the code information based on the image data; Equipped with The master image creation unit selects, as the product image, from the image data, an area that includes a product name. Information processing device.

2. a code information acquiring unit that acquires code information that enables a product to be identified by reading a code symbol displayed on a product package; an image data acquisition unit that acquires image data of the product captured when the code symbol is read; a characteristic information acquisition unit that acquires characteristic information indicating characteristics of the appearance of the product from the image data; a dictionary output unit that outputs a matching dictionary that stores one piece of code information in association with one or more pieces of feature information; a master image creation unit that creates a product image to be stored in association with the code information based on the image data; Equipped with The master image creation unit selects, as the product image, image data that does not include a code symbol. Information processing device.

3. a code information acquiring unit that acquires code information that enables a product to be identified by reading a code symbol displayed on a product package; an image data acquisition unit that acquires image data of the product captured when the code symbol is read; a characteristic information acquisition unit that acquires characteristic information indicating characteristics of the appearance of the product from the image data; a dictionary output unit that outputs a matching dictionary that stores one piece of code information in association with one or more pieces of feature information; a master image creation unit that creates a product image to be stored in association with the code information based on the image data; Equipped with The master image creation unit selects, as the product image, an image of the product facing a surface on which a product name is displayed from among the image data. Information processing device.

4. A computer equipped in a system that performs product registration related to sales data processing, a code information acquiring unit that acquires code information that enables a product to be identified by reading a code symbol displayed on a product package; an image data acquisition unit that acquires image data of the product captured when the code symbol is read; a characteristic information acquisition unit that acquires characteristic information indicating characteristics of the appearance of the product from the image data; a dictionary output unit that outputs a matching dictionary that stores one piece of code information in association with one or more pieces of feature information; a master image creation unit that creates a product image to be stored in association with the code information based on the image data; It functions as The master image creation unit is a program for selecting, as the product image, from the image data, an area that includes the name of the product.

5. A computer equipped in a system that performs product registration related to sales data processing, a code information acquiring unit that acquires code information that enables a product to be identified by reading a code symbol displayed on a product package; an image data acquisition unit that acquires image data of the product captured when the code symbol is read; a characteristic information acquisition unit that acquires characteristic information indicating characteristics of the appearance of the product from the image data; a dictionary output unit that outputs a matching dictionary that stores one piece of code information in association with one or more pieces of feature information; a master image creation unit that creates a product image to be stored in association with the code information based on the image data; It functions as The master image creation unit is a program for selecting, as the product image, image data that does not include a code symbol.

Images