Information processing apparatus and program
The information processing device addresses the challenge of unreliable product identification by automating dictionary updates and enhancing user interaction, improving the reliability and practicality of image recognition for products with code symbols.
Patent Information
- Application Number
- JP2023076627
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-08
- Publication Date
- 2024-11-20
- Estimated Expiration
- 2043-05-08
AI Technical Summary
Existing image recognition technologies for product identification face challenges in improving reliability and practicality, especially when dealing with products that have code symbols on their packages, and there is a need to expand the range of products that can be identified using image recognition.
An information processing device that includes a code information acquisition unit, an image data acquisition unit, a feature information acquisition unit, and a dictionary output unit, which automatically updates a collation dictionary to associate feature information with code information, enabling reliable product identification through image recognition.
Enhances the reliability and practicality of product identification systems by automating the update of dictionaries to accommodate new packages and improving user interaction through visual confirmation of matching products, ensuring accurate and efficient product recognition.
Smart Images

Figure 2024161704000001_ABST
Abstract
Description
[Technical field]
[0001] An embodiment of the present invention relates to an information processing device and a program. [Background technology]
[0002] In sales data processing, a technology is used to identify products using information indicating the external characteristics (shape, color, pattern, etc.) of an object obtained by image recognition (for example, Patent Document 1). Currently, this technology is mainly used to identify products that do not have code symbols such as barcodes, such as fruits and vegetables. For other products that have code symbols on their packaging, products are identified using code information obtained by reading the code symbols, as in the past.
[0003] In recent years, the scope of product identification using image recognition has been expanded to include products with code symbols, and attempts have been made to identify products from the external characteristics of the package, rather than by reading the code symbol.Identifying packaged products using image recognition poses the challenge of increasing reliability (accuracy), and efforts are being made toward practical use. Summary of the Invention [Problem to be solved by the invention]
[0004] The problem that the present invention aims to solve is to expand the range of products that can be identified by image recognition, to enable the provision of practical reliability, and to increase the practicality of the operation of the system that provides the technology. [Means for solving the problem]
[0005] An information processing device of an embodiment includes a code information acquisition unit that acquires code information that enables a product to be identified by reading a code symbol displayed on a product package, an image data acquisition unit that acquires image data of the product captured when the code symbol is read, a feature information acquisition unit that acquires feature information that is information indicating the external features of the product from the image data, a dictionary output unit that outputs a comparison dictionary that associates one or more of the feature information with one of the code information and stores it, and a master image creation unit that creates a product image to be stored in association with the code information based on the image data. [Brief description of the drawings]
[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a product registration system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of a POS terminal. [Diagram 3] FIG. 3 is a diagram showing examples of items included in the product master. [Figure 4] FIG. 4 is a diagram showing examples of items included in the registered product information. [Diagram 5] FIG. 5 is a block diagram showing an example of the configuration of a code scanner. [Figure 6] FIG. 6 is a block diagram showing an example of the configuration of the collation terminal. [Figure 7] FIG. 7 is a diagram showing examples of items included in the collation dictionary. [Figure 8] FIG. 8 is a diagram showing examples of items included in the image information. [Figure 9] FIG. 9 is a block diagram showing an example of the configuration of the store server. [Figure 10] FIG. 10 is a block diagram showing an example of the configuration of the server. [Figure 11] FIG. 11 is a diagram showing examples of items included in the product master. [Figure 12] FIG. 12 is a diagram showing examples of items included in the transaction information. [Figure 13]FIG. 13 is a diagram showing an example of various functional units included in each control unit and the corresponding relationships between them. [Figure 14] FIG. 14 is a diagram showing the operation timing and relationship of each functional unit. [Figure 15] FIG. 15 is a diagram showing the operation timing and relationship of each functional unit. [Figure 16] FIG. 16 is a diagram showing the process by the master image creation unit in more detail. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0007] (First embodiment) An embodiment will be described with reference to the drawings. Fig. 1 is a diagram showing an example of the configuration of a product registration system 100 according to the embodiment. The product registration system 100 includes a server 1, a store server 3, a POS terminal 5, a code scanner 6, and a verification terminal 7. The server 1, the store server 3, and the POS terminal 5 are examples of information processing devices.
[0008] In the product registration system 100, in order to correctly determine a product from a captured image, it is necessary to enrich the dictionary for estimating the product from the image. To do this, it is necessary to collect many images related to the product. In addition, for example, when a new product is released or the package design is changed for a limited time even if the product remains unchanged, the feature information obtained from the captured image and the matching data will no longer match. For this reason, it is necessary to continuously update the dictionary to correspond to the new package.
[0009] Since new packages appear every day, relying on manual updating of the above-mentioned dictionary is not preferable in terms of preventing omissions in the dictionary, and the number of entries would be impossible for a large-scale store. Therefore, in this embodiment, the above-mentioned dictionary is configured to be updated in an automatic learning manner, and the handling of new packages is automated.
[0010] In the case of identifying a product by comparing feature information with matching data, it is possible that the user may be asked to confirm whether the product identification is correct, or may be asked to select from several candidates. In such cases, in order to make it easier for the user to make a decision, it is preferable to show an image of a product that is estimated to match the feature information, rather than showing only text information. Therefore, this embodiment proposes a configuration for obtaining an image suitable for showing to the user during user confirmation, in parallel with the automatic learning of the dictionary described above.
[0011] The server 1 can communicate with a store server 3 via a network 2 to transmit and receive information. The store server 3 is communicably connected to one or more POS terminals 5 via a network 9 such as a LAN (Local Area Network), to transmit and receive information.
[0012] One code scanner 6 and one verification terminal 7 are provided for each POS terminal 5. The verification terminal 7 is connected to the POS terminal 5 via the code scanner 6. The code scanner 6 also relays communication between the verification terminal 7 and the POS terminal 5. The verification terminal 7 is connected to the store server 3 via a network 9 and can communicate with the store server 3. The code scanner 6 and the POS terminal 5 are connected via a USB, and the code scanner 6 and the verification terminal 7 are connected via a LAN.
[0013] The POS terminal 5 receives input from the code scanner 6, registers the product, and makes payment for the registered product. The code scanner 6 captures an image of the exterior of the product, and outputs to the POS terminal 5 the read result (code information) of a code symbol such as a barcode or a two-dimensional code, and information (code information) input from the verification terminal 7.
[0014] The collation terminal 7 obtains information (feature information) indicating the external characteristics (shape, color, pattern, etc.) of the product from the data of the image captured by the code scanner 6, and returns information (code information) capable of identifying a product that matches this information to the code scanner 6. The collation terminal 7 also provides the store server 3 with data (image data) of the image (captured image) captured by the code scanner 6 in association with the read result of the code symbol output by the code scanner 6 (code information), in association with each other.
[0015] The store server 3 is an information processing device such as a PC (Personal Computer), which is installed in the back yard of the store and is used for tasks such as monitoring the operation of the POS terminals 5 and other devices installed in the store and managing information stored in the in-store devices such as the POS terminals 5. The store server 3 also provides the store clerk with a UI (User Interface) for performing the above tasks.
[0016] The store server 3 accumulates information on the paid product (transaction information) sent from the POS terminal 5, and transfers it to the server 1. The store server 3 also transfers a set of image data and code information sent from the collation terminal 7 to the server 1. The store server 3 also receives product master data and a collation dictionary suitable for its own store from the server 1, and deploys them to each POS terminal 5 and collation terminal 7 in its own store.
[0017] The server 1 accumulates pairs of image data and code information acquired from each store via the store server 3, and generates verification data to be associated with the code information based on the pairs.
[0018] 2 is a block diagram showing an example of the configuration of the POS terminal 5. The POS terminal 5 includes a CPU (Central Processing Unit) 51, a ROM (Read Only Memory) 52, a RAM (Random Access Memory) 53, a communication unit 54, a display unit 55, an operation unit 56, an audio output unit 57, a deposit / withdrawal unit 58, a storage unit 59, etc.
[0019] The CPU 51 is an example of a processor and controls each part of the POS terminal 5. The ROM 52 stores various programs. The RAM 53 is a workspace for developing programs and various data. The CPU 51, the ROM 52, and the RAM 53 are connected via a bus or the like to constitute a control unit 50 having a computer configuration.
[0020] The communication unit 54 is a communication interface that communicatively connects the control unit 50 and an external device (e.g., the store server 3) via the network 9. The communication unit 54 also accepts input of code information output by the USB-connected code scanner 6.
[0021] The display unit 55 has a display device such as an LCD (Liquid Crystal Display), and displays various information under the control of the CPU 51.
[0022] The operation unit 56 has input devices such as a touch panel provided over the surface of the display unit 55, a keyboard, and a pointing device, and outputs to the CPU 51 the contents of operations input via the input devices.
[0023] The audio output unit 57 has a speaker and a buzzer, and outputs audio under the control of the CPU 51.
[0024] The deposit / withdrawal unit 58 is, for example, a change machine that stores cash (coins and bills) and receives deposits and dispenses change during settlement.
[0025] The storage unit 59 has a storage medium such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory, and maintains the stored contents even when the power is cut off. The storage unit 59 stores various setting information such as a program 591 executable by the CPU 51 and a product master 592. The storage unit 59 (or RAM 53) also stores registered product information 593, which is information on registered products (registered products), from the start to the completion of at least one transaction.
[0026] 3 is a diagram showing examples of items included in the product master 592. The product master 592 is an example of product information, and information on products handled by a store is compiled, for example, in the form of a table. Items of product information include, for example, "product code," "product name," "unit price," "color," and "product image."
[0027] The product code is an example of information that can identify a product, such as a JAN code. Other information (unit price, verification data, color, corresponding products, etc.) is stored in association with the product code.
[0028] The "Product Name" field records the name of the product. The "Unit Price" field records the price for one product. The "Color" field records information indicating the color and design of the product.
[0029] "Product image" is image data that is added as information explaining the product when the product with the product code is displayed as an option.
[0030] 4 is a diagram showing examples of items included in registered product information 593. Registered product information 593 is information on products registered for purchase by customers (i.e. products subject to transaction) compiled, for example, in the form of a table. Items of product information include, for example, "product code," "unit price," and "quantity." "Quantity" is the number of products purchased.
[0031] The program 591 is application software for operating the POS terminal 5. POS is an abbreviation for "Point Of Sale" and means "point of sale information management." The POS terminal is a device that performs point of sale information management.
[0032] The control unit 50 of the POS terminal 5 executes various processes by the CPU 51 operating in accordance with programs stored in the ROM 52 or the storage unit 59 and loaded into the RAM 53. The control unit 50 also provides various functional units (described later, see FIG. 13) by executing a program 591.
[0033] 5 is a block diagram showing an example of the configuration of the code scanner 6. The code scanner 6 includes a CPU 61, a ROM 62, a RAM 63, a communication unit 64, an imaging unit 65, and the like.
[0034] The CPU 61 is an example of a processor and controls each part of the code scanner 6. The ROM 62 stores various programs. The RAM 63 is a workspace for developing programs and various data. The CPU 61, the ROM 62, and the RAM 63 are connected via a bus or the like to constitute a control unit 60 having a computer configuration.
[0035] The communication unit 64 is a communication interface that communicatively connects the control unit 60 and an external device (for example, the collation terminal 7) via the network 9. The communication unit 64 also communicates with the POS terminal 5 via a USB cable.
[0036] The imaging unit 65 has an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary MOS), captures an image of an article held by an operator, and acquires image data. The imaging unit 65 acquires multiple image data for each image captured by the operator holding an article over the camera.
[0037] The control unit 60 of the code scanner 6 executes various processes by the CPU 61 operating in accordance with a program stored in the ROM 62 and loaded in the RAM 63, thereby providing the following functions.
[0038] The code scanner 6 extracts a code symbol such as a barcode or a two-dimensional code from an image (captured image) that is composed of image data, and decodes the code symbol to obtain code information. The code information is information that can identify a product, such as a product code such as a JAN code. In other words, the code symbol in this case is an encoded JAN code.
[0039] Furthermore, the code scanner 6 outputs image data to the collating terminal 7 and receives a product code from the collating terminal 7. The code scanner 6 outputs to the POS terminal 5 either the product code it obtains by decoding the code symbol itself or the product code it receives from the collating terminal 7, whichever is obtained first. Furthermore, if the code scanner 6 finishes decoding and obtains a product code before receiving the product code from the collating terminal 7, it transmits the product code to the collating terminal 7.
[0040] 6 is a block diagram showing an example of the configuration of the verification terminal 7. The verification terminal 7 includes a CPU 71, a ROM 72, a RAM 73, a communication unit 74, a storage unit 79, and the like.
[0041] The CPU 71 is an example of a processor and controls each part of the collation terminal 7. The ROM 72 stores various programs. The RAM 73 is a workspace for developing programs and various data. The CPU 71, the ROM 72, and the RAM 73 are connected via a bus or the like to constitute a control unit 70 having a computer configuration.
[0042] The communication unit 74 is a communication interface that connects the control unit 70 and an external device (for example, the store server 3 and the code scanner 6) via the network 9 so that they can communicate with each other.
[0043] The storage unit 79 has a storage medium such as an HDD, SSD, or flash memory, and maintains the stored contents even when the power is cut off. The storage unit 79 stores programs executable by the CPU 71 and various setting information. The storage unit 79 also stores a comparison dictionary 794 and image information 795.
[0044] 7 is a diagram showing an example of items included in the matching dictionary 794. The matching dictionary 794 stores product codes and matching data in association with each other in the form of a table, for example.
[0045] The matching data is a representative value of the feature information. The feature information is information that indicates the external characteristics of an item. The feature information is a value that is output (obtained) by performing a predetermined process on image data obtained by imaging the external appearance of a product, and is, for example, a quantity having a magnitude and direction, such as a vector. The matching data is a standard value of the feature information that varies depending on the imaging conditions, etc.
[0046] In order to identify a product from its external appearance (i.e., its packaging), it is necessary to be able to view the product from various angles, so it is preferable to have matching data corresponding to various angles. Therefore, multiple matching data are registered for one product code.
[0047] 8 is a diagram showing examples of items included in the image information 795. The image information 795 stores product codes and image data in association with each other, for example, in the form of a table. The image data and product codes are received from the code scanner 6.
[0048] In the control unit 70 of the collation terminal 7, the CPU 71 operates according to a program stored in the ROM 72 and loaded into the RAM 73, thereby executing various processes and providing various functional units (described later, see FIG. 13).
[0049] 9 is a block diagram showing an example of the configuration of the store server 3. The store server 3 includes a CPU 31, a ROM 32, a RAM 33, a communication unit 34, a display unit 35, an operation unit 36, a storage unit 39, and the like.
[0050] The CPU 31 is an example of a processor, and controls each part of the store server 3. The ROM 32 stores various programs. The RAM 33 is a workspace for expanding the programs and various data. The CPU 31, the ROM 32, and the RAM 33 are connected via a bus or the like, and constitute the control unit 30 of the computer configuration.
[0051] The communication unit 34 is a communication interface that communicatively connects the control unit 30 and an external device (for example, the POS terminal 5 and the collation terminal 7) via the network 9. The communication unit 34 is also communicatively connected to the server 1 via the network 2.
[0052] The display unit 35 has a display device such as an LCD, and displays various information under the control of the CPU 31.
[0053] The operation unit 36 has input devices such as a touch panel provided over the surface of the display unit 35, a keyboard, and a pointing device, and outputs operation contents input via the input devices to the CPU 31.
[0054] The storage unit 39 has a storage medium such as an HDD, SSD, or flash memory, and maintains the stored contents even when the power is cut off. The storage unit 39 stores various setting information such as a program 391 executable by the CPU 31 and a product master 392. The storage unit 39 also stores transaction information 393 and a comparison dictionary 394.
[0055] The items included in the product master 392 are the same as those included in the product master 592 shown in Fig. 3. The items included in the transaction information 393 are the same as those included in the registered product information 593 shown in Fig. 4. The items included in the matching dictionary 394 are the same as those included in the matching dictionary 794 shown in Fig. 7.
[0056] The control unit 30 of the store server 3 executes various processes by the CPU 31 operating in accordance with programs stored in the ROM 32 or the storage unit 39 and loaded into the RAM 33. The control unit 30 also provides various functional units (described later, see FIG. 13) by executing a program 391.
[0057] 10 is a block diagram showing an example of the configuration of the server 1. The server 1 includes a CPU 11, a ROM 12, a RAM 13, a communication unit 14, a storage unit 19, and the like.
[0058] The CPU 11 is an example of a processor and controls each part of the server 1. The ROM 12 stores various programs. The RAM 13 is a workspace for expanding the programs and various data. The CPU 11, the ROM 12, and the RAM 13 are connected via a bus or the like to constitute a control unit 10 of a computer configuration.
[0059] The communication unit 14 is a communication interface that connects the control unit 10 and an external device (eg, the store server 3) via the network 2 so that they can communicate with each other.
[0060] The storage unit 19 has a storage medium such as an HDD, SSD, or flash memory, and maintains the stored contents even when the power is cut off. The storage unit 19 stores various setting information such as a program 191 executable by the CPU 11 and a product master 192. The storage unit 19 also stores transaction information 193, a comparison dictionary 194, image information 195, etc.
[0061] 11 is a diagram showing examples of items included in the product master 192. The product master 192 is an example of product information, and information on products handled by a store is compiled, for example, in the form of a table. Items of product information include, for example, "store code," "product code," "product name," "unit price," "color," "product image," and "not available."
[0062] The store code is information that can identify a store, and is, for example, a unique number assigned to each store. In other words, the store code is not duplicated among stores of the same chain that are managed by the server 1.
[0063] The "Not Available" item is, for example, a flag; if the flag is OFF, it indicates that the store carries the product with the product code in the same record, and if the flag is ON, it indicates that the store does not carry the product.
[0064] The product master 192 is the product master 592 (392) shown in Fig. 3 with the addition of the fields "store code" and "not available". The server 1 is connected to the store servers 3 of one or more stores. The server 1 extracts records contained in the product master 192 that are associated with the store code of each store and have the "not available" flag set to OFF, and transmits them as the product master 592 to be used by the POS terminal 5 of each store. Each store server 3 that receives these records updates the product master 592 of the POS terminal 5 of its own store.
[0065] FIG. 12 is a diagram showing examples of items included in transaction information 193. Transaction information 193 is information about products purchased by a customer compiled, for example, in the form of a table. Items of product information are, for example, "store code," "product code," "unit price," and "quantity." The items included in transaction information 193 are the same as those of registered product information 593 shown in FIG. 4, with the addition of an item of "store code."
[0066] The items included in the matching dictionary 194 are the same as those included in the matching dictionary 794 shown in Fig. 7. The items included in the image information 195 are the same as those included in the image information 795 shown in Fig. 8.
[0067] The control unit 10 of the server 1 executes various processes by the CPU 11 operating in accordance with programs stored in the ROM 12 or the storage unit 19 and loaded into the RAM 13. The control unit 10 also executes the program 191 to provide various functional units (described later, see FIG. 13).
[0068] Fig. 13 is a diagram showing an example of various functional units and their corresponding relationships provided in each of the control units 10, 30, 50, 60, 70. Figs. 14 and 15 are diagrams showing the operation timing and relationships of each of the functional units described above.
[0069] The control unit 10 of the server 1 includes a transaction information update unit 101, a master image update unit 102, an image information storage unit 103, a matching dictionary update unit 104, a master distribution unit 105, etc. The control unit 30 of the store server 3 includes a transaction information update unit 301, a master image creation unit 302, an image information transfer unit 303, a master update unit 305, a master distribution unit 306, etc. The control unit 50 of the POS terminal 5 includes a product registration unit 501, a payment processing unit 502, a master update unit 505, etc. The control unit 70 of the matching terminal 7 includes a matching unit 701, an image information storage unit 702, an image information transfer unit 703, a matching dictionary update unit 704, etc. The operation of each unit will be described below with reference to Figs. 14 and 15.
[0070] 14 shows the process performed during business hours of the store. First, when the POS terminal 5 starts a transaction, the control unit 60 of the code scanner 6 connected to it captures an image of the product (step S61). The control unit 60 transfers the data of the captured image (image data) to the collation terminal 7 (step S62).
[0071] Furthermore, the control unit 60 cuts out a code symbol from the captured image (step S63), and decodes the cut out code symbol (step S64) to obtain a product code.
[0072] Meanwhile, in the collation terminal 7 that has received the image data, the collation unit 701 performs a predetermined process on the image data to obtain information indicating the external characteristics of the product (characteristic information) (step S71).
[0073] Next, the matching unit 701 checks the acquired characteristic information against the matching data stored in the matching dictionary 794 (step S72). Then, the matching unit 701 transmits the product code associated with the matching data that matches the characteristic information to the code scanner 6 (step S73).
[0074] If the code scanner 6 completes the decoding in step S64 at an earlier timing than step S73, the control unit 60 transmits the product code obtained by the decoding to the verification terminal 7 (step S66). In this case, the verification terminal 7 does not transmit the product code to the code scanner 6. Also, in this case, the image information storage unit 702 of the verification terminal 7 stores the image data (step S74). In step S74, the image information storage unit 702 associates the image data transferred in step S62 with the product code received in step S66, and stores them in the storage unit 79 as image information 795.
[0075] The capturing of the image by the code scanner 6 (step S61) and the transfer of the image data (step S62) are repeated until the decoding is completed (step S64) or the product code is received from the verifying terminal 7 (step S73). Therefore, there is a high possibility that a plurality of image data are transferred from the code scanner 6 to the verifying terminal 7. In step S74, the same product code is associated with all of the image data and stored.
[0076] Moreover, if the collation (step S72) is successful in the collation terminal 7, the image data received by the collation terminal 7 through the transfer in step S62 is discarded.
[0077] Then, the control unit 60 of the code scanner 6 transmits the product code acquired in step S64 or step S73 to the POS terminal 5 (step S65).
[0078] When a product code is received by the POS terminal 5, the product registration unit 501 obtains information about the product that matches the received product code (such as product name and unit price) from the product master 592, specifies the quantity, and records it as registered product information 593 (step S51).
[0079] In the product registration from the start of the transaction until the payment, a series of processes from step S61 to step S51 are repeated. When a key for instructing the transition to the payment provided in the operation unit 56 of the POS terminal 5 is operated, the control unit 50 ends the product registration and proceeds to the payment as the payment processing unit 502.
[0080] The payment processing unit 502 calculates the price of the product recorded in the registered product information 593 and performs payment processing such as transferring cash via the deposit / withdrawal unit 58 (step S52). Note that payment may be made by cashless payment such as credit card payment or electronic money payment instead of cash. When the payment processing is completed, the payment processing unit 502 transfers the registered product information 593 to the store server 3 (step S53).
[0081] When the store server 3 receives the registered product information 593, the control unit 30, functioning as the transaction information update unit 301, updates the transaction information 393 with the registered product information 593 (step S31). In other words, the transaction information update unit 301 stores the registered product information 593 received from the store server 3 in the storage unit 19 as the transaction information 393.
[0082] The transaction information update unit 301 further transmits the transaction information 393 to the server 1 (step S32). The transaction information 393 transmitted at this time may be an increment from the previous transmission (that is, an amount equivalent to the registered product information 593).
[0083] When the server 1 receives the transaction information 393, the control unit 10, functioning as the transaction information update unit 101, updates the transaction information 193 with the transaction information 393 (step S11). At this time, the transaction information update unit 101 associates the transaction information 393 with a store code that identifies the store that sent the transaction information, and stores the associated store code in the transaction information 193.
[0084] Next, a process performed at an arbitrary timing will be described with reference to Fig. 15. The arbitrary timing does not have to be during the business hours of the store, but may be outside the business hours (after closing to before opening), or may be when there is a margin of traffic in the product registration system 100.
[0085] First, the creation of a "product image" by the master image creation unit 302 and the update of the product master 192 by the master image update unit 102 are as follows. The processing by the master image creation unit 302 is executed, for example, by an operator (a store clerk) who operates the store server 3 starting up a specific application software.
[0086] The master image creation unit 302 creates a “product image” that is stored in association with the product code in the product master 192 based on the image data stored in the image information 795 .
[0087] The master image creation unit 302 accesses the memory unit 79 of the collation terminal 7 via the network 9 and acquires the image information 795 (step S33). Next, the master image creation unit 302 picks up one product code included in the acquired image information 795, and creates a "product image" from the image data associated with the product code (step S34). The master image creation unit 302 repeats the above process for all product codes included in the image information 795. Then, the master image creation unit 302 transmits the created product image associated with the product code to the server 1 (step S35).
[0088] The control unit 10 of the server 1 that has received the pair of product image and product code functions as the master image update unit 102 to update the product image in the record of the product master 192 associated with the received product code with the received product image (step S12).
[0089] Here, the creation of the product image in step S34 will be explained in more detail. As mentioned above, a "product image" is image data that is added as information explaining the product when the product with the product code is displayed as an option. For this reason, the "product image" needs to be suitable for being provided together with the information explaining the product. More specifically, it is preferable that the "product image" satisfies, for example, the following conditions (1) to (4). (1) The image includes the product name. (2) The image does not contain any code symbols such as barcodes or two-dimensional codes. (3) The image must face the product name directly on the surface. (4) The image does not include the outline of the product
[0090] Conditions other than those described above may be added. If it is difficult to satisfy all the conditions, the system may be configured to determine suitability if some of the conditions are satisfied. In this case, a priority order may be set for the conditions, and if none of the conditions are satisfied, it may be acceptable to not satisfy the conditions with the lowest priority. The order of the above conditions (1) to (4) indicates the priority order of the conditions in this embodiment.
[0091] In addition, as long as the above-mentioned conditions for being suitable as a "product image" are met, the product may be displayed at an angle in the image displayed by the image data. For such image data, the master image creation unit 302 cuts out the product part from the image displayed by the image data and rotates it to an upright position to generate appropriate image data.
[0092] For example, segmentation technology can be used for the above-mentioned process of rotating the image to an upright position and for the above-mentioned (1) determination of whether the product is facing forward. Segmentation technology makes it possible to determine the range, position, and arrangement of pixels in which the product appears in the image data, and to cut out the area surrounded by pixels that includes the outline of the product.
[0093] Fig. 16 is a diagram showing in more detail the processing by the master image creation unit 302. The processing in this figure corresponds to the processing in steps S33 to S34 in Fig. 15. The processing in steps S341 to S345 shown in Fig. 16 is included in the processing in step S34 in Fig. 15.
[0094] First, the master image creation unit 302 acquires image information 795 from the collation terminal 7 (step S33). Next, the master image creation unit 302 picks up one product code included in the acquired image information 795, and selects one or more product image candidates from one or more image data associated with the product code (step S341). The candidates selected here are image data that meet the above-mentioned conditions.
[0095] The master image creation unit 302 displays the candidates selected in step S341 as options on the display unit 35, and accepts an operation to select an option via the operation unit 36 (step S342).
[0096] When the operation unit 36 accepts an operation to select one of the options (step S343; Yes), the master image creation unit 302 determines the image data selected as the option as the "product image" (step S344).
[0097] Next, the master image creation unit 302 judges whether the process of determining the product image is completed for all the product codes included in the image information 795 acquired in step S33 (step S345), and ends this process if the process is completed (step S345; Yes). On the other hand, if the process of determining the product image is not completed for all the product codes included in the image information 795 acquired in step S33 (step S345: No), the master image creation unit 302 returns the process to step S341.
[0098] Here, in step S343, the operator (store clerk) of the store server 3 visually checks the options and selects the one he judges to be optimal. If the operator judges that "the options do not have appropriate image data," he can operate the operation unit 36 to reject all of the options. Rejection of all options can be accepted, for example, by displaying an operator for accepting rejection together with the options on the display unit 35 and accepting it on the operation unit 36.
[0099] In step S343, if all of the options are rejected (step S343; No), the master image creation unit 302 advances the process to step S345.
[0100] If only one candidate is selected in step S341, only one option is displayed in step S342. In this case, the selection received in the process of step S343 is whether to accept or reject the option.
[0101] 15, the following describes the update of the collation dictionaries 194, 394, 794 and the product masters 392, 592. The control unit 30 of the collation terminal 7 functions as the image information transfer unit 703 and transfers the image information 795 to the store server 3 (step S75). Note that the image information 795 may be deleted after being transferred in step S75.
[0102] When the shop server 3 receives the image information 795, the control unit 30, functioning as the image information transfer unit 303, transfers it to the server 1 (step S36).
[0103] When the server 1 receives the image information 795, the control unit 10, as the image information storage unit 103, updates the image information 195 (step S13). That is, the image information storage unit 103 adds the image information 795 received from the store server 3 to the storage unit 19 as the image information 195.
[0104] Here, the image information storage unit 103 functions as an example of a code information acquisition unit that acquires code information that enables a product to be identified by reading a code symbol displayed on the product's packaging, and an image data acquisition unit that acquires image data of the product captured when the code symbol is read.
[0105] Next, the process performed outside business hours will be described. The control unit 10 of the server 1 functions as the matching dictionary update unit 104 and updates the matching dictionary 194 (step S14). In this update, the matching dictionary update unit 104 uses the record that is the latest increment (the record added in the previous step S13) among the image information 195. The matching dictionary update unit 104 performs a predetermined process on the image data of each record to obtain characteristic information, and adds the product data of the record associated with the characteristic information to the matching dictionary 194. In this step, automatic learning (registration, update) of new package information in the matching dictionary 194 is realized.
[0106] Here, the comparison dictionary update unit 104 functions as an example of a characteristic information acquisition unit that acquires characteristic information, which is information indicating the external characteristics of a product, from image data.
[0107] Next, the control unit 10, functioning as the master distribution unit 105, distributes the product master 192 (step S15) and the matching dictionary 194 (step S16) to the store server 3 of each store.
[0108] First, in step S15, the master distribution unit 105 transmits records that have been changed since the previous time among records that match the store code of the destination store in the product master 192. In addition, for records that have been changed from available to unavailable since the previous time, the master distribution unit 105 transmits an instruction to the store server 3 to delete the record of the corresponding product code.
[0109] In step S16, the master distribution unit 105 extracts and distributes suitable records when distributing the matching dictionary 194. That is, the master distribution unit 105 extracts product codes that have been traded with a specified store code (of the store to which the data is to be distributed) from the product master 192, makes a list of the extracted product codes, extracts records associated with the product codes included in the list from the matching dictionary 194, and transmits the extracted records to the store server 3.
[0110] When the store server 3 receives the product master 192 and the matching dictionary 194 from the server 1, the control unit 30, functioning as the master update unit 305, updates the product master 392 (step S37) and updates the matching dictionary 394 (step S38).
[0111] When the product master 392 and the matching dictionary 394 are updated, the control unit 30 of the store server 3, as the master distribution unit 306, distributes the product master 392 to the POS terminals 5 in the store (step S39), and also distributes the matching dictionary 394 to the matching terminal 7 (step S40).
[0112] In the POS terminal 5 that has received the product master 392, the control unit 50 functions as the master update unit 505 and updates the product master 592 (step S54).
[0113] In the collation terminal 7 that has received the collation dictionary 394, the control unit 70 functions as the collation dictionary update unit 704 and updates the collation dictionary 794 (step S76).
[0114] In the product registration system 100 configured as above, image recognition is performed by the verification terminal 7 in parallel with decoding of the code symbol by the code scanner 6, and product registration and settlement are performed based on the code information obtained early.
[0115] The image data output by the code scanner 6 is input to the collation terminal 7. The collation terminal 7 generates information (characteristic information) indicating the external characteristics (shape, color, pattern, etc.) of the product captured in the captured image indicated by the image data, and compares the characteristic information with the collation data by referring to the collation dictionary 794.
[0116] The collation terminal 7 identifies the product by image recognition as described above, and returns the result to the code scanner 6. The recognition result output by the collation terminal 7 is output to the POS terminal 5 via the code scanner 6.
[0117] The POS terminal 5 registers the product and makes a payment for the registered product based on the code information output by the code scanner 6. The POS terminal 5 automatically registers the information of the identified product as a purchase target (transaction target).
[0118] Of the image data obtained by the code scanner 6 and transmitted to the verification terminal 7, those for which there is no verification data matching the characteristic information in the verification dictionary 793 are transmitted to the server 1 together with the product code specified by the code scanner 6. The server 1 generates verification data from the newly added (updated) image data and updates the verification dictionary 194. The additions (updates) to the verification dictionary 194 are transmitted to the verification terminal 7. This enables the verification terminal 7 to obtain verification data for products that could not be verified last time.
[0119] The above-described embodiment can be modified as appropriate by changing part of the configuration or function of each of the above-described devices.
[0120] For example, in implementing the present invention, a functional unit equivalent to the master image creation unit 302 may be provided in another device in the store, such as the POS terminal 5, instead of the store server 3. Also, in implementing the present invention, a functional unit equivalent to the master image creation unit 302 may be provided in the server 1.
[0121] In addition, in implementing the present invention, a functional unit equivalent to the master image creation unit 302 provided in the server 1 may be configured to be called and used from another device in the store, such as the store server 3. In this case, if the functional unit equivalent to the master image creation unit 302 provided in the server 1 is provided as an API (Application Programming Interface), it is easy to use the functional unit from an external device.
[0122] In addition, in the above embodiment, it has been described that the "product image" is determined by visual inspection by a store clerk, but in practice, visual inspection by a store clerk may be omitted and the image that is mechanically determined to be optimal may be determined.
[0123] In the above embodiment, the image data and the product code acquired at the time of product registration are used to create the product image to be stored in the product master 192, but the present invention is not limited to this. For example, the present invention may be configured such that, at a timing other than the timing of product registration, the POS terminal 5, the code scanner 6, and the collation terminal 7 are used when they are free, the image data and the product code are acquired by the code scanner 6, a product image candidate is selected from the image data, and the selection of the candidate is accepted by the display unit 55 and the operation unit 56 of the POS terminal 5 to determine the product image. Furthermore, in this case, instead of the code scanner 6, another imaging device that handles digital images, such as a Web camera, may be used. This allows the processing equivalent to the processing performed by the master image creation unit 302 to be performed.
[0124] Furthermore, in the implementation, the POS terminal 5 may be operated by a store clerk or may be operated by a customer. In the implementation, the server 1 may be composed of multiple devices rather than a single device. In the implementation, the functional unit described as being possessed by the server 1 may be performed by any of the devices in the store. In the implementation, a device equivalent to the server 1 may be provided in the store. In the implementation, the store server 3 may not be required. In the implementation, the collation terminal 7 may be shared by multiple code scanners 6. In the implementation, the memory unit 19 (39, 59, 79) is an example of a device that provides a memory area capable of storing various information, and the information that is stored in the same memory unit in the above embodiment does not need to be stored in the same memory unit in the implementation, and may be stored separately in multiple storage devices.
[0125] The programs executed by each device of the above-described embodiments are provided by being pre-installed in a ROM, etc. The programs executed by each device of the above-described embodiments may be provided by being recorded in an installable or executable format on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD).
[0126] Furthermore, the programs executed by each device of the above-described embodiments may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the programs executed by each device of the above-described embodiments may be provided or distributed via a network such as the Internet.
[0127] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]
[0128] 100...Product registration system, 1 ... server, 10 ... control unit, 101: transaction information update unit; 102: master image update unit; 103: image information storage unit; 104: Matching dictionary update unit; 105: Master distribution unit; 11...CPU, 12...ROM, 13...RAM, 14...Communication section, 19...Storage section, 191...program, 192...product master, 193...transaction information, 194... Matching dictionary, 195... Image information, 2 …Network, 3...Store server, 30 ... control unit, 301: transaction information update unit; 302: master image creation unit; 303: image information transfer unit; 305...master update unit, 306...master distribution unit, 31...CPU, 32...ROM, 33...RAM, 34...Communication section, 35...Display section, 36...Operation section, 39...Storage section, 391...program, 392...product master, 393...transaction information, 394...collation dictionary, 5 … POS terminals, 50: control unit; 501: product registration unit; 502: payment processing unit; 505: master update unit; 51...CPU, 52...ROM, 53...RAM, 54...Communication section, 55 ... display unit, 56 ... operation unit, 57 ... voice output unit, 58 ... deposit and withdrawal unit, 59 ... memory unit, 591 ... program, 592 ... product master, 593 ... registered product information, 6 ... code scanner, 60 ... control unit, 61 ... CPU, 62 ... ROM, 63 ... RAM, 64 ... communication unit, 65 ... imaging unit, 7 …collation terminal, 70 ... control unit, 701: Collation unit; 702: Image information storage unit; 703: Image information transfer unit; 704: Matching dictionary update unit; 71...CPU, 72...ROM, 73...RAM, 74...Communication section, 79 ... memory unit, 793 ... dictionary for matching, 794 ... dictionary for matching, 795 ... image information, 9...Network. [Prior art documents] [Patent documents]
[0129] [Patent Document 1] JP 2019-021105 A
Claims
1. a code information acquiring unit that acquires code information for identifying a product by reading a code symbol displayed on a package of the product; an image data acquisition unit that acquires image data of the product captured when the code symbol is read; a characteristic information acquiring unit that acquires characteristic information indicating characteristics of an external appearance of the product from the image data; a dictionary output unit that outputs a comparison dictionary in which one piece of code information is associated with one or more pieces of feature information and stored; a master image creation unit that creates a product image to be stored in association with the code information based on the image data; An information processing device comprising:
2. The master image creation unit selects, as the product image, from the image data, an area that includes a product name. The information processing device according to claim 1 .
3. The master image creation unit selects, as the product image, one of the image data that does not include a code symbol. The information processing device according to claim 1 .
4. The master image creation unit selects, as the product image, an image of the product facing a surface on which a name of the product is displayed from among the image data. The information processing device according to claim 1 .
5. A computer included in a system for registering products related to sales data processing, a code information acquiring unit that acquires code information for identifying a product by reading a code symbol displayed on a package of the product; an image data acquisition unit that acquires image data of the product captured when the code symbol is read; a characteristic information acquiring unit that acquires characteristic information indicating characteristics of an external appearance of the product from the image data; a dictionary output unit that outputs a comparison dictionary in which one piece of code information is associated with one or more pieces of feature information and stored; a master image creation unit that creates a product image to be stored in association with the code information based on the image data; A program to function as a
Citation Information
Patent Citations
Information processing device and program
JP2016110457A
Learning dataset automatic generation system, server, and learning dataset automatic generation program
JP2020095537A
Product recognition device, product recognition system, product recognition method, and program
JP7125179B1
Sales registration apparatus, program, and sales registration method
WO2015147333A1
Product reading device and product reading program
JP2019021105A
