Information processing apparatus and program

Abstract

To provide an information processing apparatus that can appropriately control processing related to image recognition and processing of uploading an image used for the image recognition to a server.SOLUTION: An information processing apparatus has a function to communicate with an external storage device, and comprises image recognition means, transmission means, and stopping means. The image recognition means performs image recognition on a picked-up image picked up by an imaging unit. The transmission means transmits, to the external storage device, the picked-up image on which the image recognition means has performed image recognition. When the transmission means transmits the picked-up image to the external storage device, the stopping means stops image recognition performed by the image recognition means.SELECTED DRAWING: Figure 14

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing device and a program. [Background technology]

[0002] POS (Point Of Sales) systems are used in stores that sell products and services (hereinafter collectively referred to as "products"). This type of POS system is equipped with a scanner that reads product codes from barcodes (hereinafter referred to as "product barcodes") attached to products. Product barcodes include one-dimensional codes and two-dimensional codes. In one example, the product barcode is printed on a sticker and affixed to the product. In another example, the product barcode is printed on the packaging that contains the product. The POS system registers information about the product that the customer is about to purchase based on the product code read by the scanner.

[0003] The POS system operator must search for the product barcode on the product and hold it up to the scanner, making the operation of reading the product code cumbersome. As a result, the operator must register product information (hereinafter also referred to as "product registration") in a time-consuming manner. In particular, in so-called self-service POS systems, customers who are not familiar with product registration must scan product barcodes, so there is a demand for improving the operability of reading product codes, in other words, the operability of the operator to identify products.

[0004] In such a POS system, image recognition technology can be used to identify and register products based on their appearance and other characteristics, and images taken when registering the products can be stored on a cloud server for AI learning. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-21168 A [Patent Document 2] JP 2023-17957 A Summary of the Invention [Problem to be solved by the invention]

[0006] The problem to be solved by the present invention is to provide an information processing device and a program capable of appropriately controlling processing relating to image recognition and processing for uploading images used in the image recognition to a server. [Means for solving the problem]

[0007] An information processing device according to an embodiment is an information processing device having a communication function with an external storage device, and includes an image recognition unit, a transmission unit, and a stop unit. The image recognition unit performs image recognition on an image captured by an imaging unit. The transmission unit transmits the captured image on which the image recognition unit has performed image recognition to the external storage device. The stop unit stops the image recognition performed by the image recognition unit when the transmission unit transmits the captured image to the external storage device. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a product information management system according to an embodiment. [Diagram 2] FIG. 2 is a perspective view showing the appearance of the self-service POS system according to the embodiment. [Diagram 3] FIG. 3 is a block diagram illustrating a hardware configuration of the POS terminal according to the embodiment. [Figure 4] FIG. 4 is a block diagram showing a hardware configuration of the scanner according to the embodiment. [Diagram 5] FIG. 5 is a block diagram illustrating a hardware configuration of the image recognition apparatus according to the embodiment. [Figure 6] FIG. 6 is a block diagram illustrating a hardware configuration of the server according to the embodiment. [Figure 7]FIG. 7 is a diagram showing a data configuration of a product master stored in the storage unit of the POS terminal of the embodiment. [Figure 8] FIG. 8 is a diagram showing a data structure of the comparison dictionary stored in the storage unit of the image recognition device according to the embodiment. [Figure 9] FIG. 9 is a block diagram illustrating a functional configuration of a control unit of the POS terminal according to the embodiment. [Figure 10] FIG. 10 is a block diagram showing the functional configuration of the control unit of the scanner according to the embodiment. [Figure 11] FIG. 11 is a block diagram illustrating a functional configuration of a control unit of the image recognition device according to the embodiment. [Figure 12] FIG. 12 is a block diagram illustrating a functional configuration of a control unit of the server according to the embodiment. [Figure 13] FIG. 13 is a flowchart showing the flow of operations of the scanner, the image recognition device, and the server according to the embodiment. [Figure 14] FIG. 14 is a flowchart showing the flow of operations of the scanner and image recognition device according to the embodiment. [Figure 15] FIG. 15 is a flowchart showing the flow of operations of the scanner according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of an information processing device and a program will be described with reference to the drawings. Note that the present invention is not limited to the embodiment described below.

[0010] (Product Information Management System) First, a product information management system MS according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing the overall configuration of the product information management system MS according to an embodiment. The product information management system MS includes a product information registration system RS and a server 40.

[0011] The product information registration system RS includes a POS terminal 10, a scanner 20, and an image recognition device 30. The POS terminal 10 and the scanner 20 are communicatively connected via a LAN (Local Area Network) or a USB (Universal Serial Bus). The scanner 20 and the image recognition device 30 are communicatively connected via a LAN.

[0012] In the embodiment, a combination of Ethernet (registered trademark) and TCP (Transmission Control Protocol) is used for LAN communication, but the communication protocol is not limited.

[0013] The POS terminal 10 and the image recognition device 30 are communicatively connected to a LAN hub LH via a LAN. The LAN hub LH is communicatively connected to a wireless router WR via a LAN. The wireless router WR is capable of wireless communication with a server 40 via a network NW.

[0014] The scanner 20 captures an image of a product to obtain a product image, which is a captured image. The scanner 20 transmits the obtained product image to the image recognition device 30 via a LAN. The scanner 20 also detects and decodes a product barcode (code information: one-dimensional code and two-dimensional code) included in the obtained product image. That is, the scanner 20 reads information held by the product barcode. The decoded result of the product barcode includes the product code held by the product barcode. The scanner 20 transmits the decoded result to the image recognition device 30 via a LAN.

[0015] The image recognition device 30 receives the product image, which is the captured image from the scanner 20, via a LAN. The image recognition device 30 has an image recognition function using a trained model by machine learning, for example, and performs image recognition on the product image using the image recognition function. In detail, the image recognition device 30 acquires a feature amount included in the product image using the image recognition function, compares the acquired feature amount with a matching dictionary in which a plurality of feature amounts and product codes are associated, and acquires a product code associated with the feature amount that is closest to the acquired feature amount. That is, the image recognition result includes the product code of the product shown in the product image. The image recognition device 30 may perform image recognition on the product image using a method such as deep learning. The image recognition device 30 transmits the image recognition result to the scanner 20 via a LAN.

[0016] The scanner 20 receives the image recognition result from the image recognition device 30 via the LAN. The scanner 20 merges the decoded result and the image recognition result. Here, merging means obtaining one product code from the decoded result and the image recognition result. Therefore, the merged result includes the product barcode contained in the product image or the product code of the product shown in the product image. The scanner 20 transmits the merged result to the POS terminal 10 via the LAN or USB. The POS terminal 10 registers the product based on the product code of the merged result received from the scanner 20.

[0017] Furthermore, the image recognition device 30 determines whether the product image received from the scanner 20 is to be used for additional learning or for analysis of various conditions (condition analysis). For example, when the difference between the acquired feature amount and the feature amount closest to the acquired feature amount is equal to or greater than a predetermined amount, the image recognition device 30 determines that the product image received from the scanner 20 is a product image to be used for additional learning or condition analysis. The image recognition device 30 stores the product image determined to be used for additional learning or condition analysis and the product code of the decoded result received from the scanner 20 in a storage unit, and transmits (uploads) them to a server 40 (external storage device) via a LAN at a preset timing.

[0018] In the image recognition device 30, the timing of uploading the product images and product codes determined to be used for additional learning and status analysis to the server 40 may be a timing preset on the image recognition device 30 side or the server 40 side, or a timing arbitrarily set by the user. For example, in the image recognition device 30, uploading may be performed at predetermined time intervals preset, or when the free space of the storage unit for storing the product images and product codes falls below a predetermined level.

[0019] In addition, the product images uploaded from the image recognition device 30 to the server 40 are not limited to product images intended for use in additional learning or state analysis, and product images intended for use for other purposes may be uploaded.

[0020] In the image recognition device 30, the process of recognizing a product image and acquiring a product code, and the process of uploading the product image and the product code to the server 40 are both processes that involve a large amount of data to be processed, so the processing load is high, and performing both processes simultaneously will result in effects such as a decrease in processing speed and a decrease in recognition accuracy. For this reason, in the product information management system MS of the embodiment, when performing a process of uploading a product image saved by the image recognition device 30 to the server 40, the process of recognizing the product image and acquiring a product code is not performed, and the product is identified by a process of detecting the product barcode with the scanner 20 and acquiring the product code. This avoids any effects on the image recognition device 30 without stopping product identification.

[0021] The server 40 receives, via the LAN, the product images and product codes uploaded from the image recognition device 30. The server 40 stores the received product images and product codes in association with each other.

[0022] The server 40 performs additional learning of a trained model for product image recognition using the accumulated product images and product codes. The server 40 updates the trained model through the additional learning. The server 40 also updates the matching dictionary in conjunction with the additional learning. Updating the matching dictionary includes correcting existing records and adding new records. The server 40 transmits the updated trained model and the updated matching dictionary to the image recognition device 30 via a LAN.

[0023] The image recognition device 30 receives the updated trained model and the updated matching dictionary from the server 40 via a LAN. The image recognition device 30 updates the currently held trained model with the received updated trained model. The image recognition device 30 also updates the currently held matching dictionary with the received updated matching dictionary.

[0024] In this product information management system MS, if the product packaging is changed, it is possible to eliminate the need to collect new product images for additional training of the trained model for product image recognition.

[0025] The server 40 also uses the product images uploaded and stored from the image recognition device 30, for example, to analyze the state (event) of image recognition in the image recognition device 30.

[0026] Note that there may be a plurality of servers 40 on the network NW to which product images are uploaded from the image recognition device 30. For example, product images used for additional learning may be uploaded to the server 40, and product images used for condition analysis may be uploaded to another server (external storage device). In this case, product codes do not necessarily need to be added to product images used for condition analysis. For example, to determine the product image to be analyzed from the uploaded product images, the determination may be made based on the creation date and time (timestamp) recorded as attribute information (properties) of the product image.

[0027] The server 40 also manages the product master. The POS terminal 10 transmits transaction processing information (information related to a processed transaction) via the LAN to the wireless router WR via the LAN hub LH. The wireless router WR transmits the received transaction processing information to the server 40 via the network NW. The server 40 manages the product master based on the received transaction processing information. For example, the server 40 receives transaction processing information from multiple POS terminals 10 and manages the product master for each of the multiple POS terminals 10. Managing the product master includes modifying existing records and adding new records. The server 40 also transmits the product master suitable for each POS terminal 10 to that POS terminal 10 via the network NW, the wireless router WR, and the LAN hub LH.

[0028] (Product Information Registration System) Hereinafter, as an example of a product information registration system, a self-service POS system operated by a customer who purchases a product will be described. However, the product information registration system is not limited to a self-service POS system. The product information registration system may be a POS system operated by a salesperson at a store that sells the product.

[0029] 2 is a perspective view showing the appearance of the self-service POS system 1. The self-service POS system 1 has a POS terminal 10, a scanner 20 built into the POS terminal 10, and an image recognition device 30 built into the POS terminal 10. Here, the self-service POS system 1 is an example of a product information registration system, the POS terminal 10 is an example of a registration device, and the scanner 20 and the image recognition device 30 are examples of an information processing device.

[0030] The scanner 20 identifies the product that the customer purchases, and transmits the product code of the identified product to the POS terminal 10. The product code is an example of product identification information that identifies the product. The POS terminal 10 executes product registration based on the product code input from the scanner 20, and also executes accounting processing for the registered product. The accounting processing is processing executed by the POS terminal 10 to allow the customer to pay for the purchase price of the product.

[0031] 2, the POS terminal 10 includes a main body 50, a first basket mounting stand 70, and a second basket mounting stand 80. The main body 50 includes a display 51 and a touch panel 52 on an upper portion, and also houses a printer 53, a card reader 54, and a change dispenser 55 (see FIG. 3 for each).

[0032] Display 51 is a display device such as an LCD (Liquid Crystal Display) and displays various information. For example, display 51 displays product information such as the product name and price of a product registered by a customer's registration operation. Display 51 also displays accounting information such as the total amount of the transaction and the amount of change in accordance with the customer's accounting operation.

[0033] The touch panel 52 is provided on the surface of the display 51, and inputs information according to the touched position into the control unit 100 (see FIG. 3) of the POS terminal 10. A customer or an administrator (both of whom are operators who are users) who uses the POS terminal 10 can operate the touch panel 52 by touching various buttons and operation keys displayed on the display 51.

[0034] As shown in FIG. 2, below the display 51, a reading window 230 for the scanner 20, an issuing port 531 for receipts printed by the printer 53, and a card insertion port 541 for the card reader 54 are formed facing the direction in which the customer is positioned.

[0035] The reading window 230 is a window through which the scanner 20 captures an image of the exterior of a commodity. A customer who is an operator operating the self-service POS system 1 performs an operation related to commodity registration by holding the commodity to be purchased over the reading window 230. The customer holds the commodity over the reading window 230 while changing the orientation of the commodity with respect to the reading window 230 until the self-service POS system 1 executes the commodity registration and generates a registration completion sound.

[0036] The issuing port 531 is an opening through which a receipt printed by the printer 53 is ejected. Information (transaction processing information) related to a transaction processed by the POS terminal 10 is printed on the receipt. The POS terminal 10 may be configured to issue an electronic receipt instead of a paper receipt printed by the printer 53.

[0037] Card insertion slot 541 is an opening for inserting a credit card or the like to be read by card reader 54. A card is inserted through card insertion slot 541 when a customer makes a payment with a credit card or when a customer has a store membership card or the like read.

[0038] The main body 50 is also formed with a coin insertion slot 551, a coin ejection slot 552, a coin receiving tray 553, a bill insertion slot 554, and a bill ejection slot 555 in correspondence with the position of the change machine 55 housed therein.

[0039] The coin slot 551 is an opening through which a customer inserts coins to be paid. Coins inserted into the coin slot 551 are stored in the change machine 55. The coin outlet 552 is an opening through which coins to be dispensed from the change machine 55 are discharged. The coin outlet 552 discharges change or coins inserted into the coin slot 551 but not accepted by the change machine 55. The coin tray 553 is a tray that receives coins discharged from the coin outlet 552. The banknote slot 554 is an opening through which a customer inserts banknotes to be paid. Banknotes inserted into the banknote slot 554 are stored in the change machine 55. The banknote outlet 555 is an opening through which banknotes that are change to be dispensed from the change machine 55 are discharged.

[0040] The first basket placement table 70 is a table on which a shopping basket (not shown) containing products that a customer wishes to purchase is placed. The first basket placement table 70 includes a first weighing unit 71. The first weighing unit 71 weighs the total weight of a shopping basket placed on the first basket placement table 70 and the products contained in the shopping basket.

[0041] The second basket placement table 80 is a table for placing shopping baskets (not shown) that contain products that have been removed from the shopping basket placed on the first basket placement table 70 and whose product codes have been read, i.e., registered products. The second basket placement table 80 includes a second weighing unit 81, a pair of support rods 82, and a temporary placement table 83.

[0042] The second weighing unit 81 weighs the total weight of a shopping basket placed on the second basket placement table 80 and the products contained in the shopping basket, or the total weight of a shopping bag (not shown) supported by a pair of support rods 82 and the products contained in the shopping bag. The support rods 82 are rods for hanging and supporting a shopping bag containing registered products. The support rods 82 support the shopping bag in an open state, making it easier to put products into the shopping bag. The temporary placement table 83 is a table on which registered products that are easily broken or deformed, such as tofu and eggs, are temporarily placed. The customer can finally place the products placed on the temporary placement table 83 into a shopping basket or shopping bag.

[0043] The customer registers the product by holding the product taken out of the shopping basket placed on the first basket placement stand 70 over the reading window 230. The customer then places the registered product in a shopping basket placed on the second basket placement stand 80 or in a shopping bag supported by the support rod 82.

[0044] The control unit 100 (see FIG. 3) recognizes the weight change when the first weighing unit 71 weighs the weight before and after an item is removed from the shopping basket placed on the first basket placement table 70. The control unit 100 recognizes the weight change when the second weighing unit 81 weighs the weight before and after an item is placed in the shopping basket placed on the second basket placement table 80 or a shopping bag supported by the support rod 82. Depending on whether these two weight changes are equal, the control unit 100 can determine whether all of the items the customer wishes to purchase have been registered.

[0045] The main body 50 also includes a cylindrical pole 57 extending upward. The pole 57 has a light-emitting section 571 at its upper end that emits light when calling a store clerk (also called the manager of the POS terminal 10).

[0046] (POS terminal hardware configuration) Next, the hardware configuration of the POS terminal 10 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the hardware configuration of the POS terminal 10.

[0047] The POS terminal 10 includes a control unit 100, a memory unit 110, a display 51, a touch panel 52, a printer 53, a card reader 54, a change machine 55, a first weighing unit 71, a second weighing unit 81, a light-emitting unit 571, an audio output unit 120, a connection interface 121, and a LAN interface 122. The control unit 100, the memory unit 110, the display 51, the touch panel 52, the printer 53, the card reader 54, the change machine 55, the first weighing unit 71, the second weighing unit 81, the light-emitting unit 571, the audio output unit 120, the connection interface 121, and the LAN interface 122 are connected to each other via a bus 123 or the like.

[0048] The control unit 100 is configured as a computer including a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103. The CPU 101, the ROM 102, and the RAM 103 are connected to each other via a bus 123.

[0049] The CPU 101 controls the entire POS terminal 10. The ROM 102 stores various programs, such as a program used to drive the CPU 101, and various data. The RAM 103 is used as a work area for the CPU 101, and loads various programs and data stored in the ROM 102 and the storage unit 110. The control unit 100 executes various control processes of the POS terminal 10, for example, by the CPU 101 operating in accordance with a control program 111 stored in the storage unit 110 and loaded in the RAM 103.

[0050] RAM 103 has a product information section 1031. Product information section 1031 stores product information read from product master 112 (described later) based on the product code acquired from scanner 20. A product is registered by storing (registering) product information in product information section 1031. Product information stored in product information section 1031 includes product code, product name, price, etc. Product information section 1031 stores product information related to one transaction, and is cleared when the accounting process for that transaction is completed.

[0051] The storage unit 110 is configured with a storage medium such as a hard disk drive (HDD) or a flash memory, and maintains the stored contents even when the power is cut off. The storage unit 110 stores a control program 111 and a product master 112.

[0052] The control program 111 is a program for realizing a function of transmitting an imaging start signal to the scanner 20 to enable the operation of the scanner 20 in response to input of transaction start information declaring the start of one transaction from the touch panel 52, a function of receiving a product code from the scanner 20, a function of executing product registration based on the received product code, a function of transmitting an imaging stop signal to the scanner 20 to disable the operation of the scanner 20 in response to input of registration completion information indicating that product registration of the product related to one transaction has been completed from the touch panel 52, a function of executing accounting processing of the registered product, a function of executing settings of various operation modes in the product information registration system RS in response to an operation by an administrator, etc. The control program 111 may be stored in the ROM 102 instead of being stored in the storage unit 110.

[0053] The product master 112 is a master file that stores product information about products handled in the store. Since the products handled in the store change daily, the product master 112 is updated as appropriate by a store server (not shown) connected via a LAN interface 122. Fig. 7 is a diagram showing the data configuration of the product master 112. The product master 112 includes product codes, product categories, product names, and prices associated with the product codes.

[0054] The product code is information that identifies a product. The product classification is information that classifies products sold in a store. The product name is information that indicates the name of the product. The price is information that indicates the price (unit price) of the product. The product master 112 may include other information in association with the product code.

[0055] Returning to Fig. 3, the description will continue of the hardware configuration of the POS terminal 10. The configurations and functions of the display 51 and the touch panel 52 are as described above.

[0056] The printer 53 is, for example, a thermal printer. The printer 53 prints information about the transaction on a roll of receipt paper and issues the paper as a receipt from an issuing port 531.

[0057] Card reader 54 reads credit information required for payment from a credit card inserted through card insertion slot 541. Card reader 54 also reads membership information identifying a member from a medium such as a membership card or a point card inserted through card insertion slot 541. Card reader 54 may magnetically read information from a card, or may be an IC card reader that reads information from an IC card with a built-in IC (Integrated Circuit) chip.

[0058] The change machine 55 accepts bills and coins paid by a customer when the customer pays for the purchase price of a product in cash, and dispenses change as necessary. Bills are inserted through a bill insertion slot 554, and coins are inserted through a coin insertion slot 551. Bills as change are discharged from a bill discharge slot 555, and coins as change are discharged from a coin discharge slot 552.

[0059] The configurations and functions of first weighing unit 71, second weighing unit 81, and light-emitting unit 571 are as described above. Audio output unit 120 generates sound as necessary to notify the customer of information. For example, audio output unit 120 outputs an alarm sound that notifies the customer that product registration has been completed. The audio output unit is an example of an alarm unit.

[0060] The connection interface 121 is an interface for communicating with the scanner 20 via a LAN or a USB. The control unit 100 transmits and receives information (data) to and from the scanner 20 via the connection interface 121 via the LAN or the USB. For example, the control unit 100 receives a product code from the scanner 20 via the connection interface 121 via the LAN or the USB.

[0061] The LAN interface 122 is an interface for communicating with the LAN hub LH via the LAN. The control unit 100 transmits and receives information (data) to and from the LAN hub LH via the LAN interface 122. For example, the control unit 100 transmits and receives the product master 112 to and from the server 40 via the LAN, via the LAN hub LH and the wireless router WR.

[0062] The LAN interface 122 is also an interface for communicating with external devices such as a store server (not shown) managed by the store. The control unit 100 transmits and receives information to and from external devices via the LAN interface 122 and the LAN.

[0063] (Scanner hardware configuration) Next, the hardware configuration of the scanner 20 will be described with reference to Fig. 4. Fig. 4 is a diagram showing the hardware configuration of the scanner 20.

[0064] The scanner 20 captures an image of a product purchased by a customer to obtain a product image. The scanner 20 then detects a product barcode included in the product image, i.e., a barcode attached to the captured product, and decodes the detected product barcode. The decoded product barcode includes the product code held by the product barcode. The scanner 20 also transmits the product image to the image recognition device 30 via a LAN.

[0065] The scanner 20 includes a control unit 200, a storage unit 210, an imaging unit 220, a connection interface 221, and a LAN interface 222. The control unit 200, the storage unit 210, the imaging unit 220, the connection interface 221, and the LAN interface 222 are connected to each other via a bus 223 or the like.

[0066] The control unit 200 is configured as a computer including a CPU 201, a ROM 202, and a RAM 203. The CPU 201, the ROM 202, and the RAM 203 are connected to each other via a bus 223.

[0067] The CPU 201 controls the entire scanner 20. The ROM 202 stores various programs, such as a program used to drive the CPU 201, and various data. The RAM 203 is used as a work area for the CPU 201, and loads various programs and data stored in the ROM 202 and the storage unit 210. The control unit 200 executes various control processes for the scanner 20, for example, by the CPU 201 operating in accordance with a control program 211 stored in the storage unit 210 and loaded in the RAM 203.

[0068] The RAM 203 has a flag section 2031. The flag section 2031 stores an upload flag ON “1” in accordance with an upload notification received from the image recognition device 30 at the timing when the image recognition device 30 uploads a product image and a product code to the server 40. The flag section 2031 also stores an upload flag OFF “0” in accordance with an upload completion notification received from the image recognition device 30 at the timing when the upload of the product image and the product code from the image recognition device 30 to the server 40 is completed.

[0069] The storage unit 210 is configured with a storage medium such as a HDD or a flash memory, and maintains the stored contents even when the power is cut off. The storage unit 210 stores a control program 211 and a processing result 212.

[0070] The control program 211 has a function of starting imaging by the imaging unit 220 in response to receiving a transaction start signal from the POS terminal 10, a function of storing in the RAM 203 a frame image (captured image) of a commodity captured by the imaging unit 220, a function of acquiring a frame image (a commodity image that is a captured image) stored in the RAM 203, a function of detecting a commodity barcode included in the commodity image, a function of decoding the commodity barcode, a function of transmitting the commodity image to the image recognition device 30 when the upload flag of the flag unit 2031 is OFF "0", a function of decoding the commodity barcode ... The control program 211 may be a program for realizing a function of transmitting the code result to the image recognition device 30 and receiving the image recognition result from the image recognition device 30, a function of merging the decoded result and the image recognition result when the upload flag of the flag unit 2031 is OFF "0", a function of transmitting the product code of the merged result to the POS terminal 10 when the upload flag of the flag unit 2031 is OFF "0" and a function of transmitting the product code of the decoded result to the POS terminal 10 when the upload flag is ON "1", a function of stopping imaging by the imaging unit 220 in response to receiving a registration completion signal from the POS terminal 10, etc. The control program 211 may be stored in the ROM 202 instead of in the storage unit 210.

[0071] The processing result 212 includes a decoded result of the product barcode, an image recognition result by the image recognition device 30, and a merged result of merging the decoded result and the image recognition result. The merged result includes a product barcode included in the product image or a product code of the product shown in the product image.

[0072] The imaging unit 220 is composed of, for example, a color CCD image sensor or a color CMOS image sensor. The imaging unit 220 outputs a product image, which is an image acquired by capturing an image of a product through the reading window 230, under the control of the control unit 200. For example, the imaging unit 220 captures a moving image at a predetermined frame rate (e.g., 30 fps). The imaging unit 220 stores the captured frame images in sequence in the RAM 203. The frame images (product images) stored in the RAM 203 are used to identify the product code of the product shown in the frame image.

[0073] The connection interface 221 is an interface for communicating with the POS terminal 10 via a LAN or a USB. The control unit 200 transmits and receives information (data) to and from the POS terminal 10 via the connection interface 221, the LAN, or the USB.

[0074] The LAN interface 222 is an interface for communicating with the image recognition device 30 via a LAN. The control unit 200 transmits and receives information (data) to and from the image recognition device 30 via the LAN interface 222 and the LAN.

[0075] The control unit 200 detects a product barcode included in a frame image (product image) held by the RAM 203, decodes the product barcode, and acquires a product code held by the product barcode. When the upload flag of the flag unit 2031 is OFF "0", the control unit 200 transmits the frame image (product image) held by the RAM 203 to the image recognition device 30 via the LAN interface 222, and receives the image recognition result (product code) by the image recognition device 30 from the image recognition device 30. Furthermore, the control unit 200 merges the decoded result and the image recognition result to identify the product barcode included in the product image or the product code of the product shown in the product image, and transmits the identified product code to the POS terminal 10 via the connection interface 221. When the upload flag of the flag unit 2031 is ON "1", the control unit 200 identifies the product code of the decoded result as the product code of the product shown in the product image, and transmits the identified product code to the POS terminal 10 via the connection interface 221.

[0076] (Hardware configuration of image recognition device) Next, the hardware configuration of the image recognition device 30 will be described with reference to Fig. 5. Fig. 5 is a diagram showing the hardware configuration of the image recognition device 30.

[0077] The image recognition device 30 receives a product image, which is a captured image from the scanner 20, via the LAN in a state in which the product image and product code stored by the image recognition device 30 have not been uploaded from the image recognition device 30 to the server 40. The image recognition device 30 starts image recognition by receiving a captured image from the scanner 20, and stops image recognition when the captured image is no longer received from the scanner 20. The image recognition device 30 performs image recognition on the product image included in the received captured image. That is, the image recognition device 30 acquires a product code of the product shown in the product image based on the feature amount of the product image. The image recognition device 30 transmits the image recognition result to the scanner 20 via the LAN. In addition, the image recognition device 30 transmits (uploads) the product image and product code determined to be used for additional learning or state analysis to the server 40 via the LAN. Furthermore, the image recognition device 30 receives an updated trained model and a matching dictionary from the server 40 via the LAN, and updates the currently held trained model and matching dictionary.

[0078] The image recognition device 30 includes a control unit 300, a storage unit 310, and a LAN interface 321. The control unit 300, the storage unit 310, and the LAN interface 321 are connected to each other via a bus 322 or the like.

[0079] The control unit 300 is configured as a computer including a CPU 301, a ROM 302, and a RAM 303. The CPU 301, the ROM 302, and the RAM 303 are connected to each other via a bus 322.

[0080] The CPU 301 controls the entire image recognition device 30. The ROM 302 stores various programs, such as a program used to drive the CPU 301, and various data. The RAM 303 is used as a work area for the CPU 301, and loads various programs and data stored in the ROM 302 and the storage unit 310. The control unit 300 executes various control processes of the image recognition device 30, for example, by the CPU 301 operating in accordance with a control program 311 stored in the storage unit 310 and loaded in the RAM 303.

[0081] The RAM 303 has a product image section 3031 and a flag section 3032. The product image section 3031 stores (saves) a product image, which is a captured image received from the scanner 20, as a target for image recognition. The product image section 3031 stores (saves) a product code, which is a decoded result received from the scanner 20, corresponding to the product image.

[0082] The flag unit 3032 stores the upload flag ON "1" when uploading the product images and product codes determined to be used for additional learning or status analysis from among the product images and product codes stored in the product image unit 3031 to the server 40 at a timing preset by the image recognition device 30. The flag unit 3032 also stores the upload flag OFF "0" at the timing when the upload of the product images and product codes from the image recognition device 30 to the server 40 is completed.

[0083] Among the product images and product codes stored in the product image section 3031, the product images and product codes that are not determined to be used for additional learning or status analysis may be deleted.

[0084] The storage unit 310 is configured with a storage medium such as a HDD or a flash memory, and maintains the stored contents even when the power is cut off. The storage unit 310 stores a control program 311, a trained model 312, and a matching dictionary 313.

[0085] The control program 311 has a function of setting (storing) an upload flag ON “1” in the flag section 3032 at a preset timing and sending an upload notification to the scanner 20, a function of setting (storing) an upload flag OFF “0” in the flag section 3032 when uploading of the captured image and product code to the server 40 is completed and sending an upload completion notification to the scanner 20, a function of receiving a product image, which is a captured image, from the scanner 20 when the upload flag in the flag section 3032 is OFF “0”, a function of performing image recognition on the product image, a function of determining whether the image recognition has been successful, and a function of transmitting the image recognition result to the scanner 20. The control program 311 includes a function for transmitting the product image data, a function for receiving the product code of the decoded result from the scanner 20, a function for determining whether the product image will be used for additional learning or status analysis, a function for retaining (saving) the product image and product code determined to be used for additional learning or status analysis in the product image section 3031 and transmitting (uploading) them to the server 40 when the upload flag in the flag section 3032 is turned ON "1", a function for receiving an updated trained model and an updated matching dictionary from the server 40, a function for updating the currently held trained model with the updated trained model, a function for updating the currently held matching dictionary with the updated matching dictionary, and the like. The control program 311 may be stored in the ROM 302 instead of being stored in the storage section 310.

[0086] The trained model 312 is an image recognition model for performing image recognition on product images. The trained model 312 has been trained by machine learning and is additionally trained by machine learning.

[0087] The matching dictionary 313 is a data table that manages products in association with the features of the appearance of the products. FIG. 8 is a diagram showing the data structure of the matching dictionary 313. The matching dictionary 313 includes product codes and features associated with the product codes. The matching dictionary 313 may have multiple features for one product code. The product codes in the matching dictionary 313 are the same as the product codes in the product master 112 held in the memory unit 110 of the POS terminal 10.

[0088] As described above, the product code is information for identifying a product. The feature amount is, for example, a parameterized feature of the product's appearance, such as the product's standard shape, surface color, pattern, unevenness, etc. The control unit 300 acquires the feature amount included in the product image, compares the acquired feature amount with the collation dictionary, and acquires the product code associated with the acquired feature amount. In other words, the control unit 300 acquires the product code of the product shown in the product image from the products pre-recorded in the collation dictionary 313 based on the feature amount acquired from the product image.

[0089] Returning to FIG. 5, the description of the hardware configuration of the image recognition device 30 will be continued.

[0090] The LAN interface 321 is an interface for communicating with the scanner 20 and the LAN hub LH. The control unit 300 transmits and receives information (data) to and from the scanner 20 and the LAN hub LH via the LAN interface 321.

[0091] For example, the control unit 300 receives a frame image (product image) from the scanner 20 via the LAN interface 321. In addition, the control unit 300 acquires the product code of the product shown in the product image as described above, and transmits the acquired product code to the scanner 20 via the LAN interface 321.

[0092] Furthermore, the control unit 300 transmits (uploads) the product images and product codes that are stored (saved) in the product image unit 3031 and determined to be used for additional learning and status analysis to the server 40 via the LAN interface 321. Furthermore, the control unit 300 receives the updated trained model 412 (see FIG. 6) and the updated matching dictionary 413 (see FIG. 6) from the server 40 via the LAN interface 321. The control unit 300 updates the currently held trained model 312 with the updated trained model 412. Furthermore, the control unit 300 updates the currently held matching dictionary 313 with the updated matching dictionary 413.

[0093] (Server hardware configuration) Next, the hardware configuration of the server 40 will be described with reference to Fig. 6. Fig. 6 is a diagram showing the hardware configuration of the server 40.

[0094] The server 40 receives and stores product images and product codes from the image recognition device 30. The server 40 uses the stored product images and product codes to perform additional learning of a trained model for product image recognition, and updates the trained model and the matching dictionary. The server 40 transmits the updated trained model and the updated matching dictionary to the image recognition device 30. The server 40 also uses the stored product images and product codes as necessary, for example, to analyze the state (event) of image recognition in the image recognition device 30. The server 40 also manages product master data. The server 40 receives transaction processing information from the POS terminal 10, and manages the product master data based on the received transaction processing information. The server 40 transmits product master data suitable for each POS terminal 10 to the POS terminal 10.

[0095] The server 40 includes a control unit 400, a storage unit 410, and a communication unit 421. The control unit 400, the storage unit 410, and the communication unit 421 are connected to each other via a bus 422 or the like.

[0096] The control unit 400 is configured by a computer including a CPU 401, a ROM 402, and a RAM 403. The CPU 401, the ROM 402, and the RAM 403 are connected to each other via a bus 422.

[0097] The CPU 401 controls the entire server 40. The ROM 402 stores various programs, such as a program used to drive the CPU 401, and various data. The RAM 403 is used as a work area for the CPU 401, and loads various programs and data stored in the ROM 402 and the storage unit 410. The control unit 400 executes various control processes of the server 40, for example, by the CPU 401 operating in accordance with a control program 411 stored in the storage unit 410 and loaded in the RAM 403.

[0098] The storage unit 410 is configured with a storage medium such as a HDD or a flash memory, and maintains the stored contents even when the power is cut off. The storage unit 410 stores a control program 411, a trained model 412, a comparison dictionary 413, a product master 414, and a product image section 415.

[0099] The control program 411 is a program for realizing a function of receiving product images and product codes uploaded from the image recognition device 30, a function of storing and accumulating the received product images and product codes in the product image section 415, a function of additionally learning and updating the trained model 412, a function of updating the matching dictionary 413, a function of transmitting the updated trained model 412 and the updated matching dictionary 413 to the image recognition device 30, a function of receiving transaction processing information from the POS terminal 10, a function of managing the product master 414, a function of transmitting the product master 414 to the POS terminal 10, a function of transmitting the product images accumulated in the product image section 415 to an external device as necessary, and the like. The control program 411 may be stored in the ROM 402 instead of in the storage section 410.

[0100] The trained model 412 is a model trained by machine learning for performing image recognition on product images. In detail, the trained model 412 is an image recognition model that acquires a feature amount included in the product image and compares the acquired feature amount with a matching dictionary 413 to acquire a product code of a product shown in the product image.

[0101] The matching dictionary 413 is a data table that manages product codes of products in association with the appearance features of the products, similar to the matching dictionary 313 held by the image recognition device 30. The matching dictionary 413 includes the matching dictionaries 313 held by multiple image recognition devices 30. The server 40 transmits to each image recognition device 30 the matching dictionary 313 suitable for that image recognition device 30.

[0102] Like the product master data 112 held by the POS terminal 10, the product master data 414 includes product codes, and product categories, product names, and prices associated with the product codes. The product master data 414 includes the product master data 112 held by multiple POS terminals 10. The server 40 manages the product master data 414 based on transaction processing information received from the POS terminals 10. For example, the server 40 updates the product master data 414 based on the transaction processing information. The server 40 sends to each POS terminal 10 the product master data 112 suitable for that POS terminal 10.

[0103] The product image section 415 accumulates product images and product codes uploaded from the image recognition device 30 for use in additional learning and condition analysis.

[0104] The communication unit 421 is an interface for communicating with the wireless router WR. The control unit 400 transmits and receives information (data) to and from the wireless router WR via the communication unit 421.

[0105] For example, the control unit 400 receives product images and product codes uploaded from the image recognition device 30 via the communication unit 421 and stores them in the product image unit 415. The control unit 400 performs additional learning of the trained model 412 using the stored product images and product codes, and updates the trained model 412 and the matching dictionary 413. The control unit 400 transmits the updated trained model and the updated matching dictionary to the image recognition device 30 via the communication unit 421. The control unit 400 transmits the stored product images to an external device as necessary.

[0106] The server 40 also receives transaction processing information from the POS terminal 10 via the communication unit 421. The server 40 manages the product master based on the received transaction processing information. The server 40 transmits the product master suitable for each POS terminal 10 to that POS terminal 10 via the communication unit 421.

[0107] (Functional configuration of POS terminal) Next, the functional configuration of the POS terminal 10 will be described with reference to Fig. 9. Fig. 9 is a block diagram showing the functional configuration of the control unit 100 of the POS terminal 10.

[0108] The control unit 100 has an input means 1001, a registration means 1002, an accounting processing means 1003, and an output means 1004. The control unit 100 functions as the input means 1001, the registration means 1002, the accounting processing means 1003, and the output means 1004 by the CPU 101 operating in accordance with a control program stored in the ROM 102 or the storage unit 110. Each of these functions may be configured by hardware such as a dedicated circuit.

[0109] Various information is input to the input means 1001 from the scanner 20, the touch panel 52, the card reader 54, the first weighing unit 71, the second weighing unit 81, etc. For example, a product code is input to the input means 1001 from the scanner 20.

[0110] Furthermore, information based on the operation of the touch panel 52 is input to the input means 1001. For example, transaction start information declaring the start of one transaction is input to the input means 1001. Furthermore, registration completion information indicating that product registration of the product related to one transaction has been completed is input to the input means 1001.

[0111] The registration means 1002 executes product registration of a product that a customer wishes to purchase through an operation of the customer. In other words, the registration means 1002 registers information on a product identified by product identification information inputted to the input means 1001. Specifically, the registration means 1002 reads product information from the product master 112 based on the product code inputted to the input means 1001. Then, the registration means 1002 stores the read product information in the product information section 1031. The product information is information including a product code, a product name, a price, etc.

[0112] The transaction processing means 1003 executes transaction processing related to the payment of the prices of the products registered by the registration means 1002. Specifically, the transaction processing means 1003 executes processing to calculate the total price (transaction amount) of the registered products, calculate the amount of change, display various information, issue receipts, etc. If the customer desires cashless payment such as credit card payment or two-dimensional code payment, the transaction processing means 1003 can execute the transaction processing necessary for payment.

[0113] The output unit 1004 outputs various information to the scanner 20, the printer 53, the change machine 55, the light emitting unit 571, the audio output unit 120, etc. For example, when transaction start information is input to the input unit 1001, the output unit 1004 outputs an imaging start signal that instructs the scanner 20 to start imaging. Also, when registration completion information is input to the input unit 1001, the output unit 1004 outputs an imaging stop signal that instructs the scanner 20 to stop imaging.

[0114] When the registration means 1002 completes the product registration for one product, the output means 1004 outputs a control signal for causing the audio output unit 120 to output an alarm sound. The output means 1004 functions as an alarm means for causing an alarm unit (audio output unit 120) to notify that the registration means 1002 has completed the registration of product information. It is also possible to provide the audio output unit 120 in the scanner 20. In this case, the audio output unit 120 may be configured to emit an alarm sound when the scanner 20 identifies the product code.

[0115] (Scanner Functional Configuration) Next, the functional configuration of the scanner 20 will be described with reference to Fig. 10. Fig. 10 is a block diagram showing the functional configuration of the control unit 200 of the scanner 20.

[0116] The control unit 200 has an acquiring means 2001, a decoding means 2002, a transmitting / receiving means 2003, a merging means 2004, a transmitting means 2005, and a flag setting means 2006. The control unit 200 functions as the acquiring means 2001, the decoding means 2002, the transmitting / receiving means 2003, the merging means 2004, the transmitting means 2005, and the flag setting means 2006 by the CPU 201 operating in accordance with a control program stored in the ROM 202 or the storage unit 210. Each of these functions may be configured by hardware such as a dedicated circuit.

[0117] The acquisition means 2001 acquires one frame image (captured image) as a product image from a moving image captured by the imaging unit 220. The acquisition means 2001 stores the acquired product image in the RAM 203. For example, the acquisition means 2001 acquires all or some of a plurality of frame images sequentially captured by the imaging unit 220 as candidates for product images, stores all or some of the acquired frame images in the RAM 203, and selects a product image from all or some of the stored frame images.

[0118] The acquiring means 2001 also acquires various other information. For example, the acquiring means 2001 acquires an imaging start signal and an imaging stop signal from the POS terminal 10. When the acquiring means 2001 acquires an imaging start signal from the POS terminal 10, the imaging unit 220 starts imaging, and the acquiring means 2001 acquires the captured image captured by the imaging unit 220. When the acquiring means 2001 acquires an imaging stop signal from the POS terminal 10, the imaging unit 220 stops imaging, and the acquiring means 2001 stops acquiring the captured image.

[0119] The decoding means 2002 detects a product barcode included in the product image, which is the captured image acquired by the acquisition means 2001, and extracts the product barcode from the product image. The decoding means 2002 decodes the extracted product barcode. The decoding means 2002 acquires the product code held by the product barcode as a decoding result of the decoding of the product barcode.

[0120] The transmission / reception means 2003 receives, from the image recognition device 30, an upload notification notifying the upload of a product image saved (stored) by the image recognition device 30 to the server 40 and an end notification notifying the end of the upload. In response to the upload notification from the image recognition device 30, the transmission / reception means 2003 transmits to the image recognition device 30 the product image acquired by the acquisition means 2001 in a state in which the upload flag stored (set) in the flag section 2031 of the RAM 203 is ON "1". The image recognition device 30 performs image recognition on the received product image and transmits the image recognition result (product code of the product shown in the product image) to the scanner 20. The transmission / reception means 2003 receives the image recognition result transmitted by the image recognition device 30. In addition, the transmission / reception means 2003 transmits the decoded result (product code held by the product barcode) by the decoding means 2002 to the image recognition device 30 when the upload flag stored (set) in the flag section 2031 of the RAM 203 is ON “1” in response to an upload notification from the image recognition device 30.

[0121] The merging means 2004 merges the decoded result (the product code held by the product barcode) and the recognition result (the product code of the product shown in the product image). That is, the merging means 2004 includes a specifying means for specifying and acquiring one product code from the decoded result and the image recognition result. The merged result includes the product code held by the product barcode included in the product image or the product code of the product shown in the product image.

[0122] If no decoded result is obtained, the merging means 2004 uses the product code of the image recognition result as the merged result. Situations in which no decoded result is obtained include when the consumer does not properly hold the product barcode over the imaging unit 220 of the scanner 20, or when the product barcode is unnaturally bent or folded.

[0123] If both the decoded result and the image recognition result are obtained and match, the merging means 2004 naturally takes the matching product code as the merged result. Also, if both the decoded result and the image recognition result are obtained but do not match, the merging means 2004 takes the product code of the more reliable decoded result as the merged result.

[0124] The transmitting means 2005 transmits the product code, which is the result of merging by the merging means 2004, to the POS terminal 10 via the connection interface 221 when the upload flag stored (set) in the flag section 2031 of the RAM 203 in response to the upload completion notification from the image recognition device 30 is OFF "0". Also, the transmitting means 2005 transmits the product code, which is the result of decoding by the decoding means 2002, to the POS terminal 10 via the connection interface 221 when the upload flag stored (set) in the flag section 2031 of the RAM 203 in response to the upload notification from the image recognition device 30 is ON "1".

[0125] The flag setting means 2006 stores (sets) the upload flag ON “1” in the flag section 2031 of the RAM 203 when an upload notification is received to send (upload) a product image from the image recognition device 30 to the server 40, and stores (sets) the upload flag OFF “0” in the flag section 2031 of the RAM 203 when an upload completion notification is received.

[0126] That is, when the image recognition device 30 is uploading a product image determined to be used for additional learning or status analysis to the server 40, the scanner 20 does not transmit the captured product image to the image recognition device 30, and the image recognition device 30 does not acquire a product code by image recognition of the product image. In this case, the scanner 20 transmits the product code, which is the decoded result of the product barcode, to the POS terminal 10. When the image recognition device 30 is not uploading a product image determined to be used for additional learning or status analysis to the server 40, the scanner 20 transmits the captured product image to the image recognition device 30, and transmits the product code acquired by merging the decoded result of the product barcode and the image recognition result of the product image to the POS terminal 10.

[0127] (Functional configuration of image recognition device) Next, the functional configuration of the image recognition device 30 will be described with reference to Fig. 11. Fig. 11 is a block diagram showing the functional configuration of the control unit 300 of the image recognition device 30.

[0128] The control unit 300 has a receiving means 3001, an image recognition means 3002, a determination means 3003, an update means 3004, a transmission means 3005, and an image upload means 3006. The control unit 300 functions as the receiving means 3001, the image recognition means 3002, the determination means 3003, the update means 3004, the transmission means 3005, and the image upload means 3006 by the CPU 301 operating in accordance with a control program stored in the ROM 302 or the storage unit 310. Each of these functions may be configured by hardware such as a dedicated circuit.

[0129] The receiving means 3001 receives a product image, which is a captured image from the scanner 20, via the LAN interface 321 while the upload flag stored (set) in the flag section 3032 of the RAM 303 is OFF “0”. The receiving means 3001 saves the received product image in the product image section 3031 of the RAM 303.

[0130] The image recognition means 3002 performs image recognition on the product images stored in the product image section 3031 by an image recognition function using the trained model 312. In detail, the image recognition means 3002 acquires features included in the product images, compares the acquired features with the matching dictionary 313, and acquires a product code associated with the feature that is closest to the acquired feature. The image recognition means 3002 stores the acquired product code in the RAM 303.

[0131] The transmission means 3005 transmits the product code, which is the image recognition result by the image recognition means 3002 , to the transmission / reception means 2003 of the scanner 20 via the LAN interface 321 .

[0132] The determination means 3003 determines whether the image recognition by the image recognition means 3002 has been successful. For example, if the difference between the acquired feature amount and the feature amount closest to the acquired feature amount is less than a predetermined amount, the determination means 3003 determines that the image recognition has been successful, and conversely, if the difference is equal to or greater than the predetermined amount, the determination means 3003 determines that the image recognition has failed. In addition, if the determination means 3003 determines that the image recognition has failed, it determines that the product image that has been subjected to image recognition is a product image to be used for additional learning or state analysis.

[0133] Furthermore, in a state where the upload flag stored in the flag section 3032 of the RAM 303 is OFF “0”, the receiving means 3001 receives the product code that is the decoded result from the scanner 20 via the LAN interface 321. The receiving means 3001 stores (saves) the received product code in association with the product image that has been subjected to the current image recognition and that is stored in the product image section 3031 of the RAM 303.

[0134] When the determination means 3003 determines that the product image that has undergone image recognition is a product image to be used for additional learning or status analysis, the determination means 3003 stores the product image and product code determined to be used for additional learning or status analysis in the product image section 3031 of the RAM 303. The product image and product code that have not been determined to be used for additional learning or status analysis are erased from the product image section 3031. The transmission means 3005 transmits (uploads) the product image and product code determined to be used for additional learning or status analysis and stored in the product image section 3031 to the server 40 via the LAN interface 321 at a preset timing. The preset timing includes a timing preset on the image recognition device 30 side or the server 40 side, or a timing arbitrarily set by the user. In the case of the image recognition device 30 of the embodiment, the preset timing is, for example, a timing at a preset time interval or a timing when the free space of the product image section 3031 for storing the product image and the product code becomes equal to or less than a predetermined amount.

[0135] The server 40 performs additional learning of the trained model 412 for product image recognition using the received product image and product code, and updates the trained model 412 and the matching dictionary 413. The server 40 transmits the updated trained model 412 and the updated matching dictionary 413 to the receiving means 3001 of the image recognition device 30 via the LAN interface 321. The server 40 may use the received product image and product code for analyzing various states (state analysis).

[0136] The receiving means 3001 receives the updated trained model 412 and the updated matching dictionary 413 from the server 40 via the LAN interface 321. The receiving means 3001 stores the received updated trained model 412 and updated matching dictionary 413 in the RAM 303.

[0137] The update means 3004 updates the trained model 312 currently held in the storage unit 310 with the received updated trained model 412. In addition, the image recognition device 30 updates the matching dictionary 313 currently held in the storage unit 310 with the received updated matching dictionary 413.

[0138] The image upload means 3006 transmits an inquiry signal to the server 40 at a preset timing (for example, at predetermined time intervals or when the free space in the product image section 3031 falls below a predetermined level) by the transmission means 3005, inquiring about permission to upload a product image. When the image upload means 3006 receives an permission signal permitting uploading from the server 40 by the reception means 3001, the image upload means 3006 sets the upload flag in the flag section 3032 to ON “1” and transmits an upload notification to the scanner 20 by the transmission means 3005. The image upload means 3006 transmits (uploads) the product image and product code determined to be used for additional learning or status analysis, which are stored (saved) in the product image section 3031, to the server 40 by the transmission means 3005.

[0139] Furthermore, when the image uploading means 3006 finishes uploading the product image and product code in the product image section 3031 to the server 40, it erases the uploaded (transmitted) product image and product code. Then, the image uploading means 3006 sets the upload flag in the flag section 3032 to OFF “0”, and transmits an upload completion notification to the scanner 20 by the transmitting means 3005.

[0140] (Server Functionality Configuration) Next, the functional configuration of the server 40 will be described with reference to Fig. 12. Fig. 12 is a block diagram showing the functional configuration of the control unit 400 of the server 40.

[0141] The control unit 400 has a receiving means 4001, an additional learning means 4002, a master management means 4003, a transmitting means 4004, and an image saving means 4005. The control unit 400 functions as the receiving means 4001, the additional learning means 4002, the master management means 4003, the transmitting means 4004, and the image saving means 4005 by the CPU 401 operating in accordance with a control program stored in the ROM 402 or the storage unit 410. Each of these functions may be configured by hardware such as a dedicated circuit.

[0142] The receiving means 4001 receives a product image and its product code for additional learning (for status analysis) to be used for additional learning and status analysis, which are uploaded from the image recognition device 30, via the communication unit 421. The receiving means 4001 stores the received product image and product code in the product image section 415 of the storage unit 410 by the image storage means 4005.

[0143] The additional learning means 4002 performs additional learning on the trained model 412 held in the storage unit 410 at an appropriate timing, for example, when a certain amount of product images and product codes for additional learning have been accumulated in the product image unit 415, and updates the trained model 412. In addition, the additional learning means 4002 updates the matching dictionary 413 held in the storage unit 410 in accordance with the additional learning of the trained model 412.

[0144] The transmitting means 4004 transmits the updated trained model 412 and the matching dictionary 413 updated by the additional learning means 4002 to the receiving means 3001 of the image recognition device 30 via the communication unit 421, for example as an image file or a compressed file.

[0145] The receiving means 4001 also receives transaction processing information from the POS terminal 10 via the communication unit 421. The receiving means 4001 stores the received transaction processing information in RAM 403.

[0146] The master data management means 4003 manages the product master data 414 held in the storage unit 410 based on the received transaction processing information. For example, the product master data 414 includes a store code that identifies the store, the product master data 112 for each store code, and information such as whether or not a product with a product code for each store code is available and the remaining quantity, and the master data management means 4003 updates the product master data 414, for example, information such as the remaining quantity of a product with a product code, based on the received transaction processing information.

[0147] The transmission means 4004 transmits the product master 112 suitable for each store to the POS terminal 10 of the store via the communication unit 421 based on the product master 414 updated by the master management means 4003.

[0148] The image storage means 4005 stores (stores) the product image and product code for additional learning (for status analysis) uploaded from the image recognition device 30 and received by the receiving means 4001 in the product image section 415 of the storage unit 410. When the image storage means 4005 receives an inquiry signal from the image recognition device 30 by the receiving means 4001 inquiring about permission to upload, the image storage means 4005 determines whether the free space of the storage unit 410 is sufficient to store the product image and product code for additional learning (for status analysis) or more than a predetermined amount. When the image storage means 4005 determines that the free space of the storage unit 410 is not sufficient to store the product image and product code for additional learning (for status analysis) or more than a predetermined amount, the image storage means 4005 recovers the free space by, for example, erasing old product images and product codes for additional learning (for status analysis) that have been created for a predetermined period of time since the creation date and time (timestamp). When the image storage means 4005 determines that the free space of the storage unit 410 is sufficient to store the product image and product code for additional learning (for status analysis) or more than a predetermined amount of time, the image storage means 4005 transmits an authorization signal for authorization to upload to the image recognition device 30 by the transmitting means 4004.

[0149] (Operation of scanner, image recognition device and server) Next, with reference to FIG. 13, the operation of the scanner 20, the image recognition device 30, and the server 40 for uploading product images and product codes for additional learning (for status analysis) from the image recognition device 30 to the server 40 will be described. FIG. 13 is a flowchart showing the flow of the operation of the scanner 20, the image recognition device 30, and the server 40 according to the embodiment. In FIG. 13, the operation of the scanner 20 is shown on the left, the operation of the image recognition device 30 is shown in the center, and the operation of the server 40 is shown on the right. The operation of the scanner 20 is performed by each part of the scanner 20 under the control of the control unit 200 that executes the control program 211. The operation of the image recognition device 30 is performed by each part of the image recognition device 30 under the control of the control unit 300 that executes the control program 311. The operation of the server 40 is performed by each part of the server 40 under the control of the control unit 400 that executes the control program 411.

[0150] In ACT41, the image recognition device 30 determines whether a preset predetermined time has elapsed. Every time the image recognition device 30 determines that the predetermined time has elapsed (if Yes in ACT41), in ACT43, the image recognition device 30 transmits an inquiry signal to the server 40 to inquire about permission to upload.

[0151] In ACT42, the image recognition device 30 checks whether the free space in the product image section 3031 of the RAM 303 is equal to or less than a predetermined amount. If the free space in the product image section 3031 is equal to or less than a predetermined amount (Yes in ACT42), the image recognition device 30 transmits an inquiry signal to the server 40 in ACT43, inquiring about permission to upload.

[0152] The inquiry about upload permission in ACT43 is made to the server 40 at a timing preset by the image recognition device 30 (at predetermined time intervals or when the free space in the product image section 3031 falls below a predetermined level). However, the present invention is not limited to this, and the inquiry about uploading may be made from the server 40 to the image recognition device 30 at a timing preset on the server 40 side.

[0153] In ACT61, the server 40 checks whether an inquiry signal has been received from the image recognition device 30. When the server 40 receives an inquiry signal from the image recognition device 30 (if Yes in ACT61), in ACT62, the server 40 determines whether the free space in the storage unit 410 is equal to or greater than a predetermined amount.

[0154] In ACT62, if the server 40 determines that the free space in the memory unit 410 is less than a specified amount (less than a specified amount) (if No in ACT62), in ACT63, the server 40 recovers the free space by performing a process such as erasing old data in the memory unit 410 (a process of organizing data).

[0155] In ACT62, when the server 40 determines that the free space in the storage unit 410 is equal to or greater than a predetermined amount (Yes in ACT62), in ACT64, the server 40 transmits to the image recognition device 30 a permission signal that permits uploading.

[0156] In ACT44, the image recognition device 30 checks whether it has received a permission signal from the server 40. When the image recognition device 30 receives the permission signal from the server 40 (if Yes in ACT44), it sets (stores) the upload flag ON “1” in the flag section 3032 of the RAM 303 in ACT45.

[0157] In ACT46, the image recognition device 30 transmits an upload notification to the scanner 20 to notify the start of uploading.

[0158] In ACT71, the scanner 20 checks whether an upload notification has been received from the image recognition device 30. When the scanner 20 receives an upload notification from the image recognition device 30 (if Yes in ACT71), the scanner 20 sets (stores) the upload flag ON “1” in the flag section 2031 of the RAM 203 in ACT72.

[0159] In ACT47, the image recognition device 30 transmits (uploads) the product image and product code for additional learning (for status analysis) stored in the product image section 3031 of the RAM 303 to the server 40. The image recognition device 30 deletes the product image and product code in the product image section 3031 uploaded to the server 40.

[0160] In ACT65, the server 40 receives product images and product codes for additional learning (for status analysis) uploaded from the image recognition device 30, and in ACT66, it saves (stores) and accumulates the uploaded product images and product codes for additional learning (for status analysis) in the product image section 415 of the memory unit 410.

[0161] In ACT48, the image recognition device 30 checks whether the upload of the product image and product code for additional learning (for status analysis) to the server 40 is completed. When the upload is completed (if Yes in ACT48), the image recognition device 30 sets (stores) the upload flag OFF “0” in the flag section 3032 of the RAM 303 in ACT49.

[0162] In ACT50, the image recognition device 30 transmits a completion notification to the scanner 20 to notify the completion of the upload.

[0163] In ACT73, the scanner 20 checks whether an upload completion notification has been received from the image recognition device 30. When the scanner 20 receives an upload completion notification from the image recognition device 30 (if Yes in ACT73), the scanner 20 sets (stores) the upload flag OFF “0” in the flag section 2031 of the RAM 203 in ACT74.

[0164] In this way, the image recognition device 30 uploads the product images and product codes for additional learning (for status analysis) stored in the product image section 3031 of RAM 303 to the server 40 at preset timing (here, at predetermined time intervals or when the free space in the product image section 3031 falls below a predetermined level).

[0165] During upload processing by the image recognition device 30, the upload flag in the flag section 3032 of the image recognition device 30 is set to ON "1", and the upload flag in the flag section 2031 of the scanner 20 is also set to ON "1".

[0166] In ACT44, if the image recognition device 30 does not receive an permission signal from the server 40 within a certain period of time, for example (if No in ACT44), in ACT51, the image recognition device 30 sends an error notification to the scanner 20 indicating that upload is not possible.

[0167] In ACT75, when the scanner 20 receives an error notification from the image recognition device 30 (if Yes in ACT75), in ACT76, the scanner 20 transmits a notification signal to the POS terminal 10 to notify the POS terminal 10 of an error that the image recognition device 30 cannot upload. When the POS terminal 10 receives the notification signal from the scanner 20, the light-emitting unit 571 is illuminated to notify the manager that uploading from the image recognition device 30 to the server 40 is not possible. In this case, for example, the manager of the store can know that uploading from the image recognition device 30 to the server 40 is not possible from the illumination of the light-emitting unit 571 of the POS terminal 10, and can promptly start work for recovery.

[0168] (Scanner and image recognition device operation) Next, with reference to Figs. 14 and 15, the operation of the scanner 20 and the image recognition device 30 for identifying the commodity code of the commodity from the commodity image, which is the captured image of the commodity captured by the scanner 20, for each commodity during the period between when the scanner 20 receives an imaging start signal accompanying the start of a transaction and an imaging stop signal accompanying the completion of registration from the POS terminal 10 will be described. Fig. 14 is a flowchart showing the flow of the operation of the scanner 20 and the image recognition device 30 of the embodiment. In Fig. 14, the operation of the scanner 20 is shown on the left side, and the operation of the image recognition device 30 is shown on the right side. Fig. 15 is a flowchart showing the flow of the operation of the scanner 20 of the embodiment. The operation of the scanner 20 is performed by each unit of the scanner 20 under the control of the control unit 200 executing the control program 211. The operation of the image recognition device 30 is performed by each unit of the image recognition device 30 under the control of the control unit 300 executing the control program 311.

[0169] In ACT10, the scanner 20 checks whether the upload flag set in the flag section 2031 of the RAM 203 is ON “1” or not (OFF “0”), i.e., whether the image recognition device 30 is currently executing a process to upload product images for additional learning (for status analysis) to the server 40.

[0170] When the scanner 20 confirms that the upload flag set in the flag unit 2031 is OFF "0", that is, the image recognition device 30 is not executing a process of uploading to the server 40 (in the case of No in ACT10), in ACT11, the scanner 20 acquires a product image, which is one captured image, from the moving image captured by the imaging unit 220. For example, the scanner 20 first stores all or some of the multiple frame images sequentially captured by the imaging unit 220 in the RAM 203 as candidates for the product image. Next, the scanner 20 selects one frame image from the multiple frame images held in the RAM 203, and sets the selected frame image as the product image. In addition, the scanner 20 assigns an identification code for identifying the product image to the product image. In the following, for convenience, the product image is assumed to include an identification code.

[0171] In ACT12, the scanner 20 transmits the product image to the image recognition device 30 via the LAN, for example, by TCP communication.

[0172] In ACT20, the image recognition device 30 checks whether the upload flag set in the flag section 3032 of the RAM 303 is ON “1” or not (OFF “0”), i.e., whether a process of uploading a product image for additional learning (for status analysis) to the server 40 is being executed.

[0173] When the image recognition device 30 confirms that the upload flag set in the flag unit 3032 is OFF "0", that is, the process of uploading to the server 40 is not being executed (No in ACT20), it executes the process related to image recognition from ACT21 onwards. Also, when the image recognition device 30 confirms that the upload flag set in the flag unit 3032 is ON "1", that is, the process of uploading to the server 40 is being executed (Yes in ACT20), it stops and does not execute the process related to image recognition from ACT21 onwards (stopping means).

[0174] In ACT20, when the image recognition device 30 confirms that the upload flag is OFF “0” (No in ACT20), in ACT21, the image recognition device 30 receives the product image and the identification code from the scanner 20 via the LAN. The image recognition device 30 stores the received product image in the product image section 3031 of the RAM 303.

[0175] In ACT22, the image recognition device 30 performs image recognition on the product image stored in the product image section 3031. For example, the image recognition device 30 performs image recognition based on features such as the contour and color of the product depicted in the product image. In this case, first, the image recognition device 30 acquires the features included in the product image. Next, the image recognition device 30 checks the acquired features against the matching dictionary 313 to search for features that are closest to the acquired features. Next, the image recognition device 30 acquires the product code associated with the search result (the feature that is closest to the acquired feature).

[0176] In ACT23, the image recognition device 30 determines whether the image recognition has been successful. For example, the image recognition device 30 first obtains a difference between the acquired feature amount and a feature amount closest to the acquired feature amount. Next, the image recognition device 30 compares the obtained difference with a predetermined amount.

[0177] If the comparison result shows that the difference is less than a predetermined amount, the image recognition device 30 determines that the image recognition has been successful. In this case, the image recognition device 30 stores the product code acquired in ACT22 in the RAM 303 as the image recognition result. Then, the operation proceeds to ACT24.

[0178] On the other hand, if the comparison result shows that the difference is equal to or greater than the predetermined amount, the image recognition device 30 determines that the image recognition has failed. In this case, the image recognition device 30 requests the scanner 20 to acquire and transmit another product image, and performs the operations of ACT21 to ACT23 again. The image recognition device 30 also determines that the product image that has undergone image recognition is a product image to be used for additional learning or condition analysis.

[0179] In ACT24, the image recognition device 30 transmits the image recognition result held in the RAM 303 to the scanner 20 via the LAN. The image recognition device 30 also transmits an identification code to the scanner 20.

[0180] In ACT13, the scanner 20 detects a product barcode from the product image. In other words, the scanner 20 determines whether or not a product barcode is captured in the product image.

[0181] If the product barcode is detected in ACT 13 (YES in ACT 13), the scanner 20 extracts and decodes the detected product barcode in ACT 14, and obtains the product code held by the product barcode as the decoding result. The scanner 20 stores the obtained product code in the RAM 203.

[0182] Next, in ACT15, the scanner 20 transmits the product code of the decoded result (the product code held by the product barcode) to the image recognition device 30. Since the operations of ACT14 and ACT15 are performed when the product barcode is detected in ACT13, the scanner 20 may not transmit the product code of the decoded result in ACT15.

[0183] In ACT16, the scanner 20 checks whether it has received the image recognition result of the product image (the product code of the product shown in the product image) from the image recognition device 30. If it has received the image recognition result (Yes in ACT16), the scanner 20 stores the received image recognition result in the RAM 203. The scanner 20 may identify the product image corresponding to the image recognition result based on the identification code received together with the image recognition result. Then, it proceeds to the merging operation in ACT17.

[0184] If the product barcode cannot be detected in ACT13 (No in ACT13), the operations in ACT14 and ACT15 are skipped and the process moves to the operation of confirming receipt of the image recognition result in ACT16.

[0185] If the image recognition result has not been received in ACT16 (No in ACT16), the scanner 20 repeats the operations of ACT11 and ACT13 to ACT15. At this time, since the image recognition device 30 does not require a new product image, the product image transmission in ACT12 is omitted. That is, in a state in which the image recognition device 30 is not performing a process of uploading to the server 40, the scanner 20 continues a series of regular operations such as obtaining a product image in ACT11, detecting a product barcode in ACT13, decoding in ACT14, and transmitting the product code of the decoded result in ACT15 until the image recognition result is received from the image recognition device 30.

[0186] In ACT17, the scanner 20 merges the decoded result (the product code held by the product barcode) and the image recognition result (the product code of the product shown in the product image). That is, the scanner 20 obtains one product code that is considered to be the most correct from the decoded result and the image recognition result.

[0187] If no decoded result is obtained, the product code from the image recognition result is treated as the merged result. Situations in which no decoded result is obtained include when the consumer does not hold the product barcode over the image capturing unit 220 of the scanner 20 correctly, or when the product barcode is unnaturally bent or folded.

[0188] When both the decoded result and the image recognition result are obtained, the scanner 20 compares the decoded result with the image recognition result. If the comparison shows that the decoded result and the image recognition result match, the scanner 20 merges the product code of the matching decoded result and image recognition result. If the comparison shows that the decoded result and the image recognition result do not match, the scanner 20 merges the product code of the more reliable decoded result.

[0189] In ACT 18, the scanner 20 transmits the merged product code (the product code that is considered to be the most correct) to the POS terminal 10.

[0190] In ACT16 to ACT18 of this embodiment, the scanner 20, after receiving the image recognition result from the image recognition device 30 (in the case of Yes in ACT16), uploads (transmits) to the upper device (here, the POS terminal 10) any product code obtained by merging the product code decoded from the product barcode, which is the scan result, and the product code, which is the image recognition result, into the upper device. However, the scanner 20 may first transmit the product code decoded from the product barcode, which is the scan result, to the upper device in a state where the image recognition result has not yet been received from the image recognition device 30, and then, when the image recognition result is received from the image recognition device 30, transmit to the upper device any product code obtained by merging the product code decoded from the product barcode, which is the scan result, and the product code, which is the image recognition result, into the upper device.

[0191] When the scanner 20 transmits the decoded product code (the product code held by the product barcode) in ACT 15, the image recognition device 30 receives the decoded product code in ACT 25. The image recognition device 30 stores the received decoded product code in the product image section 3031 of the RAM 303 in association with the product image that has been image-recognized this time (in ACT 22).

[0192] In ACT 26, the image recognition device 30 determines whether there is a product image to be used for additional learning or status analysis. For example, if there is a product image for which image recognition has failed in ACT 23 and a product code of the decoded result has been received from the scanner 20 in ACT 25, the image recognition device 30 determines that there is a product image to be used for additional learning or status analysis.

[0193] If it is determined in ACT26 that there is a product image to be used for additional learning or status analysis, in ACT27, the image recognition device 30 retains (stores) the product image that failed image recognition and the product code of the decoded result received from the scanner 20, which are stored in the product image section 3031 of the RAM 303.

[0194] If it is determined in ACT26 that there are no product images to be used for additional learning or state analysis, the operation in ACT27 is skipped and the process ends.

[0195] That is, when the image recognition device 30 is not executing a process of uploading to the server 40, the scanner 20 merges the product code decoded from the product barcode detected by the scanner 20 based on the product image captured by the scanner 20 with the product code acquired by the image recognition device 30 through image recognition. Then, the product code resulting from the merging by the scanner 20 is transmitted to the POS terminal 10, and the product is registered by the POS terminal 10.

[0196] The operator of the product information registration system RS, who is the customer, has more freedom in the orientation of the product held over the scanner 20, improving operability when registering products.

[0197] In ACT10, when the scanner 20 confirms that the upload flag set in the flag section 2031 of the RAM 203 is ON “1”, i.e., the image recognition device 30 is executing the process of uploading to the server 40 (if Yes in ACT10), in ACT31 of Figure 15, the scanner 20 acquires a product image, which is one captured image, from the moving image captured by the imaging section 220.

[0198] In ACT32, the scanner 20 detects a product barcode from the product image.

[0199] If the product barcode is detected in ACT 32 (YES in ACT 32), the scanner 20 extracts and decodes the detected product barcode in ACT 33, and obtains the product code held by the product barcode as the decoding result. The scanner 20 stores the obtained product code in the RAM 203.

[0200] In ACT 34, the scanner 20 transmits the product code stored in the RAM 203 as the result of decoding the product barcode to the POS terminal 10.

[0201] In other words, when the image recognition device 30 is performing a process of uploading to the server 40, the image recognition device 30 does not perform image recognition processing, and instead transmits the product code obtained by decoded the product barcode detected by the scanner 20 to the POS terminal 10, and the product is registered by the POS terminal 10.

[0202] In the product information registration system RS, products are identified by detecting product barcodes with a scanner 20 and obtaining product codes, without being affected by reduced processing speed or reduced recognition accuracy that would result from simultaneously performing the process of image recognition of product images and obtaining product codes in an image recognition device 30 and the process of uploading the product images and product codes to a server 40.

[0203] In the image recognition device 30, the increase in the load on each process due to simultaneously performing the process of recognizing product images and the process of uploading the product images to the server 40 becomes noticeable when the free space in the memory section used for each process, i.e., in this embodiment, the free space of RAM 303, becomes limited, resulting in effects such as a decrease in processing speed and a decrease in recognition accuracy.

[0204] The operator of the product information registration system RS, who is the customer, can continue product registration regardless of whether the image recognition device 30 is currently uploading product images for additional learning (for status analysis) to the server 40.

[0205] (Summary of the embodiment) According to the embodiment of the product information management system MS, when the process of uploading (transmitting) the product image recognized by the image recognition device 30 to the server 40 (external storage device) is not being performed, the scanner 20 merges the product code decoded from the product barcode detected by the scanner 20 from the product image with the product code acquired by the image recognition device 30 through image recognition, and transmits the product code identified as a result of the merge to the POS terminal 10 to register the product.

[0206] When uploading the product image recognized by the image recognition device 30 to the server 40, the process of recognizing the product image is stopped (stopping means), and the product code obtained by decoding the product barcode detected by the scanner 20 from the product image is transmitted to the POS terminal 10 to register the product.

[0207] This allows the customer, an operator of the product information registration system RS, to continue registering products without being affected by a decrease in processing speed or recognition accuracy, even when the image recognition device 30 is processing the uploading of product images to the server 40.

[0208] Therefore, it becomes possible to appropriately control the process relating to image recognition and the process of uploading the image used in the image recognition to the server.

[0209] (others) In the above embodiment, the control programs executed by the POS terminal 10, the scanner 20, the image recognition device 30, and the server 40 may be configured to be provided by recording them on a computer-readable recording medium such as a CD-ROM. Also, the control programs executed by the POS terminal 10, the scanner 20, the image recognition device 30, and the server 40 in the above embodiment may be configured to be stored on a computer connected to a network such as the Internet and provided by downloading them via the network, or may be configured to be provided via a network such as the Internet.

[0210] Although the embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0211] MS...Product information management system, RS...Product information registration system, 1...Self-service POS system, 10...POS terminal, 100...Control unit, 1001...Input means, 1002...Registration means, 1003...Accounting processing means, 1004...Output means, 101...CPU, 102...ROM, 103...RAM, 1031...Product information unit, 110...Memory unit, 111...Control program, 112...Product master, 120...Audio output unit, 121...Connection interface, 122...LAN interface, 123...Bus, 50...Main body, 51...Display, 52...Touch panel, 53...Printer, 531...issuing port, 54...card reader, 541...card insertion port, 55...change machine, 551...coin insertion port, 552...coin discharge port, 553...coin tray, 554...banknote insertion port, 555...banknote discharge port, 57...pole, 571...light emitting unit, 70...first basket placement stand, 71...first weighing unit, 80...second basket placement stand, 81...second weighing unit, 82...support rod, 83...temporary placement stand, 20...scanner, 200...control unit, 2001...acquisition means, 2002...decoding means, 2003...transmission / reception means, 2004...merging means, 2005...transmission means, 2006...flag setting means, 201...CPU, 202...ROM, 203...RAM, 2031...flag section, 210...storage section, 211...control program, 212...processing result, 220...imaging section, 221...connection interface, 222...LAN interface, 223...bus, 230...reading window, 30...image recognition device, 300...control section, 3001...receiving means, 3002...image recognition means, 3003...determination means, 3004...updating means, 3005...transmitting means, 3006...image uploading means, 301...CPU, 302...ROM, 303...RAM, 3031...product image section, 3032...flag section, 310...storage section, 311...control program, 312...trained model, 313...matching dictionary, 321...LAN interface, 322...bus, 40...server, 400...control unit, 4001...receiving means, 4002...additional learning means, 4003...master management means, 4004...transmitting means, 4005...image storage means, 401...CPU, 402...ROM, 403...RAM, 410...storage unit, 411...control program, 412...trained model, 413...matching dictionary, 414...product master, 415...product image unit, 421...communication unit, 422...bus, LH...LAN hub, NW...network,WR: wireless router.

Claims

1. An information processing device having a communication function with an external storage device, an image recognition means for performing image recognition on an image captured by the imaging unit; a transmission means for transmitting the captured image on which the image recognition means has performed image recognition to the external storage device; a stop means for stopping the image recognition performed by the image recognition means when the transmission means transmits the captured image to the external storage device; An information processing device comprising:

2. the transmission means includes means for transmitting an inquiry signal to the external storage device inquiring about permission for uploading the captured image to the external storage device when the captured image is to be transmitted to the external storage device; a receiving means for receiving an enable signal transmitted from the external storage device when the free space in the storage unit of the external storage device is equal to or greater than a predetermined value in response to the transmission of the inquiry signal, the transmitting means transmits the captured image to the external storage device when the receiving means receives the permission signal. The information processing device according to claim 1 .

3. a code acquisition means for acquiring a product code of the product based on code information attached to the product included in the captured image, the image recognition means acquires a product code of the product based on an appearance feature of the product included in the captured image; 3. The information processing device according to claim 1 or 2.

4. and a specifying means for specifying, when the transmission means is not transmitting the captured image to the external storage device, the product code acquired by the code acquisition means or the product code acquired by the image recognition means as the product code of the product. The information processing device according to claim 3 .

5. the transmission means transmits the captured image stored in the captured image section to the external storage device at predetermined time intervals or when the free space of a captured image section that stores the captured image to be recognized by the image recognition means becomes equal to or less than a predetermined space.

3. The information processing device according to claim 1 or 2.

6. A computer of an information processing device having a communication function with an external storage device, An image recognition means for performing image recognition on an image captured by the imaging unit; a transmission means for transmitting the captured image on which the image recognition means has performed image recognition to the external storage device; a stop means for stopping the image recognition performed by the image recognition means when the transmission means transmits the captured image to the external storage device; A program to function as a

Images