Information processing device and program

Abstract

To provide an information processing device capable of properly controlling the operation of a system when communication between at least one set of devices constituting the system is disconnected.SOLUTION: An information processing device having a communication function with an external device includes setting means and switching means. The setting means pre-sets the operation of the information processing device in the case that communication with the external device is disconnected to one of a plurality of operations. The switching means switches the operation of the information processing device to the operation set by the setting means when communication with the external device is disconnected.SELECTED DRAWING: Figure 16

Description

[Technical Field]

[0001] The present invention relates to an information processing device and a program. [Background technology]

[0002] POS (Point Of Sales) systems are used in stores that sell products and services (hereinafter collectively referred to as "products"). This type of POS system is equipped with a scanner that reads product codes from barcodes attached to products (hereinafter also referred to as "product barcodes"). Product barcodes include one-dimensional codes and two-dimensional codes. In one example, the product barcode is printed on a sticker and attached to the product. In another example, the product barcode is printed on the packaging that contains the product. The POS system registers information about the product that a customer intends to purchase based on the product code read by the scanner.

[0003] The operator of a POS system must find the product barcode on the product and hold it up to the scanner, making the operation of reading the product code cumbersome. As a result, the operator's task of registering product information (hereinafter also referred to as "product registration") is time-consuming. In particular, in so-called self-service POS systems, customers who are unfamiliar with product registration must scan product barcodes, so there is a need to improve the operability of reading product codes, in other words, the operator's operability for identifying products.

[0004] In such a POS system, it is conceivable that the multiple devices that make up the system are connected and communicate via a LAN (Local Area Network).If communication between devices connected via the LAN is interrupted, it is desirable to appropriately control the operation of the system. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-21168 [Patent Document 2] Japanese Patent Application Laid-Open No. 2003-317156 Summary of the Invention [Problem to be solved by the invention]

[0006] The problem to be solved by the present invention is to provide an information processing device and a program that can appropriately control the operation of a system when communication between at least one pair of devices that make up the system is cut off. [Means for solving the problem]

[0007] The information processing apparatus according to the embodiment includes: Image Recognition The information processing device has a function of communicating with a device, and includes a setting unit and a switching unit. Image Recognition If communication with the device is lost Should switch Operation of the information processing device mode of, Switchable Multiple actions mode Any of the following actions mode The switching means is configured to Image Recognition When communication with the device is cut off, the operation of the information processing device mode The operation set by the setting means mode It switches to. and a code transmitting means for transmitting the product code acquired by the image recognition device based on an appearance feature of the product included in the captured image to another external device. The setting means pre-sets an operation mode of the information processing device to be switched to when communication with the image recognition device is disconnected to one of a plurality of switchable operation modes in response to a user operation, and the plurality of operation modes includes at least an operation mode in which the code acquiring means continues to acquire the product code. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a product information management system according to an embodiment. [Figure 2] FIG. 2 is a perspective view showing the appearance of the self-service POS system according to the embodiment. [Figure 3] FIG. 3 is a block diagram illustrating a hardware configuration of the POS terminal according to the embodiment. [Figure 4]FIG. 4 is a block diagram showing the hardware configuration of the scanner according to the embodiment. [Figure 5] FIG. 5 is a block diagram illustrating a hardware configuration of the image recognition apparatus according to the embodiment. [Figure 6] FIG. 6 is a block diagram illustrating a hardware configuration of a server according to the embodiment. [Figure 7] FIG. 7 is a diagram showing a data configuration of a product master stored in the storage unit of the POS terminal of the embodiment. [Figure 8] FIG. 8 is a diagram showing the data structure of the setting information stored in the storage unit of the scanner according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating a data structure of a comparison dictionary stored in a storage unit of the image recognition device according to the embodiment. [Figure 10] FIG. 10 is a block diagram illustrating the functional configuration of the control unit of the POS terminal according to the embodiment. [Figure 11] FIG. 11 is a block diagram showing the functional configuration of the control unit of the scanner according to the embodiment. [Figure 12] FIG. 12 is a block diagram illustrating a functional configuration of a control unit of the image recognition device according to the embodiment. [Figure 13] FIG. 13 is a block diagram illustrating a functional configuration of a control unit of the server according to the embodiment. [Figure 14] FIG. 14 is a flowchart showing the flow of operations of the POS terminal and the scanner according to the embodiment. [Figure 15] FIG. 15 is a flowchart showing the flow of operations of the scanner and image recognition device according to the embodiment. [Figure 16] FIG. 16 is a flowchart showing the flow of operations of the scanner and the POS terminal according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described with reference to the drawings, but the present invention is not limited to the embodiments described below.

[0010] (Product Information Management System) First, a product information management system MS according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing the overall configuration of the product information management system MS according to an embodiment. The product information management system MS includes a product information registration system RS and a server 40.

[0011] The product information registration system RS includes a POS terminal 10, a scanner 20 (information processing device), and an image recognition device 30 (external device). The POS terminal 10 (another external device) and the scanner 20 (information processing device) are communicably connected via a LAN (Local Area Network) or a USB (Universal Serial Bus). The scanner 20 (information processing device) and the image recognition device 30 (external device) are communicably connected via a LAN.

[0012] In the embodiment, a combination of Ethernet (registered trademark) and TCP (Transmission Control Protocol) is used for LAN communication, but the communication protocol is not limited.

[0013] The POS terminal 10 and the image recognition device 30 are communicably connected to a LAN hub LH via a LAN. The LAN hub LH is communicably connected to a wireless router WR via the LAN. The wireless router WR can wirelessly communicate with a server 40 via a network NW.

[0014] The scanner 20 captures an image of a product to acquire the product image. The scanner 20 transmits the acquired product image to the image recognition device 30 via a LAN. The scanner 20 also detects and decodes product barcodes (one-dimensional codes and two-dimensional codes) included in the acquired product image. That is, the scanner 20 reads the information held in the product barcode. The decoded product barcode results include the product code held in the product barcode. The scanner 20 transmits the decoded results to the image recognition device 30 via a LAN.

[0015] The image recognition device 30 receives product images from the scanner 20 via a LAN. The image recognition device 30 has an image recognition function that uses, for example, a trained model based on machine learning, and performs image recognition on the product images using the image recognition function. In more detail, the image recognition device 30 acquires features contained in the product images using the image recognition function, compares the acquired features with a matching dictionary that associates multiple features with product codes, and acquires the product code associated with the feature that is closest to the acquired feature. In other words, the image recognition result includes the product code of the product depicted in the product image. The image recognition device 30 may also perform image recognition on the product images using techniques such as deep learning. The image recognition device 30 transmits the image recognition result to the scanner 20 via a LAN.

[0016] The scanner 20 receives the image recognition results from the image recognition device 30 via a LAN. The scanner 20 merges the decoded results and the image recognition results. Here, merging means obtaining a single product code from the decoded results and the image recognition results. Therefore, the merged result includes the product barcode contained in the product image or the product code of the product shown in the product image. The scanner 20 transmits the merged result to the POS terminal 10 via a LAN or a USB. The POS terminal 10 registers the product based on the product code of the merged result received from the scanner 20.

[0017] Furthermore, the image recognition device 30 determines whether to use the product image received from the scanner 20 for additional learning. For example, if the difference between the acquired feature amount and the feature amount closest to the acquired feature amount is equal to or greater than a predetermined amount, the image recognition device 30 determines that the product image received from the scanner 20 is a product image to be used for additional learning. The image recognition device 30 transmits the product image determined to be used for additional learning and the product code of the decoded result received from the scanner 20 to the server 40 via LAN.

[0018] The server 40 receives product images and product codes from the image recognition device 30 via the LAN. The server 40 stores the received product images and product codes in association with each other. The server 40 performs additional training of a trained model for product image recognition using the stored product images and product codes. The server 40 updates the trained model through the additional training. The server 40 also updates the matching dictionary in conjunction with the additional training. Updating the matching dictionary includes modifying existing records and adding new records. The server 40 transmits the updated trained model and updated matching dictionary to the image recognition device 30 via the LAN.

[0019] The image recognition device 30 receives the updated trained model and the updated matching dictionary from the server 40 via a LAN. The image recognition device 30 updates the currently stored trained model with the received updated trained model. The image recognition device 30 also updates the currently stored matching dictionary with the received updated matching dictionary.

[0020] In this product information management system MS, if the product packaging changes, it is possible to eliminate the need to collect new product images for additional training of the trained model for product image recognition.

[0021] The server 40 also manages the product master data. The POS terminal 10 transmits transaction processing information (information related to the processed transaction) via the LAN to the wireless router WR via the LAN hub LH. The wireless router WR then transmits the received transaction processing information to the server 40 via the network NW. The server 40 manages the product master data based on the received transaction processing information. For example, the server 40 receives transaction processing information from multiple POS terminals 10 and manages the product master data for each of the multiple POS terminals 10. Managing the product master data includes modifying existing records and adding new records. The server 40 also transmits the product master data appropriate for each POS terminal 10 to that POS terminal 10 via the network NW, wireless router WR, and LAN hub LH.

[0022] The image recognition device 30 may transmit the product images received from the scanner 20 to the server 40 regardless of whether they are used for additional learning. In this case, the product images stored in the server 40 may be used, for example, to analyze the state (event) of image recognition in the image recognition device 30.

[0023] (Product Information Registration System) As an example of a product information registration system, a self-service POS system operated by a customer who purchases a product will be described below. However, the product information registration system is not limited to a self-service POS system. The product information registration system may also be a POS system operated by a store clerk who sells the product.

[0024] 2 is a perspective view showing the appearance of the self-service POS system 1. The self-service POS system 1 has a POS terminal 10, a scanner 20 built into the POS terminal 10, and an image recognition device 30 built into the POS terminal 10. Here, the self-service POS system 1 is an example of a product information registration system, the POS terminal 10 is an example of a registration device, and the scanner 20 and the image recognition device 30 are examples of an information processing device.

[0025] The scanner 20 identifies the product to be purchased by the customer and transmits the product code of the identified product to the POS terminal 10. The product code is an example of product identification information that identifies the product. The POS terminal 10 registers the product based on the product code input from the scanner 20 and also performs a transaction process for the registered product. The transaction process is a process that the POS terminal 10 performs to allow the customer to pay for the product.

[0026] 2, the POS terminal 10 includes a main body 50, a first basket stand 70, and a second basket stand 80. The main body 50 includes a display 51 and a touch panel 52 on the top, and houses a printer 53, a card reader 54, and a change dispenser 55 (see FIG. 3 for each).

[0027] The display 51 is a display device such as an LCD (Liquid Crystal Display) and displays various information. For example, the display 51 displays product information such as the product name and price of a product registered by a customer's registration operation. The display 51 also displays accounting information such as the total transaction amount and change amount in accordance with the customer's accounting operation. The display 51 also displays, for example, a setting screen for setting various operating modes within the product information registration system RS in response to operations by an administrator of the system RS.

[0028] The touch panel 52 is provided on the surface of the display 51, and inputs information according to the touched position into the control unit 100 (see FIG. 3) of the POS terminal 10. A customer or an administrator (either of whom is an operator as a user) using the POS terminal 10 can operate the touch panel 52 by touching various buttons and operation keys displayed on the display 51.

[0029] As shown in Figure 2, below the display 51, facing the direction where the customer is located, there are formed a reading window 230 for the scanner 20, an issuing port 531 for issuing receipts printed by the printer 53, and a card insertion port 541 for the card reader 54.

[0030] The reading window 230 is a window through which the scanner 20 captures an image of the exterior of a product. A customer, who is an operator operating the self-service POS system 1, performs an operation related to product registration by holding the product to be purchased over the reading window 230. The customer holds the product over the reading window 230 while changing the orientation of the product relative to the reading window 230 until the self-service POS system 1 executes product registration and generates a registration completion sound.

[0031] Issuance port 531 is an opening through which receipts printed by printer 53 are ejected. Information (transaction processing information) related to transactions processed by POS terminal 10 is printed on the receipt. Note that POS terminal 10 may be configured to issue electronic receipts instead of paper receipts printed by printer 53.

[0032] Card insertion slot 541 is an opening for inserting a credit card or the like to be read by card reader 54. A card is inserted through card insertion slot 541 when a customer makes a payment by credit card or when a customer has a store membership card or the like read.

[0033] In addition, the main body 50 is formed with a coin insertion slot 551, a coin ejection slot 552, a coin receiving tray 553, a bill insertion slot 554, and a bill ejection slot 555 corresponding to the position of the change dispenser 55 housed inside.

[0034] The coin slot 551 is an opening through which customers insert coins to be paid. Coins inserted into the coin slot 551 are stored in the change machine 55. The coin discharge slot 552 is an opening through which coins to be dispensed from the change machine 55 are discharged. The coin discharge slot 552 discharges change or coins inserted into the coin slot 551 but not accepted by the change machine 55. The coin tray 553 is a tray that receives coins discharged from the coin discharge slot 552. The banknote slot 554 is an opening through which customers insert banknotes to be paid. Banknotes inserted into the banknote slot 554 are stored in the change machine 55. The banknote discharge slot 555 is an opening through which banknotes that are change to be dispensed from the change machine 55 are discharged.

[0035] The first basket stand 70 is a stand on which a shopping basket (not shown) containing items that a customer wishes to purchase is placed. The first basket stand 70 is equipped with a first weighing unit 71. The first weighing unit 71 weighs the total weight of the shopping basket placed on the first basket stand 70 and the items contained in the shopping basket.

[0036] The second basket stand 80 is a stand for placing shopping baskets (not shown) containing products that have been removed from the shopping basket placed on the first basket stand 70 and whose product codes have been read, i.e., registered products. The second basket stand 80 includes a second weighing unit 81, a pair of support rods 82, and a temporary stand 83.

[0037] The second weighing unit 81 weighs the total weight of a shopping basket placed on the second basket placement table 80 and the items contained in the shopping basket, or the total weight of a shopping bag (not shown) supported by a pair of support rods 82 and the items contained in the shopping bag. The support rods 82 are rods for hooking and supporting a shopping bag containing registered items. The support rods 82 support the shopping bag in an open state, making it easier to place items into the shopping bag. The temporary placement table 83 is a table for temporarily placing registered items that are easily broken or deformed, such as tofu or eggs. The customer can then place the items placed on the temporary placement table 83 into a shopping basket or shopping bag.

[0038] The customer registers the product by holding the product taken out of the shopping basket placed on the first basket placement stand 70 over the reading window 230. The customer then places the registered product into a shopping basket placed on the second basket placement stand 80 or a shopping bag supported by the support rod 82.

[0039] The control unit 100 (see FIG. 3) recognizes the weight change when the first weighing unit 71 measures the weight before and after an item is removed from a shopping basket placed on the first basket placing table 70. The control unit 100 recognizes the weight change when the second weighing unit 81 measures the weight before and after an item is placed in a shopping basket placed on the second basket placing table 80 or a shopping bag supported by the support rod 82. Depending on whether these two weight changes are equal, the control unit 100 can determine whether all of the items the customer wishes to purchase have been registered.

[0040] The main body 50 also has a cylindrical pole 57 extending upward. The pole 57 has a light-emitting section 571 at its upper end that emits light when calling a store clerk (also referred to as the manager of the POS terminal 10).

[0041] (POS terminal hardware configuration) Next, the hardware configuration of the POS terminal 10 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the hardware configuration of the POS terminal 10.

[0042] The POS terminal 10 includes a control unit 100, a memory unit 110, a display 51, a touch panel 52, a printer 53, a card reader 54, a change dispenser 55, a first weighing unit 71, a second weighing unit 81, a light-emitting unit 571, an audio output unit 120, a connection interface 121, and a LAN interface 122. The control unit 100, the memory unit 110, the display 51, the touch panel 52, the printer 53, the card reader 54, the change dispenser 55, the first weighing unit 71, the second weighing unit 81, the light-emitting unit 571, the audio output unit 120, the connection interface 121, and the LAN interface 122 are connected to one another via a bus 123 or the like.

[0043] The control unit 100 is configured as a computer including a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103. The CPU 101, the ROM 102, and the RAM 103 are connected to each other via a bus 123.

[0044] The CPU 101 controls the entire POS terminal 10. The ROM 102 stores various programs, such as a program used to drive the CPU 101, and various data. The RAM 103 is used as a work area for the CPU 101, and loads various programs and data stored in the ROM 102 and the storage unit 110. The control unit 100 executes various control processes of the POS terminal 10, for example, by the CPU 101 operating in accordance with a control program 111 stored in the storage unit 110 and loaded in the RAM 103.

[0045] RAM 103 has a product information section 1031. Product information section 1031 stores product information read from product master 112 (described later) based on the product code acquired from scanner 20. A product is registered by storing (registering) product information in product information section 1031. Product information stored in product information section 1031 includes the product code, product name, price, etc. Product information section 1031 stores product information related to one transaction, and is cleared when the transaction transaction processing is completed.

[0046] The storage unit 110 is configured with a storage medium such as a hard disk drive (HDD) or flash memory, and maintains its stored contents even when power is cut off. The storage unit 110 stores a control program 111 and a product master 112.

[0047] The control program 111 is a program for realizing functions such as receiving a product code from the scanner 20, registering a product based on the received product code, processing the transaction for the registered product, and setting various operation modes within the product information registration system RS in response to operations by an administrator. The control program 111 may be stored in the ROM 102 instead of in the memory unit 110.

[0048] The product master 112 is a master file that stores product information about products handled by the store. Because the products handled by the store change daily, the product master 112 is updated as appropriate by a store server (not shown) connected via the LAN interface 122. Figure 7 is a diagram showing the data structure of the product master 112. The product master 112 includes product codes, product categories, product names, and prices associated with the product codes.

[0049] The product code is information that identifies a product. The product classification is information that classifies the products sold in the store. The product name is information that indicates the name of the product. The price is information that indicates the price (unit price) of the product. Note that the product master 112 may also include other information in association with the product code.

[0050] 3, we will continue to explain the hardware configuration of the POS terminal 10. The configurations and functions of the display 51 and the touch panel 52 are as described above.

[0051] The printer 53 is configured by, for example, a thermal printer. The printer 53 prints information about the transaction on a roll of receipt paper and issues the paper as a receipt from the issuing port 531.

[0052] Card reader 54 reads credit information required for payment from a credit card inserted through card insertion slot 541. Card reader 54 also reads member information that identifies the member from a medium such as a membership card or point card inserted through card insertion slot 541. Card reader 54 may be one that magnetically reads information from a card, or may be an IC card reader that reads information from an IC card with a built-in IC (Integrated Circuit) chip.

[0053] When a customer pays for a product in cash, the change dispenser 55 accepts the bills and coins paid by the customer and dispenses change as needed. Bills are inserted through a bill insertion slot 554, and coins are inserted through a coin insertion slot 551. Furthermore, bills as change are ejected from a bill ejection slot 555, and coins as change are ejected from a coin ejection slot 552.

[0054] The configurations and functions of the first weighing unit 71, the second weighing unit 81, and the light-emitting unit 571 are as described above. The audio output unit 120 generates sound as needed to notify the customer of information. For example, the audio output unit 120 outputs an alarm sound to notify the customer that product registration has been completed. The audio output unit is an example of an alarm unit.

[0055] Connection interface 121 is an interface for communicating with scanner 20 via a LAN or a USB. Control unit 100 transmits and receives information (data) to and from scanner 20 via connection interface 121 via a LAN or a USB. For example, control unit 100 receives a product code from scanner 20 via connection interface 121 via a LAN or a USB. Also, for example, control unit 100 transmits data on an operating mode selected in response to an administrator's operation to scanner 20 via connection interface 121 via a LAN or a USB.

[0056] The LAN interface 122 is an interface for communicating with the LAN hub LH via the LAN. The control unit 100 transmits and receives information (data) to and from the LAN hub LH via the LAN interface 122. For example, the control unit 100 transmits and receives the product master 112 to and from the server 40 via the LAN, via the LAN hub LH and the wireless router WR.

[0057] The LAN interface 122 is also an interface for communicating with external devices such as a store server (not shown) managed by the store. The control unit 100 transmits and receives information to and from external devices via the LAN interface 122 and the LAN.

[0058] (Scanner hardware configuration) Next, the hardware configuration of the scanner 20 will be described with reference to Fig. 4. Fig. 4 is a diagram showing the hardware configuration of the scanner 20.

[0059] The scanner 20 captures an image of a product purchased by a customer to obtain the product image. The scanner 20 then detects a product barcode included in the product image, i.e., a barcode attached to the captured product, and decodes the detected product barcode. The decoded product barcode contains the product code stored in the product barcode. The scanner 20 then transmits the product image to the image recognition device 30 via a LAN. If communication with the image recognition device 30, which is connected via a LAN, is interrupted, the scanner 20 operates according to an operation mode preselected in response to an administrator's operation of the POS terminal 10.

[0060] The scanner 20 includes a control unit 200, a storage unit 210, an imaging unit 220, a connection interface 221, and a LAN interface 222. The control unit 200, the storage unit 210, the imaging unit 220, the connection interface 221, and the LAN interface 222 are connected to one another via a bus 223 or the like.

[0061] The control unit 200 is configured as a computer including a CPU 201, a ROM 202, and a RAM 203. The CPU 201, the ROM 202, and the RAM 203 are connected to one another via a bus 223.

[0062] The CPU 201 controls the entire scanner 20. The ROM 202 stores various programs, such as a program used to drive the CPU 201, and various data. The RAM 203 is used as a work area for the CPU 201, and loads various programs and data stored in the ROM 202 and the storage unit 210. The control unit 200 executes various control processes for the scanner 20, for example, by the CPU 201 operating in accordance with a control program 211 stored in the storage unit 210 and loaded in the RAM 203.

[0063] The storage unit 210 is configured with a storage medium such as a HDD or flash memory, and maintains its stored contents even when the power is turned off. The storage unit 210 stores a control program 211, processing results 212, and setting information 213.

[0064] The control program 211 is a program for realizing functions such as capturing an image of a commodity using the imaging unit 220 and saving it in the RAM 203, acquiring a frame image (commodity image) stored in the RAM 203, detecting a commodity barcode included in the commodity image, decoding the commodity barcode, transmitting the commodity image to the image recognition device 30, receiving an image recognition result from the image recognition device 30, merging the decoded result with the image recognition result, transmitting the commodity code of the merged result to the POS terminal 10, and switching to a pre-selected operation mode when communication with the image recognition device 30 is disconnected. The control program 211 may be stored in the ROM 202 instead of in the storage unit 210.

[0065] The processing result 212 includes a decoding result obtained by decoding the product barcode, an image recognition result obtained by the image recognition device 30, and a merged result obtained by merging the decoding result and the image recognition result. The merged result includes the product barcode included in the product image or the product code of the product shown in the product image.

[0066] The setting information 213 is, for example, information for setting an operation mode to be switched to when communication with the image recognition device 30 is disconnected, which has been selected in advance in response to an administrator's operation on the POS terminal 10. FIG. 8 is a diagram showing the data configuration of the setting information 213. The setting information 213 includes information indicating a plurality of operation modes to which the POS terminal 10 can be switched when communication with the image recognition device 30 is disconnected (here, operation mode "1" and operation mode "2"), and corresponding switching information (e.g., a flag) for setting which of the plurality of operation modes is selected as the operation mode to be switched to. Here, the operation mode corresponding to the switching information "1" is set as the operation mode to be switched to. The number of switchable operation modes is not limited to two.

[0067] The imaging unit 220 is configured with, for example, a color CCD image sensor or a color CMOS image sensor. The imaging unit 220, under the control of the control unit 200, captures an image of a product through the reading window 230 and outputs the acquired product image. For example, the imaging unit 220 captures moving images at a predetermined frame rate (e.g., 30 fps). The imaging unit 220 stores the captured frame images in RAM 203. The frame images (product images) stored in RAM 203 are used to identify the product code of the product shown in the frame image.

[0068] The connection interface 221 is an interface for communicating with the POS terminal 10 via a LAN or a USB. The control unit 200 transmits and receives information (data) to and from the POS terminal 10 via the connection interface 221, the LAN, or the USB.

[0069] The LAN interface 222 is an interface for communicating with the image recognition device 30 via a LAN. The control unit 200 transmits and receives information (data) to and from the image recognition device 30 via the LAN interface 222 and the LAN.

[0070] The control unit 200 detects a product barcode included in a frame image (product image) stored in the RAM 203, decodes the product barcode, and acquires the product code stored in the product barcode. The control unit 200 also transmits the frame image (product image) stored in the RAM 203 to the image recognition device 30 via the LAN interface 222, and receives the image recognition result (product code) by the image recognition device 30 from the image recognition device 30. The control unit 200 then merges the decoded result with the image recognition result to identify the product barcode included in the product image or the product code of the product shown in the product image, and transmits the identified product code to the POS terminal 10 via the connection interface 221. The control unit 200 also switches to the operation mode set in the setting information 213 when communication with the image recognition device 30 via the LAN interface 222 is disconnected.

[0071] (Hardware configuration of image recognition device) Next, the hardware configuration of the image recognition device 30 will be described with reference to Fig. 5. Fig. 5 is a diagram showing the hardware configuration of the image recognition device 30.

[0072] The image recognition device 30 receives product images from the scanner 20 via a LAN. The image recognition device 30 performs image recognition on the received product images. That is, the image recognition device 30 acquires the product code of the product depicted in the product image based on the feature quantities of the product image. The image recognition device 30 transmits the image recognition results to the scanner 20 via a LAN. Furthermore, the image recognition device 30 transmits the product image or the product image and the product code to the server 40 via a LAN, as necessary. Furthermore, the image recognition device 30 receives the updated trained model and matching dictionary from the server 40 via a LAN, and updates the currently stored trained model and matching dictionary.

[0073] The image recognition device 30 includes a control unit 300, a storage unit 310, and a LAN interface 321. The control unit 300, the storage unit 310, and the LAN interface 321 are connected to one another via a bus 322 or the like.

[0074] The control unit 300 is configured as a computer including a CPU 301, a ROM 302, and a RAM 303. The CPU 301, the ROM 302, and the RAM 303 are connected to one another via a bus 322.

[0075] The CPU 301 controls the entire image recognition device 30. The ROM 302 stores various programs, such as a program used to drive the CPU 301, and various data. The RAM 303 is used as a work area for the CPU 301, and loads various programs and data stored in the ROM 302 and the storage unit 310. The control unit 300 executes various control processes of the image recognition device 30, for example, by the CPU 301 operating in accordance with a control program 311 stored in the storage unit 310 and loaded in the RAM 303.

[0076] The storage unit 310 is configured with a storage medium such as a HDD or flash memory, and maintains its stored contents even when power is cut off. The storage unit 310 stores a control program 311, a trained model 312, and a matching dictionary 313.

[0077] The control program 311 is a program for realizing functions such as receiving product images from the scanner 20, performing image recognition on the product images, determining whether the image recognition was successful, transmitting the image recognition results to the scanner 20, receiving the decoded product code from the scanner 20, determining whether the product image will be used for additional learning, transmitting the product image or the product image and the product code to the server 40, receiving an updated trained model and an updated matching dictionary from the server 40, updating the currently held trained model with the updated trained model, and updating the currently held matching dictionary with the updated matching dictionary. The control program 311 may be stored in the ROM 302 instead of in the memory unit 310.

[0078] The trained model 312 is an image recognition model for performing image recognition on product images. The trained model 312 has been trained by machine learning and is additionally trained by machine learning.

[0079] The matching dictionary 313 is a data table that manages products in association with their appearance features. FIG. 9 is a diagram showing the data structure of the matching dictionary 313. The matching dictionary 313 includes product codes and feature values ​​associated with the product codes. The matching dictionary 313 may have multiple feature values ​​for one product code. The product codes in the matching dictionary 313 are the same as the product codes in the product master 112 stored in the memory unit 110 of the POS terminal 10.

[0080] As described above, the product code is information for identifying a product. The feature amounts are, for example, parameterized features of the product's appearance, such as the product's standard shape, surface color, pattern, and unevenness. The control unit 300 acquires the feature amounts contained in the product image, compares the acquired feature amounts with the matching dictionary, and acquires the product code associated with the acquired feature amounts. In other words, the control unit 300 acquires the product code of the product shown in the product image from among the products pre-recorded in the matching dictionary 313, based on the feature amounts acquired from the product image.

[0081] Returning to FIG. 5, the description of the hardware configuration of the image recognition device 30 will be continued.

[0082] The LAN interface 321 is an interface for communicating with the scanner 20 and the LAN hub LH. The control unit 300 transmits and receives information (data) to and from the scanner 20 and the LAN hub LH via the LAN interface 321.

[0083] For example, the control unit 300 receives a frame image (product image) from the scanner 20 via the LAN interface 321. Furthermore, the control unit 300 acquires the product code of the product shown in the product image as described above, and transmits the acquired product code to the scanner 20 via the LAN interface 321.

[0084] Furthermore, the control unit 300 transmits, as necessary, the product image that has been subjected to image recognition, or the product image that has been subjected to image recognition and the product code that is the decoded result received from the scanner 20, to the server 40 via the LAN interface 321. The control unit 300 also receives the updated trained model 412 (see FIG. 6) and the updated matching dictionary 413 (see FIG. 6) from the server 40 via the LAN interface 321. The control unit 300 updates the currently held trained model 312 with the updated trained model 412. The control unit 300 also updates the currently held matching dictionary 313 with the updated matching dictionary 413.

[0085] (Server hardware configuration) Next, the hardware configuration of the server 40 will be described with reference to Fig. 6. Fig. 6 is a diagram showing the hardware configuration of the server 40.

[0086] The server 40 receives and stores product images and product codes from the image recognition device 30. The server 40 uses the stored product images and product codes to perform additional training on a trained model for product image recognition and updates the trained model and matching dictionary. The server 40 sends the updated trained model and updated matching dictionary to the image recognition device 30. The server 40 also manages product master data. The server 40 receives transaction processing information from the POS terminal 10 and manages the product master data based on the received transaction processing information. The server 40 sends product master data appropriate for each POS terminal 10 to that POS terminal 10.

[0087] The server 40 includes a control unit 400, a storage unit 410, and a communication unit 421. The control unit 400, the storage unit 410, and the communication unit 421 are connected to one another via a bus 422 or the like.

[0088] The control unit 400 is configured as a computer including a CPU 401, a ROM 402, and a RAM 403. The CPU 401, the ROM 402, and the RAM 403 are connected to one another via a bus 422.

[0089] The CPU 401 controls the entire server 40. The ROM 402 stores various programs, such as a program used to drive the CPU 401, and various data. The RAM 403 is used as a work area for the CPU 401, and loads various programs and data stored in the ROM 402 and the storage unit 410. The control unit 400 executes various control processes of the server 40, for example, by the CPU 401 operating in accordance with a control program 411 stored in the storage unit 410 and loaded in the RAM 403.

[0090] The storage unit 410 is configured with a storage medium such as a HDD or flash memory, and maintains its stored contents even when the power is turned off. The storage unit 410 stores a control program 411, a trained model 412, a matching dictionary 413, and a product master 414.

[0091] The control program 411 is a program for realizing functions such as receiving product images and product codes from the image recognition device 30, storing product images and product codes, additionally learning and updating the trained model 412, updating the matching dictionary 413, sending the updated trained model 412 and the updated matching dictionary 413 to the image recognition device 30, receiving transaction processing information from the POS terminal 10, managing the product master 414, sending the product master 414 to the POS terminal 10, and sending stored product images to an external device as needed. The control program 411 may be stored in the ROM 402 instead of in the storage unit 410.

[0092] The trained model 412 is a model trained by machine learning for performing image recognition on product images. In detail, the trained model 412 is an image recognition model that acquires feature amounts contained in the product images and compares the acquired feature amounts with a matching dictionary 413 to acquire the product codes of the products shown in the product images.

[0093] Similar to the matching dictionary 313 held by the image recognition device 30, the matching dictionary 413 is a data table that manages product codes and appearance feature quantities of the products in association with each other. The matching dictionary 413 includes the matching dictionaries 313 held by multiple image recognition devices 30. The server 40 transmits to each image recognition device 30 the matching dictionary 313 suitable for that image recognition device 30.

[0094] Like the product master data 112 held by the POS terminal 10, the product master data 414 includes product codes, and product categories, product names, and prices associated with the product codes. The product master data 414 includes the product master data 112 held by multiple POS terminals 10. The server 40 manages the product master data 414 based on transaction processing information received from the POS terminal 10. For example, the server 40 updates the product master data 414 based on the transaction processing information. The server 40 sends each POS terminal 10 a product master data 112 appropriate for that POS terminal 10.

[0095] The communication unit 421 is an interface for communicating with the wireless router WR. The control unit 400 transmits and receives information (data) to and from the wireless router WR via the communication unit 421.

[0096] For example, the control unit 400 receives and stores product images and image recognition results from the image recognition device 30 via the communication unit 421. The control unit 400 performs additional training on the trained model 412 using the stored product images and image recognition results, and updates the trained model 412 and the matching dictionary 413. The control unit 400 transmits the updated trained model and updated matching dictionary to the image recognition device 30 via the communication unit 421. The control unit 400 transmits the stored product images to an external device as needed.

[0097] The server 40 also receives transaction processing information from the POS terminal 10 via the communication unit 421. The server 40 manages the product master based on the received transaction processing information. The server 40 then sends the product master appropriate for each POS terminal 10 via the communication unit 421 to that POS terminal 10.

[0098] (POS terminal functional configuration) Next, the functional configuration of the POS terminal 10 will be described with reference to Fig. 10. Fig. 10 is a block diagram showing the functional configuration of the control unit 100 of the POS terminal 10.

[0099] Control unit 100 has input means 1001, registration means 1002, accounting processing means 1003, output means 1004, and setting means 1005. Control unit 100 functions as input means 1001, registration means 1002, accounting processing means 1003, output means 1004, and setting means 1005 when CPU 101 operates in accordance with a control program stored in ROM 102 or storage unit 110. Note that each of these functions may be configured using hardware such as a dedicated circuit.

[0100] Various information is input to the input means 1001 from the scanner 20, the touch panel 52, the card reader 54, the first weighing unit 71, the second weighing unit 81, etc. For example, a product code is input to the input means 1001 from the scanner 20.

[0101] Furthermore, information based on the operation of the touch panel 52 is input to the input means 1001. For example, transaction start information declaring the start of one transaction is input to the input means 1001. Furthermore, registration completion information indicating that product registration of the product related to one transaction has been completed is input to the input means 1001.

[0102] Furthermore, the input means 1001 receives a disconnection notification signal transmitted from the scanner 20 when communication between the scanner 20 and the image recognition device 30 is disconnected, and notifies the user of the disconnection. The input means 1001 also receives selection information for an operation mode to which the operation of the scanner 20 should be switched when communication between the scanner 20 and the image recognition device 30 is disconnected.

[0103] The registration means 1002 executes product registration of a product that the customer wishes to purchase through an operation by the customer. In other words, the registration means 1002 registers information about the product identified by the product identification information input to the input means 1001. Specifically, the registration means 1002 reads product information from the product master 112 based on the product code input to the input means 1001. Then, the registration means 1002 stores the read product information in the product information section 1031. The product information is information including the product code, product name, price, etc.

[0104] The transaction processing means 1003 executes transaction processing related to payment for the products registered by the registration means 1002. Specifically, the transaction processing means 1003 executes processing to calculate the total price (transaction amount) of the registered products, calculate the amount of change, display various information, issue receipts, etc. If the customer desires cashless payment such as credit card payment or two-dimensional code payment, the transaction processing means 1003 can execute the transaction processing necessary for payment.

[0105] The output means 1004 outputs various information to the scanner 20, the printer 53, the change dispenser 55, the light emitting unit 571, the audio output unit 120, etc. For example, when transaction start information is input to the input means 1001, the output means 1004 outputs an imaging start signal that instructs the scanner 20 to start imaging. Also, when registration completion information is input to the input means 1001, the output means 1004 outputs an imaging stop signal that instructs the scanner 20 to stop imaging.

[0106] When the registration means 1002 completes the product registration for one product, the output means 1004 outputs a control signal to the audio output unit 120 to output an alert sound. The output means 1004 functions as an alert means for causing the alert unit (audio output unit 120) to notify that the registration means 1002 has completed the registration of product information. The audio output unit 120 may be provided in the scanner 20. In this case, the audio output unit 120 may be configured to emit an alert sound when the scanner 20 identifies the product code.

[0107] Furthermore, when the output means 1004 receives a disconnection notification signal transmitted from the scanner 20 via the input means 1001 to notify of the disconnection, the output means 1004 outputs a control signal for emitting light from the light emitting unit 571. The output means 1004 also functions as a notification means for causing the notification unit (light emitting unit 571) to notify that communication between the scanner 20 and the image recognition device 30 has been disconnected.

[0108] Furthermore, when the selection information of the operation mode is input to the input means 1001, the output means 1004 transmits the information of the selected operation mode to the scanner 20.

[0109] For example, in response to an administrator's operation on the touch panel 52, the setting means 1005 displays on the display 51 a setting screen (not shown) for setting the operation mode of the scanner 20 when communication between the scanner 20 and the image recognition device 30 is disconnected. The setting means 1005 transmits selection information of the operation mode to which the operation of the scanner 20 should be switched, which information is input by the input means 1001 based on the setting screen, to the scanner 20 via the output means 1004 as information of the selected operation mode.

[0110] (Scanner functional configuration) Next, the functional configuration of the scanner 20 will be described with reference to Fig. 11. Fig. 11 is a block diagram showing the functional configuration of the control unit 200 of the scanner 20.

[0111] The control unit 200 has an acquiring means 2001, a decoding means 2002, a transmitting / receiving means 2003, a merging means 2004, a transmitting means 2005, a setting means 2006, and a switching means 2007. The control unit 200 functions as the acquiring means 2001, the decoding means 2002, the transmitting / receiving means 2003, the merging means 2004, the transmitting means 2005, the setting means 2006, and the switching means 2007 when the CPU 201 operates in accordance with a control program stored in the ROM 202 or the storage unit 210. Note that each of these functions may be configured by hardware such as a dedicated circuit.

[0112] The acquisition means 2001 acquires one frame image as a product image from a moving image captured by the imaging unit 220. The acquisition means 2001 stores the acquired product image in the RAM 203. For example, the acquisition means 2001 acquires all or some of the multiple frame images sequentially captured by the imaging unit 220 as product image candidates, stores all or some of the acquired frame images in the RAM 203, and selects a product image from all or some of the stored frame images.

[0113] The acquiring unit 2001 also acquires various other information. For example, the acquiring unit 2001 acquires an imaging start signal and an imaging stop signal from the POS terminal 10. The acquiring unit 2001 acquires information on the selected operation mode to which the operation of the scanner 20 should be switched from the POS terminal 10.

[0114] The decoding means 2002 detects the product barcode included in the product image acquired by the acquisition means 2001 and extracts the product barcode from the product image. The decoding means 2002 decodes the extracted product barcode. The decoding means 2002 acquires the product code held by the product barcode as a decoding result of the decoding of the product barcode.

[0115] The transmitting / receiving means 2003 transmits the product image acquired by the acquiring means 2001 to the image recognition device 30. The image recognition device 30 performs image recognition on the received product image and transmits the image recognition result (product code of the product shown in the product image) to the scanner 20. The transmitting / receiving means 2003 receives the image recognition result transmitted by the image recognition device 30. The transmitting / receiving means 2003 also transmits the decoding result by the decoding means 2002 (product code held by the product barcode) to the image recognition device 30.

[0116] The merging means 2004 merges the decoded result (the product code held by the product barcode) and the recognition result (the product code of the product shown in the product image). That is, the merging means 2004 acquires one product code from the decoded result and the image recognition result. The merged result includes the product code held by the product barcode included in the product image or the product code of the product shown in the product image.

[0117] If no decoded result is obtained, the merging means 2004 uses the product code from the image recognition result as the merged result. Situations in which a decoded result cannot be obtained include when the consumer does not correctly hold the product barcode over the imaging unit 220 of the scanner 20, or when the product barcode is unnaturally bent or folded.

[0118] If both the decoded result and the image recognition result are obtained and match, the merging means 2004 naturally takes the matching product code as the merged result. Also, if both the decoded result and the image recognition result are obtained but do not match, the merging means 2004 takes the product code of the more reliable decoded result as the merged result.

[0119] The transmitting means 2005 transmits the product code resulting from the merging by the merging means 2004 to the POS terminal 10 via the connection interface 221. Furthermore, the transmitting means 2005 transmits a disconnection notification signal, which notifies the POS terminal 10 of the disconnection when communication with the image recognition device 30 is disconnected, via the connection interface 221.

[0120] The setting means 2006 stores setting information 213 (see Figure 8) of the operating mode to be switched when communication with the image recognition device 30 is disconnected in the memory unit 210, in accordance with the information of the selected operating mode to which the operation of the scanner 20 should be switched, which information is acquired from the POS terminal 10 by the acquisition means 2001.

[0121] The switching means 2007 acquires a communication signal with the image recognition device 30 transmitted and received by the transmitting and receiving means 2003, and detects a disconnection of communication. When the switching means 2007 detects a disconnection of communication with the image recognition device 30, the switching means 2007 switches the operation mode of the scanner 20 in accordance with the setting information 213.

[0122] In detail, when the switching information (flag) of the setting information 213 (see FIG. 8) is set to operation mode "1", the switching means 2007 continues to detect and decode product barcodes from product images performed by the acquisition means 2001 and the decoding means 2002, and continues to acquire product codes. The switching means 2007 stops the transmission (transfer) of the product images to the image recognition device 30 by the transmission / reception means 2003, and transmits a disconnection notification signal to the POS terminal 10 by the transmission means 2005. The POS terminal 10 emits light from the light-emitting unit 571 to notify the administrator that communication between the scanner 20 and the image recognition device 30 has been disconnected.

[0123] Furthermore, if the switching information (flag) of the setting information 213 (see FIG. 8) is set to operation mode "2," and if the acquisition means 2001 and the decoding means 2002 are currently identifying a product code, the switching means 2007 continues detecting and decoding the product barcode and continues acquiring the product code. The switching means 2007 stops the transmission (transfer) of the product image to the image recognition device 30 by the transmission / reception means 2003. If the product code is not currently being identified, the switching means 2007 stops the acquisition of the product image by the acquisition means 2001 and transmits a disconnection notification signal to the POS terminal 10 by the transmission means 2005. The POS terminal 10 emits light from the light-emitting unit 571 to notify the administrator that communication between the scanner 20 and the image recognition device 30 has been disconnected.

[0124] (Functional configuration of image recognition device) Next, the functional configuration of the image recognition device 30 will be described with reference to Fig. 12. Fig. 12 is a block diagram showing the functional configuration of the control unit 300 of the image recognition device 30.

[0125] The control unit 300 has a receiving means 3001, an image recognition means 3002, a determination means 3003, an update means 3004, and a transmission means 3005. The control unit 300 functions as the receiving means 3001, the image recognition means 3002, the determination means 3003, the update means 3004, and the transmission means 3005 when the CPU 301 operates in accordance with a control program stored in the ROM 302 or the storage unit 310. Note that each of these functions may be configured by hardware such as a dedicated circuit.

[0126] The receiving means 3001 receives the product image from the scanner 20 via the LAN interface 321. The receiving means 3001 stores the received product image in the RAM 303.

[0127] The image recognition means 3002 performs image recognition on the product image using an image recognition function that uses the trained model 312. In detail, the image recognition means 3002 acquires features contained in the product image, compares the acquired features with the matching dictionary 313, and acquires the product code associated with the feature that is closest to the acquired feature. The image recognition means 3002 stores the acquired product code in the RAM 303.

[0128] The transmitting means 3005 transmits the product code, which is the result of image recognition by the image recognizing means 3002 , to the transmitting / receiving means 2003 of the scanner 20 via the LAN interface 321 .

[0129] The determination means 3003 determines whether the image recognition by the image recognition means 3002 was successful. For example, if the difference between the acquired feature amount and the feature amount closest to the acquired feature amount is less than a predetermined amount, the determination means 3003 determines that the image recognition was successful, and conversely, if the difference is equal to or greater than the predetermined amount, the determination means 3003 determines that the image recognition was unsuccessful. Furthermore, if the determination means 3003 determines that the image recognition was unsuccessful, it determines that the product image that was subjected to image recognition is a product image to be used for additional learning.

[0130] Furthermore, the receiving means 3001 receives the decoded product code from the scanner 20 via the LAN interface 321. The receiving means 3001 stores the received product code in the RAM 303.

[0131] If the determination means 3003 determines that the product image that has undergone image recognition is a product image to be used for additional learning, the transmission means 3005 transmits the product image and its product code to the server 40 via the LAN interface 321.

[0132] The server 40 performs additional training on the trained model 412 for product image recognition using the received product images and product codes, and updates the trained model 412 and the matching dictionary 413. The server 40 transmits the updated trained model 412 and the updated matching dictionary 413 to the receiving means 3001 of the image recognition device 30 via the LAN interface 321.

[0133] The receiving means 3001 receives the updated trained model 412 and the updated matching dictionary 413 from the server 40 via the LAN interface 321. The receiving means 3001 stores the received updated trained model 412 and updated matching dictionary 413 in the RAM 303.

[0134] The update means 3004 updates the trained model 312 currently held in the storage unit 310 with the received updated trained model 412. Furthermore, the image recognition device 30 updates the matching dictionary 313 currently held in the storage unit 310 with the received updated matching dictionary 413.

[0135] (Server functional configuration) Next, the functional configuration of the server 40 will be described with reference to Fig. 13. Fig. 13 is a block diagram showing the functional configuration of the control unit 400 of the server 40.

[0136] The control unit 400 has a receiving means 4001, an additional learning means 4002, a master management means 4003, and a transmitting means 4004. The control unit 400 functions as the receiving means 4001, the additional learning means 4002, the master management means 4003, and the transmitting means 4004 when the CPU 401 operates in accordance with a control program stored in the ROM 402 or the storage unit 410. Note that each of these functions may be configured by hardware such as a dedicated circuit.

[0137] The receiving means 4001 receives, via the communication unit 421, product images for additional learning to be used for additional learning and product codes of the product images for additional learning from the image recognition device 30. The receiving means 4001 stores the received product images and product codes in the RAM 403. The receiving means 4001 may store and accumulate the received product images in the RAM 403 or the storage unit 410 as product images for analyzing the state (event) of image recognition in the image recognition device 30, for example.

[0138] The additional learning means 4002 performs additional learning on the trained model 412 held in the storage unit 410 at an appropriate timing, for example, when a certain number of product images and product codes for additional learning have been accumulated, and updates the trained model 412. In addition, the additional learning means 4002 updates the matching dictionary 413 held in the storage unit 410 in accordance with the additional learning of the trained model 412.

[0139] The transmitting means 4004 transmits the updated trained model 412 and the matching dictionary 413 updated by the additional learning means 4002 to the receiving means 3001 of the image recognition device 30 via the communication unit 421, for example, as an image file or a compressed file.

[0140] The receiving unit 4001 also receives transaction processing information from the POS terminal 10 via the communication unit 421. The receiving unit 4001 stores the received transaction processing information in RAM 403.

[0141] The master data management means 4003 manages the product master data 414 stored in the storage unit 410 based on the received transaction processing information. For example, the product master data 414 includes a store code that identifies the store, a product master data 112 for each store code, and information such as whether or not a product with a product code for each store code is available and how many products are left. Based on the received transaction processing information, the master data management means 4003 updates the product master data 414, such as information on the remaining quantity of a product with a product code.

[0142] The transmitting means 4004 transmits the product master 112 suitable for each store to the POS terminal 10 of the store via the communication unit 421 based on the product master 414 updated by the master management means 4003.

[0143] (POS terminal and scanner operation) Next, with reference to Fig. 14, the operation of the POS terminal 10 and the scanner 20 for presetting the operation mode of the scanner 20 when communication between the scanner 20 and the image recognition device 30 is disconnected will be described. Fig. 14 is a flowchart showing the flow of the operation of the POS terminal 10 and the scanner 20 according to the embodiment. In Fig. 14, the operation of the POS terminal 10 is shown on the left side, and the operation of the scanner 20 is shown on the right side. The operation of the POS terminal 10 is performed by each unit of the POS terminal 10 under the control of the control unit 100 which executes a control program 111. The operation of the scanner 20 is performed by each unit of the scanner 20 under the control of the control unit 200 which executes a control program 211.

[0144] The process of presetting the operation mode of the scanner 20 in case of disconnection of communication between the scanner 20 and the image recognition device 30 is started, for example, when an administrator (which may be either the administrator of the product information registration system RS or the administrator of the store) operates the touch panel 52 of the POS terminal 10. This is performed depending on the situation on the store side, for example, whether the operator who is the administrator is absent from the store or is always present there.

[0145] In ACT31, the POS terminal 10 displays a setting screen (not shown) on the display 51 in response to the administrator's operation on the touch panel 52, prompting the operator (here, the administrator) to select the operating mode to be performed by the scanner 20 when communication with the image recognition device 30 is disconnected.

[0146] In ACT32, the operator selects one of multiple operating modes according to the setting screen displayed on the display 51 (if Yes in ACT32), and in ACT33, the POS terminal 10 transmits information on the selected operating mode (operating mode selection information) to the scanner 20.

[0147] In ACT41, when the scanner 20 receives the selection information of the operation mode transmitted from the POS terminal 10 (if Yes in ACT41), in ACT42, the scanner 20 stores the setting information 213 (see FIG. 8) of the operation mode to be switched when communication with the image recognition device 30 is disconnected in the storage unit 210. Here, the scanner 20 sets (sets) the switching information (flag) in association with the information of the selected operation mode.

[0148] In the embodiment, the number of operation modes of the scanner 20 that can be switched when communication with the image recognition device 30 is disconnected is two, but this is not limited to this. In the setting information 213 of the embodiment (see FIG. 8), either operation mode "1" or operation mode "2" is set. Here, operation mode "1" is an operation mode in which the image recognition device 30 cannot identify product codes, but the scanner 20 continues to detect and decode product barcodes, and continues to obtain product codes. Operation mode "1" is suitable, for example, when a managerial operator is not present in the store.

[0149] Furthermore, in operation mode "2," when an operator who is a customer is working on registering a product, detection of product barcodes and acquisition of product codes based on the decoding of the barcodes continues until the product registration is completed, and after the product registration is completed, capturing of product images stops and notifies the operator (here, the manager) of an abnormality (disconnection of communication with the image recognition device 30), which is a so-called error mode. Operation mode "2" is suitable, for example, for situations where an operator who is a manager is always present in the store.

[0150] (Scanner and image recognition device operation) Next, the operations of the scanner 20 and the image recognition device 30 will be described with reference to Fig. 15. Fig. 15 is a flowchart showing the flow of operations of the scanner 20 and the image recognition device 30 according to the embodiment. In Fig. 15, the operations of the scanner 20 are shown on the left side, and the operations of the image recognition device 30 are shown on the right side. The operations of the scanner 20 are performed by each unit of the scanner 20 under the control of the control unit 200 which executes a control program 211. The operations of the image recognition device 30 are performed by each unit of the image recognition device 30 under the control of the control unit 300 which executes a control program 311.

[0151] In ACT11, the scanner 20 acquires one product image from a moving image captured by the imaging unit 220. For example, the scanner 20 first stores all or some of the multiple frame images sequentially captured by the imaging unit 220 in the RAM 203 as product image candidates. Next, the scanner 20 selects one frame image from the multiple frame images stored in the RAM 203 and sets the selected frame image as the product image. The scanner 20 also assigns an identification code to the product image to identify it. For convenience, the product image will be referred to as including the identification code below.

[0152] In ACT12, the scanner 20 transmits the product image to the image recognition device 30 via a LAN, for example, by TCP communication.

[0153] In ACT21, the image recognition device 30 receives the product image and the identification code from the scanner 20 via the LAN. The image recognition device 30 stores the received product image in the RAM 303.

[0154] In ACT22, the image recognition device 30 performs image recognition on the product image. For example, the image recognition device 30 performs image recognition based on feature quantities such as the outline and color of the product depicted in the product image. In this case, first, the image recognition device 30 acquires the feature quantities included in the product image. Next, the image recognition device 30 compares the acquired feature quantities with the matching dictionary 313 and searches for feature quantities that are closest to the acquired feature quantities. Next, the image recognition device 30 acquires the product code associated with the search result (the feature quantities that are closest to the acquired feature quantities).

[0155] In ACT 23, the image recognition device 30 determines whether the image recognition was successful. For example, the image recognition device 30 first calculates the difference between the acquired feature amount and the feature amount closest to the acquired feature amount. Next, the image recognition device 30 compares the calculated difference with a predetermined amount.

[0156] If the comparison result shows that the difference is less than a predetermined amount, the image recognition device 30 determines that the image recognition was successful. In this case, the image recognition device 30 stores the product code acquired in ACT 22 as the image recognition result in the RAM 303. Then, the process proceeds to the operation of ACT 24.

[0157] On the other hand, if the comparison result shows that the difference is equal to or greater than the predetermined amount, the image recognition device 30 determines that the image recognition has failed. In this case, the image recognition device 30 requests the scanner 20 to acquire and transmit another product image, and performs the operations of ACT21 to ACT23 again. The image recognition device 30 also determines that the product image that has undergone image recognition is the product image to be used for additional learning.

[0158] In ACT24, the image recognition device 30 transmits the image recognition result stored in the RAM 303 to the scanner 20 via the LAN. The image recognition device 30 also transmits an identification code to the scanner 20.

[0159] In ACT 13, the scanner 20 detects a product barcode in the product image. In other words, the scanner 20 determines whether or not a product barcode is captured in the product image.

[0160] If the product barcode is detected in ACT 13 (if the answer is Yes in ACT 13), the scanner 20 extracts and decodes the detected product barcode and acquires the product code stored in the product barcode as the decoding result in ACT 14. The scanner 20 stores the acquired product code in RAM 203.

[0161] Next, in ACT15, the scanner 20 transmits the decoded product code (the product code held by the product barcode) to the image recognition device 30. The operations in ACT14 and ACT15 are performed when the product barcode is detected in ACT13, so in ACT15, the scanner 20 may not transmit the decoded product code.

[0162] In ACT16, the scanner 20 checks whether it has received the image recognition result of the product image (product code of the product shown in the product image) from the image recognition device 30. If it has received the image recognition result (Yes in ACT16), the scanner 20 stores the received image recognition result in RAM 203. The scanner 20 may identify the product image corresponding to the image recognition result based on the identification code received together with the image recognition result. Then, it proceeds to the merging operation in ACT17.

[0163] If the product barcode cannot be detected in ACT13 (No in ACT13), the operations in ACT14 and ACT15 are skipped and the process moves to the operation of confirming the reception of the image recognition result in ACT16.

[0164] If the image recognition result has not been received in ACT16 (No in ACT16), the scanner 20 repeats the operations of ACT11 and ACT13 to ACT15. At this time, the image recognition device 30 does not require a new product image, so the scanner 20 omits sending the product image in ACT12. That is, until the scanner 20 receives the image recognition result from the image recognition device 30, it continues a series of regular operations, such as obtaining a product image in ACT11, detecting a product barcode in ACT13, decoding in ACT14, and sending the product code of the decoded result in ACT15.

[0165] In ACT 17, the scanner 20 merges the decoded result (the product code stored in the product barcode) and the image recognition result (the product code of the product shown in the product image). That is, the scanner 20 obtains one product code that is considered to be the most correct from the decoded result and the image recognition result.

[0166] If no decoded result is obtained, the product code from the image recognition result is used as the merged result. Situations in which a decoded result cannot be obtained include when the consumer does not correctly hold the product barcode over the image capturing unit 220 of the scanner 20, or when the product barcode is unnaturally bent or folded.

[0167] When both the decoded result and the image recognition result are obtained, the scanner 20 compares the decoded result with the image recognition result. If the comparison shows that the decoded result and the image recognition result match, the scanner 20 merges the product codes of the matching decoded result and image recognition result. If the comparison shows that the decoded result and the image recognition result do not match, the scanner 20 merges the product code of the more reliable decoded result.

[0168] In ACT 18, the scanner 20 transmits the merged product code (the single product code that is considered to be the most correct) to the POS terminal 10.

[0169] In ACTs 16 to 18 of this embodiment, an example has been shown in which the scanner 20, after receiving the image recognition result from the image recognition device 30 (in the case of Yes in ACT 16), uploads (transmits) to the upper device (here, the POS terminal 10) any product code obtained by merging the product code obtained by decoded the product barcode that is the scan result with the product code that is the image recognition result. However, the scanner 20 may first transmit the product code obtained by decoded the product barcode that is the scan result to the upper device in a state in which the image recognition result has not yet been received from the image recognition device 30, and then, when the image recognition result is received from the image recognition device 30, transmit to the upper device any product code obtained by merging the product code obtained by decoded the product barcode that is the scan result with the product code that is the image recognition result.

[0170] If the scanner 20 transmits the decoded product code (the product code held by the product barcode) in ACT 15, the image recognition device 30 receives the decoded product code in ACT 25. The image recognition device 30 stores the received decoded product code in the RAM 303.

[0171] In ACT 26, the image recognition device 30 determines whether there are any product images to be used for additional learning. For example, if there are any product images for which image recognition has failed in ACT 23 and a product code as a decoded result has been received from the scanner 20 in ACT 25, the image recognition device 30 determines that there are any product images to be used for additional learning.

[0172] If it is determined in ACT26 that there are product images to be used for additional learning, in ACT27, the image recognition device 30 transmits to the server 40 the product images that failed image recognition and are stored in the RAM 303, and the product codes that are the decoded results received from the scanner 20.

[0173] If it is determined in ACT26 that there are no product images to be used for additional learning, the operation of ACT27 is skipped and the process ends.

[0174] (Scanner and POS terminal operation) Next, with reference to Fig. 16, the operation of the scanner 20 and the POS terminal 10 for appropriately switching the operation mode of the scanner 20 when communication between the scanner 20 and the image recognition device 30 is disconnected will be described. Fig. 16 is a flowchart showing the flow of the operation of the scanner 20 and the POS terminal 10 according to the embodiment. In Fig. 16, the operation of the scanner 20 is shown on the left side, and the operation of the POS terminal 10 is shown on the right side. The operation of the scanner 20 is performed by each unit of the scanner 20 under the control of the control unit 200 which executes a control program 211. The operation of the POS terminal 10 is performed by each unit of the POS terminal 10 under the control of the control unit 100 which executes a control program 111.

[0175] As described above, the scanner 20 and the image recognition device 30 in this embodiment communicate via TCP over a LAN, but there is a possibility that the communication may be interrupted for some reason. For example, the communication may be interrupted if an error occurs in the operation of the image recognition device 30, causing the control unit 300 to forcibly terminate the processing being executed, or if an abnormality occurs in the LAN cable connecting the image recognition device 30 to the image recognition device 30.

[0176] In ACT51, the scanner 20 acquires and monitors a communication signal transmitted and received with the image recognition device 30 via the LAN interface 222, and in ACT52, detects whether the communication has been disconnected.

[0177] In ACT 52, when the scanner 20 detects a disconnection from the image recognition device 30, for example, because a communication signal from the image recognition device 30 is no longer received (if Yes in ACT 52), in ACT 53 the scanner 20 checks the setting information 213 (see FIG. 8) stored in the storage unit 210. That is, the scanner 20 determines whether the switching information (flag) of the setting information 213 is set to the operation mode "1" or the operation mode "2."

[0178] In ACT 54, when the scanner 20 determines that the switching information (flag) of the setting information 213 is set to operation mode "1" (in the case of (1) in ACT 54), the scanner 20 continues to acquire the product code based on the detection of the product barcode from the product image in ACT 55. In addition, in ACT 56, the scanner 20 stops transmitting (transferring) the product image to the image recognition device 30.

[0179] 15, except for the processing related to the image recognition device 30 with which communication has been disconnected, the scanner 20 continues to acquire product codes based on the detection of product barcodes from product images in ACT11, ACT13-14, and ACT17-18, and to transmit the product codes to the POS terminal 10. When communication between the scanner 20 and the image recognition device 30 is disconnected, the operator, who is the customer, can continue to register products even if, for example, the store manager is not present.

[0180] Returning to FIG. 16, in ACT57, the scanner 20 transmits a disconnection notification signal to the POS terminal 10 to notify that communication with the image recognition device 30 has been disconnected.

[0181] In ACT71, the POS terminal 10 receives a disconnection notification signal from the scanner 20, and in ACT72, the POS terminal 10 emits light from the light-emitting unit 571 to notify the manager that communication between the scanner 20 and the image recognition device 30 has been disconnected. In this case, even if the manager is not in the store, when the manager makes a regular visit to the store, for example, the manager can know that communication between the scanner 20 and the image recognition device 30 has been disconnected by the light emitted from the light-emitting unit 571, and can quickly start work to restore communication.

[0182] In ACT 73, for example, when the administrator stops the light emission of the light emitting unit 571 of the POS terminal 10, the POS terminal 10 determines that the administrator has confirmed the disconnection notification (if Yes in ACT 73). Then, in ACT 74, the POS terminal 10 transmits to the scanner 20 a confirmation signal indicating that the administrator has confirmed the disconnection of communication between the scanner 20 and the image recognition device 30.

[0183] On the other hand, if the scanner 20 determines in ACT 54 that the switching information (flag) of the setting information 213 is set to operation mode "2" (case (2) in ACT 54), the scanner 20 determines in ACT 59 whether the scanner 20 is currently identifying a product code from a product image. That is, the scanner 20 determines whether the scanner 20 is currently identifying a product code (registering a product) depending on whether the scanner 20 is in the middle of the process from acquiring a product image to transmitting a product code in the flowchart of the scanner 20 in Fig. 15.

[0184] If the scanner 20 determines in ACT 59 that the product code is being identified (product registration is in progress) (if Yes in ACT 59), then in ACT 60 the scanner 20 continues to acquire the product code based on the detection of the product barcode from the product image. Furthermore, in ACT 61 the scanner 20 stops sending (transferring) the product image to the image recognition device 30. If the scanner 20 determines in ACT 59 that the product code is not being identified (product registration is in progress) (if No in ACT 59), then in ACT 62 the scanner 20 stops acquiring the product image and stops the processing of the scanner 20 in the flowchart of FIG. 15. If the customer, who is an operator, is in the middle of product registration, he or she can continue until the product registration is completed.

[0185] In ACT 57, the scanner 20 transmits a disconnection notification signal to the POS terminal 10 to notify that communication with the image recognition device 30 has been disconnected. When the POS terminal 10 receives the disconnection notification signal from the scanner 20 in ACT 71, the POS terminal 10 emits light from the light-emitting unit 571 to notify the manager that communication between the scanner 20 and the image recognition device 30 has been disconnected in ACT 72. In this case, the manager who is stationed at the store can know that communication between the scanner 20 and the image recognition device 30 has been disconnected from the light-emitting unit 571 of the POS terminal 10, and can quickly start work to restore communication. In this case, the POS terminal 10 may output a notification sound from the audio output unit 120 in addition to emitting light from the light-emitting unit 571, to notify not only the store manager but also the operator, who is a customer, that new product registration is not possible.

[0186] In ACT 73, for example, when the administrator stops the light emission of the light emitting unit 571 of the POS terminal 10, the POS terminal 10 determines that the administrator has confirmed the disconnection notification (if Yes in ACT 73). Then, in ACT 74, the POS terminal 10 transmits to the scanner 20 a confirmation signal indicating that the administrator has confirmed the disconnection of communication between the scanner 20 and the image recognition device 30.

[0187] In ACT58, when the scanner 20 receives a confirmation signal from the POS terminal 10 (if Yes in ACT58), the scanner 20 ends the series of processes for switching the operating mode of the scanner 20 when communication with the image recognition device 30 is disconnected.

[0188] (Summary of the embodiment) According to the embodiment of the product information registration system RS, when communication between the scanner 20 and the image recognition device 30 connected via a LAN is cut off, the scanner 20 switches to operation in operation mode "1", which is suitable for when the manager is not present in the store, or operation mode "2", which is suitable for when the manager is present in the store, in accordance with the setting information 213 of the operation mode that has been preset in response to the operation of the operator who is, for example, the manager.

[0189] In operation mode "1," the scanner 20 continues to detect product barcodes and obtain product codes based on their decoding, even when the image recognition device 30 is unable to obtain product codes from product images, allowing the customer / operator to continue product registration without interruption.

[0190] In operation mode "2," if the scanner 20 is identifying a product code (registering a product), it continues to detect the product barcode and acquire the product code based on the decoded product barcode. After the scanner 20 has identified the product code, it stops acquiring product images, transmits a disconnection notification signal to the POS terminal 10 to notify that communication has been disconnected, causes the light-emitting unit 571 of the POS terminal 10 to emit light, and outputs a notification sound from the audio output unit 120. If a product is currently being registered, the operator (customer) can continue until the product registration is complete, and can also know that a new product cannot be registered. Furthermore, the operator (administrator) can quickly begin work to restore communication.

[0191] The number of switchable operation modes preset in the scanner 20 is not limited to two as described above in the embodiment. In other words, a plurality of operation modes may be set appropriate for the situation in which the product information registration system RS will be placed when communication between the scanner 20 and the image recognition device 30 is cut off, i.e., the situation of the store and the customer.

[0192] (others) In the above embodiment, the control programs executed by the POS terminal 10, the scanner 20, the image recognition device 30, and the server 40 may be configured to be recorded on a computer-readable recording medium such as a CD-ROM and provided. Also, the control programs executed by the POS terminal 10, the scanner 20, the image recognition device 30, and the server 40 in the above embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network, or may be provided via a network such as the Internet.

[0193] Although an embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its modifications are included within the scope and spirit of the invention, and are also included in the inventions described in the claims and their equivalents. The following is a summary of the scope of claims as originally filed in this application. [C1] An information processing device having a communication function with an external device, a setting means for setting in advance an operation of the information processing device when communication with the external device is disconnected to one of a plurality of operations; a switching means for switching the operation of the information processing device to the operation set by the setting means when communication with the external device is disconnected; An information processing device comprising: [C2] the setting means sets in advance an operation of the information processing device when communication with the external device is disconnected to one of a plurality of operations in response to a user operation; The information processing device described in C1. [C3] image acquisition means for acquiring a captured image of a commodity; a code acquisition means for acquiring a product code of the product based on code information attached to the product included in the captured image acquired by the image acquisition means; a transmitting means for transmitting the captured image acquired by the image acquiring means to the external device; a receiving means for receiving a product code of the product acquired by the external device based on the external characteristics of the product included in the captured image; a code transmitting means for transmitting the product code acquired by the code acquiring means or the product code received by the receiving means to another external device; The plurality of operations includes at least an operation of continuing acquisition of a product code by the code acquisition means. The information processing device described in C2. [C4] The plurality of operations include at least an operation of stopping the acquisition of the captured image by the image acquisition means. The information processing device described in C3. [C5] a notification means for notifying the user of the disconnection of communication with the external device when the communication with the external device is disconnected; The information processing device according to C3 or C4. [C6] A computer of an information processing device having a communication function with an external device, a setting means for setting in advance an operation of the information processing device when communication with the external device is disconnected to one of a plurality of operations; a switching means for switching the operation of the information processing device to the operation set by the setting means when communication with the external device is disconnected; A program to function as a [Explanation of symbols]

[0194] 1...self-service POS system, 10...POS terminal, 100...control unit, 1001...input means, 1002...registration means, 1003...accounting processing means, 1004...output means, 1005...setting means, 101...CPU, 102...ROM, 103...RAM, 1031...product information unit, 110...storage unit, 111...control program, 112...product master, 120...audio output unit, 121...connection interface, 122...LAN interface, 123...bus, 50...main body, 51...display, 52...touch panel, 53...printer, 531 ...issuing port, 54...card reader, 541...card insertion port, 55...change dispenser, 551...coin insertion port, 552...coin discharge port, 553...coin tray, 554...banknote insertion port, 555...banknote discharge port, 57...pole, 571...light emitting unit, 70...first basket placement table, 71...first weighing unit, 80...second basket placement table, 81...second weighing unit, 82...support rod, 83...temporary placement table, 20...scanner, 200...control unit, 2001...acquiring means, 2002...decoding means, 2003...transmitting / receiving means, 2004...merging means, 2005...transmitting means, 2006...setting means , 2007...switching means, 201...CPU, 202...ROM, 203...RAM, 210...storage unit, 211...control program, 212...processing result, 213...setting information, 220...imaging unit, 221...connection interface, 222...LAN interface, 223...bus, 230...reading window, 30...image recognition device, 300...control unit, 3001...receiving means, 3002...image recognition means, 3003...determination means, 3004...updating means, 3005...transmitting means, 301...CPU, 302...ROM, 303...RAM, 310...storage unit, 31 1...control program, 312...trained model, 313...matching dictionary, 321...LAN interface, 322...bus, 40...server, 400...control unit, 4001...receiving means, 4002...additional learning means, 4003...master management means, 4004...transmitting means, 401...CPU, 402...ROM, 403...RAM, 410...memory unit, 411...control program, 412...trained model, 413...matching dictionary, 414...product master, 421...communication unit, 422...bus, LH...LAN hub, NW...network, WR...wireless router.

Claims

1. An information processing device having a communication function with an image recognition device, a setting means for setting in advance an operation mode of the information processing device to be switched when communication with the image recognition device is disconnected to one of a plurality of switchable operation modes; a switching means for switching the operation mode of the information processing device to the operation mode set by the setting means when communication with the image recognition device is disconnected; Equipped with image acquisition means for acquiring a captured image of a commodity; a code acquisition means for acquiring a product code of the product based on code information attached to the product included in the captured image acquired by the image acquisition means; a transmitting means for transmitting the captured image acquired by the image acquisition means to the image recognition device; a receiving means for receiving a product code of the product acquired by the image recognition device based on the external features of the product included in the captured image; a code transmitting means for transmitting the product code acquired by the code acquiring means or the product code received by the receiving means to another external device; the setting means pre-sets an operation mode of the information processing device to be switched to when communication with the image recognition device is disconnected to one of a plurality of switchable operation modes in response to a user operation; the plurality of operation modes includes at least an operation mode in which the code acquisition means continues to acquire the product code; Information processing device.

2. the plurality of operation modes includes at least an operation mode in which acquisition of captured images by the image acquisition means is stopped; The information processing device according to claim 1 .

3. a notification means for notifying the user of a disconnection of communication with the image recognition device when the communication with the image recognition device is disconnected; 3. The information processing device according to claim 1.

4. A computer of an information processing device having a communication function with an image recognition device, a setting means for setting in advance an operation mode of the information processing device to be switched when communication with the image recognition device is disconnected to one of a plurality of switchable operation modes; a switching means for switching the operation mode of the information processing device to the operation mode set by the setting means when communication with the image recognition device is disconnected; It functions as image acquisition means for acquiring a captured image of the product; a code acquisition means for acquiring a product code of the product based on code information attached to the product included in the captured image acquired by the image acquisition means; a transmitting means for transmitting the captured image acquired by the image acquisition means to the image recognition device; a receiving means for receiving a product code of the product acquired by the image recognition device based on the external features of the product included in the captured image; a code transmitting unit configured to transmit the product code acquired by the code acquiring unit or the product code received by the receiving unit to another external device; the setting means pre-sets an operation mode of the information processing device to be switched to when communication with the image recognition device is disconnected to one of a plurality of switchable operation modes in response to a user operation; the plurality of operation modes includes at least an operation mode in which the code acquisition means continues to acquire the product code; A program to make it work like this.

Images