Recognition device, recognition method, and program

Abstract

This recognition device comprises: an acquisition unit that acquires, from a plurality of cameras that capture images of an article placed on a placement table from mutually different angles, captured images of the article; an object recognition unit that recognizes the article placed on the placement table by object recognition processing using the images; and an output unit that outputs information indicating the recognized article.

Description

Recognition Device, Recognition Method, and Program

[0001] The present disclosure relates to a recognition device, a recognition method, and a program.

[0002] There are stores that install self-checkout (unattended checkout) to reduce labor costs and eliminate checkout queues. In self-checkout, for example, it is important to improve the ease of use for customers and to suppress misrecognition of items to be purchased (registered).

[0003] In Patent Document 1, in order to more efficiently select products, when extracting the feature amount of a product image and the code symbol attached to the product, a product data processing device that presents product information corresponding to the code symbol is disclosed.

[0004] Japanese Patent Application Laid-Open No. 2013-89084

[0005] As described in Patent Document 1, the technology of recognizing an item based on the features in an image is useful when it is difficult to know the position where a barcode or the like is attached to the item. However, the features of an item do not necessarily appear when the item is photographed at a specific angle. For example, it is difficult to distinguish an item such as a bottle from an image photographed directly from above. Also, in an environment where a wide variety of items are handled, such as a retail store, since the characteristic parts are different for each item, it is difficult to adjust at what angle the item should be photographed. Therefore, a recognition device that can appropriately photograph the characteristic part of an item is desired.

[0006] Also, the need to appropriately photograph the characteristic part of an item is not limited to a system for retail stores such as self-checkout. For example, in order to manage items such as parts and work-in-progress in a warehouse or factory, information on the items is registered, or in order to manage items such as luggage at a distribution site, information on the luggage is registered. In such cases as well, there is a need to be able to appropriately photograph the characteristic part of an item in order to recognize the item based on an image.

[0007] Therefore, the non-limiting embodiments of the present disclosure contribute to providing a recognition device, a recognition method, and a program that can appropriately photograph the characteristic part when recognizing various items by an image.

[0008] An embodiment of the recognition device according to the present disclosure includes an acquisition unit that acquires images of an article placed on a mounting platform from a plurality of cameras that photograph the article from different angles; an object recognition unit that recognizes the article placed on the mounting platform by object recognition processing using the images; and an output unit that outputs information indicating the recognized article.

[0009] In one embodiment of the present disclosure, the recognition device acquires images of an article placed on a platform from a plurality of cameras that photograph the article from different angles, recognizes the article placed on the platform by object recognition processing using the images, and outputs information indicating the recognized article.

[0010] A program according to one embodiment of the present disclosure causes a computer to acquire images of an article placed on a platform from multiple cameras that photograph the article from different angles, recognize the article placed on the platform using the images through object recognition processing, and output information indicating the recognized article.

[0011] These comprehensive or specific embodiments may be implemented as a system, apparatus, method, integrated circuit, computer program, or recording medium, or as any combination of a system, apparatus, method, integrated circuit, computer program, and recording medium.

[0012] According to one embodiment of the present disclosure, the recognition device can appropriately photograph characteristic parts of an article.

[0013] Further advantages and effects of one embodiment of this disclosure will be made apparent from the specification and drawings. Such advantages and / or effects are provided by several embodiments and features described in the specification and drawings, but not all of them are necessarily provided in order to obtain one or more identical features.

[0014] Diagram of the recognition system according to the first embodiment Diagram of the AI ​​scanner viewed from approximately the front Diagram of the AI ​​scanner viewed from the front and diagonally above Diagram of the AI ​​scanner viewed from above Diagram of the AI ​​scanner viewed from the front and diagonally below Diagram of the AI ​​scanner viewed from the front and diagonally below Diagram showing a part of the mounting base viewed from diagonally above Perspective view of the AI ​​scanner equipped with a turntable Front view of the AI ​​scanner with the front plate of the base omitted Perspective view of the AI ​​scanner with the front plate and side plates of the base omitted Hardware configuration diagram of the control device Functional block diagram of the control device Flowchart showing the learning operation of the product Diagram showing an image displayed on the display a top plate according to the second embodiment viewed from below Example of operation of the control device according to the second embodiment Flowchart 21: Flowchart for the integrated judgment process in the diagram 21. Diagram showing the field of view of the reader when the AI ​​scanner is viewed from the front. Diagram showing the field of view of the reader when the AI ​​scanner is viewed from the front right diagonally. Diagram showing the field of view of the reader when the AI ​​scanner is viewed from the front right diagonally upper. Diagram showing the field of view of the camera when the AI ​​scanner is viewed from the front right. Diagram showing the field of view of the second reader when the AI ​​scanner is viewed from the front. Diagram illustrating the position where a barcode is attached to an item. Diagram showing an example of item information output by the information output unit. Diagram showing an example of rules for determining the output result.

[0015] The embodiments of this disclosure will be described in detail below, with reference to the drawings as appropriate. However, some unnecessarily detailed explanations may be omitted. For example, detailed explanations of already well-known matters and redundant explanations of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily verbose and to facilitate understanding for those skilled in the art.

[0016] The attached drawings and the following description are provided to enable a person skilled in the art to fully understand this disclosure, and are not intended to limit the subject matter described in the claims.

[0017] (First Embodiment) <System Configuration Diagram> Figure 1 is a configuration diagram of the recognition system according to the first embodiment. As shown in Figure 1, the recognition system includes an AI (Artificial Intelligence) scanner 1, a hand scanner 2, a payment terminal 3, and a POS (Point of Sale) device 4. The recognition system is installed, for example, in a store and used as a cell register for customers to purchase goods. The AI ​​scanner 1, the hand scanner 2, and the payment terminal 3 are connected to the POS device 4.

[0018] AI Scanner 1 is a recognition device that recognizes objects. AI Scanner 1 has a platform 11 on which objects are placed. AI Scanner 1 recognizes objects placed on the platform 11. Figure 1 shows a PET bottle of beverage A1 as an example of an object placed on the platform 11. As will be described later, AI Scanner 1 is equipped with a reader that reads identification information printed on the object, such as a barcode or a 2D code, and a camera that photographs the object, and recognizes (identifies) the object placed on the platform 11. In the following explanation, the identification information attached to the object will be described as a barcode.

[0019] The AI ​​scanner 1 outputs information indicating the item it recognizes using both or both of its reader and camera to the POS device 4. The information indicating the item includes, for example, an item identification number such as a JAN (Japan Article Number) code. In addition to the item identification number, the information indicating the item may also include information about the item (additional information) such as the number of items, product name, price, size, and weight. Hereafter, the information indicating the item may be referred to as item information.

[0020] The hand scanner 2 reads the barcode attached to the item and outputs the item information to the POS device 4. The hand scanner 2 is used, for example, when the AI ​​scanner 1 cannot recognize the item. In the following, expressions such as reading and obtaining the barcode may include reading and obtaining the barcode code (value).

[0021] The payment terminal 3 is a terminal that processes payment for goods using, for example, a credit card, electronic money, or a 2D code. The payment terminal 3 outputs the payment information for the goods to the POS device 4.

[0022] POS device 4 is a device that registers (manages) information at the time of sale of goods, such as sales performance of goods.

[0023] The hand scanner 2 and payment terminal 3 may be connected to the AI ​​scanner 1 and communicate with the POS device 4 via the AI ​​scanner 1.

[0024] The operation of the recognition system, as shown in Diagram 1, can be divided into (1) item registration operation and (2) item learning operation. For example, in registration (recognition) mode, the recognition system performs the item registration operation, and in learning mode, it performs the item learning operation.

[0025] (1) Item registration operation The customer places the item to be purchased on the platform 11 of the AI ​​scanner 1. When the AI ​​scanner 1 recognizes the item placed on the platform 11, it outputs the item information to the POS device 4. Based on the item information from the AI ​​scanner 1, the POS device 4 registers the item that the customer intends to purchase. When the POS device 4 registers the item, the AI ​​scanner 1 notifies the customer that the item placed on the platform 11 has been registered by sound or by an image on the display.

[0026] When the AI ​​scanner 1 recognizes that an item has been registered, the customer places the next item on the tray 11 if there are still items they wish to purchase. If the customer has no further items to purchase, they use the payment terminal 3 to pay for the registered items. The POS device 4 processes the payment for the registered items in response to the customer's operation of the payment terminal 3.

[0027] In this way, customers can complete the payment process simply by placing the items they wish to purchase on the display stand of the AI ​​scanner 1.

[0028] If the AI ​​scanner 1 fails to recognize an item placed on the mounting table 11, it will notify the customer that the item could not be recognized by sound or by an image on the display.

[0029] If a customer receives notification that the AI ​​scanner 1 was unable to recognize an item, they follow instructions, for example, from the AI ​​scanner 1 (either a sound or an image on the display) and scan the barcode on the item using the hand scanner 2. The POS device 4 receives the item information from the hand scanner 2.

[0030] Thus, if the AI ​​scanner 1 cannot recognize an item, the customer can use the hand scanner 2 to pay for the item.

[0031] The above describes an example of use in a self-checkout system, but it is not limited to this. A store employee may place the customer's purchased items on the display tray 11 on behalf of the customer and have the system recognize the items.

[0032] (2) Learning operation of items The AI ​​scanner 1 recognizes and learns the items placed on the mounting table 11. For example, if an item has a barcode, the AI ​​scanner 1 learns to associate the item (item image) captured by the camera with the barcode. Also, for example, in-store products such as prepared foods or items without barcodes such as vegetables, the AI ​​scanner learns to associate the item captured by the camera with the identification code managed by the store.

[0033] As a result, AI scanner 1 can acquire the barcode of an item purchased (registered) by a customer using a reader, and can also acquire the barcode of an item or an identification code managed by the store from an image captured by the camera using the learned results. In the following, barcodes will be assumed to include an identification code managed by the store.

[0034] Furthermore, the learning of items may be performed, for example, by an AI scanner 1 placed in the back room of the store, or by an AI scanner 1 placed in the checkout area of ​​the sales floor. If the learning is performed by an AI scanner 1 placed in the checkout area of ​​the sales floor, the AI ​​scanner 1 may learn the items that customers place on the display stand 11 for item registration.

[0035] Various learning methods are known for object recognition using images. For example, there are methods that use a machine learning model that pre-records correct images of objects and determines whether there are similar correct images of the objects included in a captured image, and methods that directly train the machine learning model on objects. In the former case, even without updating the machine learning model itself, it is possible to recognize new objects or improve the recognition accuracy of known objects by adding correct images. In the latter case, since the machine learning model itself is trained on the characteristics of objects, it is necessary to rebuild the machine learning model each time training is performed. In the form of this implementation, these methods will be collectively referred to as "training".

[0036] Furthermore, if the AI ​​scanner 1 is learning about items in, for example, the back room of a store, it may be equipped with a turntable (described later) on the mounting base 11. The turntable may be attachable and detachable.

[0037] <Configuration of the AI ​​scanner> Figure 2 is a view of the AI ​​scanner 1 from approximately the front. Figure 3 is a view of the AI ​​scanner 1 from the front and obliquely above. Figure 4 is a view of the AI ​​scanner 1 from above. Figures 5 and 6 are views of the AI ​​scanner 1 from the front and obliquely below. Figure 7 is a view of a part of the mounting base 11 from obliquely above. Figure 8 is a perspective view of the AI ​​scanner 1 equipped with a turntable. Figure 9 is a front view of the AI ​​scanner 1 with the front panel of the base 1a omitted. Figure 10 is a perspective view of the AI ​​scanner 1 with the front panel and side panels of the base 1a omitted.

[0038] In FIGS. 2 to 8, the same components as those in FIG. 1 are denoted by the same reference numerals. Note that the AI scanner 1 shown in FIG. 1 is a simplified illustration of the AI scanners 1 shown in FIGS. 2 to 8, and some components are shown with omissions. The right and left described below refer to the right and left when the AI scanner 1 is viewed from the front.

[0039] 1. Overall Configuration As shown in FIG. 2, the AI scanner 1 has a base portion 1a, a back plate portion 1b, and a top plate portion 1c.

[0040] The base portion 1a has a substantially rectangular parallelepiped shape. The base portion 1a has a plate 1aa on its right side surface. The hand scanner 2 and the payment terminal 3 are placed on the plate 1aa, for example.

[0041] Note that FIG. 6 is a view in which the front plate 1ab of the base portion 1a shown in FIG. 5 is not shown. That is, FIG. 6 shows the inside of the base portion 1a.

[0042] The back plate portion 1b is a substantially rectangular plate-like member. The back plate portion 1b is provided so as to extend upward on the back side of the base portion 1a.

[0043] The top plate portion 1c is a substantially rectangular plate-like member. The top plate portion 1c is provided at the upper end of the back plate portion 1b.

[0044] 2. Mounting Table As shown in FIGS. 2, 3, and 4, the base portion 1a has a rectangular mounting table 11 on its upper surface. The mounting table 11 is formed of a substantially transparent (including transparent) member such as a glass plate, for example.

[0045] An article is placed on the mounting table 11. Since the mounting table 11 is substantially transparent, the article placed on the mounting table 11 can be seen from the inside of the base portion 1a. Thereby, a reader (described later) disposed below the mounting table 11 can read the barcode of the article placed on the mounting table 11.

[0046] As shown in FIG. 3, the mounting table 11 has a marker 11a indicating a predetermined range, such as guiding the placement of an article to the center of the mounting table 11. The marker 11a is substantially transparent so as not to affect the reading of the barcode of the reader disposed below the mounting table 11. The marker 11a may be a thin plate or may be attached to the front or back surface of the mounting table 11.

[0047] 3. Reader As shown in FIGS. 2, 4, and 6, the back plate portion 1b has readers 12a and 12b at its left and right ends. In other words, the readers 12a and 12b are disposed above (upper side) the mounting table 11.

[0048] As shown in FIGS. 4, 6, and 7, the base portion 1a has readers 12c, 12d, 12e, and 12f inside. In other words, the readers 12c, 12d, 12e, and 12f are disposed below (lower side) the mounting table 11.

[0049] The readers 12a and 12b disposed at the upper left and right of the back plate portion 1b irradiate, for example, a laser from above and in the left-right direction of the mounting table 11 toward the marker 11a portion of the mounting table 11. The readers 12c and 12d disposed at the front and left and right inside the base portion 1a irradiate, for example, a laser from below, in the front, and in the left-right direction of the mounting table 11 toward the marker 11a portion of the mounting table 11. The readers 12e and 12f disposed at the left and right centers and in the front and back inside the base portion 1a irradiate, for example, a laser from below, at the center, and in the front and back direction of the mounting table 11 toward the marker 11a portion of the mounting table 11.

[0050] The readers 12a to 12f detect, with an optical sensor, the laser light reflected by the barcode attached to the article placed on the marker 11a portion of the mounting table 11 and read the barcode. The readers 12a to 12f enable the reading of the barcode in all directions of the article placed on the mounting table 11 by the above-described arrangement and laser irradiation.

[0051] As shown in Figure 33, there are various possible locations where barcodes are attached to items. Therefore, if there are blind spots in the barcode reader, there is a risk that the barcode will not be read. In this embodiment, the barcode is configured so that its field of view covers the entire surface.

[0052] Readers 12a to 12f include readers positioned opposite each other across the mounting base 11. For example, when viewing the AI ​​scanner 1 from above, readers 12a, 12b, 12c, and 12d are positioned at the vertices of a rectangle. Readers 12a and 12d are positioned on the diagonal of the rectangle, and readers 12b and 12c are also positioned on the diagonal of the rectangle. Furthermore, readers 12a and 12d are positioned opposite each other across the mounting base 11. Readers 12b and 12c are also positioned opposite each other across the mounting base 11. This arrangement of readers 12a, 12b, 12c, and 12d improves the reading of barcodes on items placed on the mounting base 11. Items placed on the mounting base 11 may be placed with the barcode side facing up or facing down. In this embodiment, since at least one reader 12 is positioned on either side of a substantially transparent mounting base 11, barcodes can be read regardless of which side of the item is facing upwards.

[0053] As will be described later, when an item is placed on top of the marker 11a (placed within a predetermined range on the mounting platform 11), the AI ​​scanner 1 performs object recognition using the camera described below. Meanwhile, readers 12a to 12f are positioned to enable barcode reading from a range wider than the predetermined range. Readers 12a to 12f read barcodes regardless of whether the item is located within the predetermined range or not. The AI ​​scanner 1 detects the item placed on top of the mounting platform 11 and then recognizes the item. This item detection can be performed by taking the background difference between the mounting platform 11 with nothing on it and the item, and can therefore be performed even before object recognition. Accordingly, barcode reading by reader 12 may start when it is detected that an item has been placed on top of the mounting platform by item detection. Alternatively, it may be determined whether an item is placed on top of the marker 11a by estimating the position where the item was detected as the position of the item.

[0054] In other words, barcode recognition by readers 12a to 12f has the advantage of being less constrained than object recognition using a camera. Taking advantage of the aforementioned features, readers 12a to 12f are positioned to be able to read barcodes from a range wider than a predetermined range. Furthermore, readers 12a to 12f perform barcode reading regardless of whether the item is located within the predetermined range or not. This further improves the barcode reading performance of the AI ​​scanner 1.

[0055] Furthermore, as long as barcodes can be read from all directions of the items placed on the mounting platform 11, the positions and number of readers installed above and below the mounting platform 11 are not limited to the above example.

[0056] 4. Camera As shown in Figures 2, 5, and 6, the top plate 1c has cameras 13a to 13d. Camera 13a is located at the left end of the top plate 1c, camera 13b is located at the right end of the top plate 1c, and camera 13c is located in the center of the top plate 1c. Cameras 13a to 13c are used for image recognition of an object placed on the mounting base 11. Cameras 13a to 13c are, for example, visible light cameras that capture images. Note that the images do not need to be full-color images, and include black and white and grayscale images.

[0057] Camera 13a photographs the item placed on the mounting platform 11 from above and to the left. Camera 13b photographs the item placed on the mounting platform 11 from above and to the right. Camera 13c photographs the item placed on the mounting platform 11 from above and to the center. In other words, cameras 13a to 13c, which are installed on the top plate portion 1c, photograph the entire upper part of the item placed on the mounting platform 11. That is, cameras 13a to 13c each photograph the mounting platform 11 at different angles, thereby photographing the item placed on the mounting platform 11 from a variety of angles.

[0058] Camera 13d is positioned at the left end of the top plate 1c. Camera 13d is a visible light camera that captures images above the mounting base 11. Camera 13d is used to capture images of the customer's hands when registering items with the AI ​​scanner 1. Camera 13d is used to capture images to monitor suspicious customer behavior, etc. However, camera 13d may also be used together with cameras 13a to 13c to capture images used for image recognition of items.

[0059] Of the readers 12a to 12f described above, at least one is positioned to read the barcode of an item placed on the mounting platform 11 from a direction approximately opposite to the shooting direction of the cameras 13a to 13c that photograph the item. For example, reader 12c is positioned to read the barcode of an item placed on the mounting platform 11 from a direction approximately opposite to the shooting direction of camera 13b. Reader 12d is positioned to read the barcode of an item placed on the mounting platform 11 from a direction approximately opposite to the shooting direction of camera 13a. Readers 12e and 12f are positioned to read the barcode of an item placed on the mounting platform 11 from a direction approximately opposite to the shooting direction of camera 13c.

[0060] Characteristic markings such as product names and images of contents are generally placed on the front of an item, while barcodes are often placed on the back. Furthermore, ingredient lists are frequently found on the back of items, but these are generally written in similar formats, making it difficult to discern the unique characteristics of each item. Therefore, when recognizing an object using image recognition, an image of the front of the item is more likely to yield accurate recognition results than an image of the back. In other words, characteristic markings that are easily recognized by a camera and barcodes are often placed on opposite sides of the item. As described above, at least one of the readers 12a to 12e is positioned to read the barcode of the item placed on the mounting table 11 from approximately opposite directions to the shooting direction of the cameras 13a to 13c that photograph the item. This allows the AI ​​scanner 1 to acquire item information using both the reader and the camera.

[0061] Furthermore, the position and number of cameras are not limited to the above example, as long as the entire upper part of the item placed on the mounting platform 11 can be photographed.

[0062] 5. Display As shown in Figures 2, 3, 4, and 6, the AI ​​scanner 1 has displays 14a and 14b. Display 14a is mounted substantially perpendicular to the surface (front) of the back plate portion 1b at the rear of the mounting base 11. Display 14b is fixed to the end (right end) of the back plate portion 1b.

[0063] The display 14a shows instructions on how to use the AI ​​scanner 1. For example, the display 14a shows customer actions, such as instructing the customer to place an item in the center of the placement platform 11 (on the marker 11a). The display 14a also shows a group of similar items to the item placed on the placement platform 11. The display of similar items is explained in Figure 19. The display 14a also shows the recognition result of the item placed on the placement platform 11. For example, the display 14a may show an image of the recognized item as the recognition result. The image of the recognized item may be a pre-registered image or an image taken by cameras 13a to 13c.

[0064] As described above, the display 14a is positioned behind the stand 11 where the customer places items. Therefore, the images displayed on the display 14a can naturally come into the customer's line of sight.

[0065] Display 14b displays information related to the payment terminal 3, such as the payment results from the payment terminal 3. Display 14b also displays information related to the POS device 4, such as a list of purchased items.

[0066] The displays 14a and 14b are equipped with input devices such as touch panels for operation by customers or store staff. The AI ​​scanner 1 may be equipped with input devices such as key input devices in addition to, or instead of, the touch panels of the displays 14a and 14b.

[0067] Furthermore, the content displayed on display 14a may be displayed on display 14a, and the content displayed on display 14b may be displayed on display 14a. Also, there may be one display or three or more displays. If there is one display, the content of displays 14a and 14b may be displayed on that one display.

[0068] 6. Light-emitting devices As shown in Figures 2, 3, and 5, the AI ​​scanner 1 has rectangular light-emitting devices 15a, 15b, and 15c. Light-emitting devices 15a and 15b are positioned to the left and right of the display 14a. Light-emitting device 15c is positioned on the top or top plate 1c of the display 14a. In other words, the light-emitting devices 15a, 15b, and 15c are positioned above the mounting base 11 and emit light from above the mounting base 11 toward the mounting base 11.

[0069] As shown in Figures 6 and 7, the AI ​​scanner 1 has rectangular light-emitting devices 15d to 15g. The light-emitting devices 15c to 15g are located inside the base portion 1a, that is, below the mounting table 11. The light-emitting devices 15c to 15g irradiate light from below the mounting table 11 toward the mounting table 11.

[0070] The light-emitting devices 15a to 15g (light-emitting surfaces) are positioned on either side of the mounting base 11, not facing the readers 12a to 12f (light-receiving surfaces of the readers). For example, the light-emitting devices 15a, 15b, and 15c positioned above the mounting base 11 are positioned not facing the readers 12c, 12d, 12e, and 12f positioned below the mounting base 11. These light-emitting devices 15 are used to illuminate the surface of an object to improve the accuracy of image recognition, or to illuminate a barcode to improve the accuracy of barcode reading. However, highlights caused by the light from the light-emitting devices 15 may cause some parts of the object to be overexposed, or shadows caused by the light from the light-emitting devices 15 interfering with the unevenness of the object may cause some parts of the object to be underexposed. Image recognition captures the overall characteristics of an object, so even if some overexposure or underexposure occurs, there is a high probability of obtaining accurate recognition results. On the other hand, barcodes read black and white patterns, so if overexposure or underexposure occurs, there is a high probability of reading failure. Therefore, the light-emitting devices 15d, 15e, 15f, and 15g, which are positioned below the mounting base 11, are positioned so as not to face the readers 12a and 12b, which are positioned above the mounting base 11. This suppresses overexposure and underexposure of items by the readers 12a to 12f, allowing the readers 12a to 12f to properly read the barcodes of the items.

[0071] 7. Turntable As shown in Figure 8, the AI ​​scanner 1 is placed on the mounting base 11, or a circular turntable 16 is placed in place of the mounting base 11.

[0072] 8. Control Device As shown in Figures 9 and 10, the AI ​​scanner 1 has a control device 20 within the base unit 1a. The control device 20 has, for example, a housing that is roughly rectangular in shape and has hardware described later inside the housing. The control device 20 may be composed of, for example, a personal computer. The control device 20 may be attached externally to the AI ​​scanner 1.

[0073] Figure 11 is a hardware configuration diagram of the control device 20. The control device 20 includes a processor 21, a storage unit 22, a communication unit 23, an audio processing unit 24, and a bus 25. The processor 21, storage unit 22, communication unit 23, and audio processing unit 24 are connected via the bus 25.

[0074] The processor 21 controls the readers 12a to 12f, cameras 13a to 13c, displays 14a and 14b, and light-emitting devices 15a to 15g. The processor 21 is composed of, for example, a CPU (Central Processing Unit). The processor 21 realizes predetermined functions by executing programs stored in the memory unit 22.

[0075] The storage unit 22 is composed of, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), a ROM (Read Only Memory), flash memory, and RAM (Random Access Memory). The storage unit 22 stores the OS (Operating System) program to be executed by the processor 21 and application programs related to the self-checkout system.

[0076] The communication unit 23 is connected to readers 12a-12f, cameras 13a-13c, displays 14a, 14b, light-emitting devices 15a-15g, input devices such as touch panels or key input devices for displays 14a, 14b, a turntable 16, a hand scanner 2, a payment terminal 3, a POS device 4, a second light-emitting device such as an LED (Light Emitting Diode) (not shown in Figures 2-8), a human presence sensor, and a server (not shown in Figures 2-8) that controls the recognition system shown in Figure 1. In addition, terminal devices such as smartphones or tablets carried by store employees are wirelessly connected to the communication unit 23. The processor 21 controls the aforementioned devices via the communication unit 23.

[0077] The audio processing unit 24 is connected to an audio output device such as a speaker (not shown in Figures 2 to 8). The audio processing unit 24 converts the digital audio signal output from the processor 21 into an analog audio signal and outputs it to the audio output device. The audio output device may be a display 14a or 14b.

[0078] Furthermore, since the second item is estimated based on the images captured by each of the cameras 13a to 13c, different items may be estimated as the second item by each camera. In this case, the item with the highest recognition score, as described later, will be treated as the second item.

[0079] The motion sensor is positioned to easily detect a person standing in front of the AI ​​scanner 1. For example, the motion sensor is located on the back panel 1b or the top panel 1c. In the standby state, if the motion sensor does not detect a person for a certain period of time, the control device 20 puts the AI ​​scanner 1 into sleep mode. In the sleep state, if the motion sensor detects a person, the control device 20 puts the AI ​​scanner 1 into standby mode.

[0080] The second light-emitting device is positioned in a location easily visible to store staff, such as the top of the tabletop 1c. The second light-emitting device indicates the status of the AI ​​scanner 1 by the color of the light it emits. For example, blue indicates that the AI ​​scanner 1 is in standby mode. Green indicates that the AI ​​scanner 1 has recognized the barcode of an item. Yellow indicates that the AI ​​scanner 1 was unable to recognize the barcode of an item. Red indicates that the AI ​​scanner 1 is in sleep mode or learning mode (learning). Store staff can support customers, for example, by the color of the light emitted by the second light-emitting device. For example, if the light emitted by the second light-emitting device is yellow, a store staff member can instruct the customer to rearrange the items.

[0081] <Functions of the Control Device> Figure 12 is a functional block diagram of the control device 20. As shown in Figure 12, the control device 20 has an object recognition unit 31, a recognition judgment unit 32, an information output unit 33, and a learning unit 34. The functions of each unit shown in Figure 12 are realized, for example, by the processor 21 executing a program stored in the storage unit 22.

[0082] 1. Object Recognition Processing The object recognition unit 31 obtains the recognition result of the item placed on the mounting table 11 by performing object recognition processing using the images captured by cameras 13a to 13c. The object recognition unit 31 also obtains the barcode attached to the item from the object recognition result.

[0083] The object recognition unit 31 performs object recognition processing using images captured by cameras 13a to 13c when the object is located within a predetermined range on the mounting table 11, and does not perform object recognition processing using images captured by cameras 13a to 13c when the object is located outside the predetermined range on the mounting table 11, regardless of whether the image contains the object or not.

[0084] The object recognition unit 31 may, if the turntable 16 is connected to the AI ​​scanner 1 (control device 20) (when it detects the connection), photograph the items placed on the turntable using the cameras 13a to 13c. The information output unit 33, described later, may output the images captured by the cameras 13a to 13c to an external device such as a POS device 4.

[0085] If the turntable 16 is connected to the AI ​​scanner 1, the object recognition unit 31 may use cameras 13a to 13c to photograph the items placed on the turntable 16, extract only the item portion from the captured image, and perform object recognition processing on the items.

[0086] 2. Recognition and Judgment Processing The recognition and judgment unit 32 determines the item placed on the mounting table 11 based on at least one of the barcode read by readers 12a to 12f and the recognition result from the object recognition processing.

[0087] The recognition determination unit 32 may determine that the item indicated by the success is the item placed on the display stand 11 if only one of the two processes—reading the barcode by readers 12a to 12f or obtaining the recognition result of an item through object recognition processing—is successful.

[0088] The recognition and judgment unit 32 detects the position of an item placed on the mounting table 11, for example, by object recognition processing of cameras 13a to 13c. If the item is located within a predetermined range on the mounting table 11, the recognition and judgment unit 32 obtains the recognition result of the item from the object recognition processing. If the item is located outside the predetermined range on the mounting table 11, the recognition and judgment unit 32 does not need to obtain the recognition result from the object recognition processing, regardless of whether the item is included in the image or not. By specifying a predetermined range, the control device 20 eliminates unnecessary recognition and judgment processing, improving misrecognition and recognition processing speed.

[0089] If the recognition and determination unit 32 fails to read the barcode using readers 12a to 12f and to obtain the recognition result of the item through object recognition processing, it may display an image on the display 14a instructing the user to remove the item placed on the placement table 11 and place it again.

[0090] If the recognition determination unit 32 determines that an item recognized by the object recognition process cannot be paid for by the AI ​​scanner 1, it may display an image on the display 14a prompting the user to pay at a manned register.

[0091] 3. Information Output Processing The information output unit 33 acquires the results of the recognition process, determining the item indicated by the barcode read by readers 12a to 12f and the item indicated by the recognition result, and outputs the information indicating the item to an external device such as a POS device 4.

[0092] In the following, the item indicated by the barcode read by readers 12a to 12f may be referred to as the first item. The item indicated by the recognition result of object recognition processing using images from cameras 13a to 13c may be referred to as the second item.

[0093] Furthermore, since the second item is estimated based on the images captured by each of the cameras 13a to 13c, different items may be estimated as the second item by each camera. In this case, the item with the highest recognition score, as described later, will be treated as the second item.

[0094] The information output unit 33 may output item information to an external device such as a POS device 4 if the item indicated by the barcode read by readers 12a to 12f matches the item indicated by the recognition result from object recognition processing using images from cameras 13a to 13c. This allows the control device 20 to output more accurate item information.

[0095] The object recognition unit 31 may acquire a recognition score indicating the certainty of the recognition result of the object recognition process for an item. If the first item and the second item do not match, the information output unit 33 may output item information of the second item to an external device if the recognition score of the second item is above a predetermined threshold. If the recognition score of the second item is below a predetermined threshold, the information output unit 33 may output item information of the first item to an external device. In this way, if the first item and the second item do not match, the information output unit 33 outputs item information of the first or second item to an external device based on the recognition score of the second item, thereby enabling the output of more accurate item information. For example, a six-pack of beer has barcodes on each individual beer can and barcodes on the paper packaging that contains the six cans of beer. Readers 12a to 12f can acquire the barcodes on the beer cans and the barcodes on the packaging, while the object recognition process can recognize the six-pack of beer. Therefore, if the recognition score of the 6-pack obtained by the object recognition process is higher than the threshold, the information output unit 33 outputs the item information of the 6-pack obtained by the object recognition process to an external device. This allows the control device 20 to output more accurate item information.

[0096] The information output unit 33 may output information indicating the first item to an external device if the first item and the second item do not match, and as a result of the object recognition process, there is only one candidate for the first item. In other words, if the barcode read by readers 12a to 12f does not match the barcode acquired by the object recognition process, and there is only one barcode acquired by the object recognition process, the item information of the barcode read by readers 12a to 12f may be prioritized and output to the external device.

[0097] The information output unit 33 may output the item information of a second item to an external device if the barcode of an item read by readers 12a to 12f includes barcodes of multiple items on a single item. For example, when a 6-pack of beer is placed on the stand 11, readers 12a to 12f can read barcodes of multiple items (beer cans contained in the 6-pack) on a single item (the 6-pack), whereas object recognition processing can recognize the 6-pack of beer as a whole, rather than the individual beer cans contained within it. In such cases, the information output unit 33 may output the item information of a second item acquired by image recognition processing to an external device.

[0098] The information output unit 33 may output information indicating a second item to an external device if the barcode of an item read by readers 12a to 12f includes barcodes from multiple items. For example, as described above, a six-pack of beer contains multiple barcodes. When multiple barcodes are read by readers 12a to 12f, that is, when the barcode on one beer can and the barcode on the paper package containing the six beer cans are read, the information output unit 33 may output item information obtained through object recognition processing to an external device. This is because, in object recognition processing, when a six-pack of beer is placed on the platform 11, the six-pack, rather than individual beers, can be recognized.

[0099] The information output unit 33 may, if the barcode of an item read by readers 12a to 12f contains barcodes for multiple items and the item information for the second item is not acquired, notify the customer to move the item. For example, the information output unit 33 may notify the customer to move the item by voice or by displaying an image on the display 14a.

[0100] If the first item and the second item do not match, the information output unit 33 may display the first item and the second item on the display 14a and output the item information of the item selected by the customer from among the first item and the second item to an external device.

[0101] If the first item and the second item do not match, the information output unit 33 may notify an external device that the first item and the second item do not match. The external device here may be a display 14a, a terminal device held by a store employee, a second light-emitting device, or an audio output device.

[0102] If the first item and the second item do not match, the information output unit 33 may, in response to the customer's operation of the input device (for example, pressing a button on the touch panel on the display 14), notify the customer that they need the assistance of a store employee. For example, the information output unit 33 may output a control signal to control the lighting of the second light-emitting device, causing the second light-emitting device to emit light of a predetermined color. For example, the information output unit 33 may notify the terminal device held by the store employee that the customer needs the assistance of a store employee. In this case, the information output unit 33 may display images of the first item and the second item on the display of the terminal device held by the store employee. For example, the information output unit 33 may notify the customer that they need the assistance of a store employee by voice. Through these notifications, for example, the store employee can check whether there is a problem or whether any fraud has occurred, and can also remotely check the recognition results and the images from the object recognition process.

[0103] If the second item is included in a pre-set group of similar items (see Figure 19), the information output unit 33 may display the group of similar items including the second item on the display 14a and output the item information of the item selected by the customer from among the items included in the group of similar items including the second item to an external device.

[0104] The information output unit 33 may output the barcode attached to the item as item information to a registration device that registers the item as a processing target, such as a POS device 4.

[0105] When outputting item information for the first item, the information output unit 33 may output the barcode read by readers 12a to 12f, and when outputting item information for the second item, it may convert the recognition result of the object recognition process into a barcode and output it.

[0106] The information output unit 33 may notify the customer to move the item within the predetermined range if the item is located outside the predetermined range. For example, the information output unit 33 may notify the customer to move the item within the predetermined range by voice or by displaying an image on the display 14a. For example, the information output unit 33 may notify a terminal device or second light-emitting device held by the store clerk that the store clerk is instructing the customer to move the item within the predetermined range.

[0107] The information output unit 33 outputs the item information of the first item indicated by the barcode read by readers 12a to 12f if object recognition processing is not performed and readers 12a to 12f have read the barcode. Since the information output unit 33 outputs the item information of readers 12a to 12f even if the requirements for performing object recognition processing are not met, it appears to the customer (in terms of operation) that item recognition has been successfully performed normally (and quickly). In this case, it is not necessary to notify the customer of an error.

[0108] Figure 34 shows an example of item information output by the information output unit 33. The item identification ID is the JAN code of the recognized item. Image 1 is a registration image, which is used as the recognition target for the recognition engine. If the recognition target is an image taken from various angles and at different times, there will be multiple registration images. Image 2 is a display image, which is used to display the recognition result. The display image is an image selected from among the registration images, for example, an image that the recognition engine determined to contain characteristic parts of the item during recognition. Specifically, the registration image with the highest recognition score, which indicates the certainty of the item recognition result, is selected as the display image. Alternatively, the user may select the display image from among the registration images. The item information output by the information output unit 33 can also be used in the learning process described later. In this case, the registration images may be used as learning images for the recognition engine, and an image selected by the user from among them may also be recorded as a display image. When outputting the recognition results of the recognition engine trained in this manner, instead of selecting a display image from the registration images acquired during recognition, a recorded display image may be used to display the recognition results. The item name and price are the item name and price of the recognized item. The classification is information indicating the category to which the recognized item belongs. Examples of categories include "food" and "beverages," and may be categories arbitrarily set by the store operator, etc. Additional information includes information such as the number of items, product name, price, size, and weight. Note that information that overlaps with the item name and price may or may not be included in the additional information. Furthermore, the information output by the information output unit 33 only needs to include at least the item identification ID, image 1, and image 2, and whether or not to include other information is optional.

[0109] 4. The learning unit 34 uses the images captured by cameras 13a to 13c to train the recognition engine, which performs object recognition processing, as images of the items indicated by the barcodes. This learning can be performed either as part of the recognition process for the purpose of settling the payment for the items, or solely for the purpose of learning.

[0110] When learning is performed during the recognition process for the purpose of payment, if the reader 12a to 12f successfully reads the barcode but the object recognition process fails, the learning unit 34 may train the recognition engine that performs object recognition using the image as the image of the item indicated by the barcode. In other words, if the object recognition process fails, the learning unit 34 may learn the image from the camera 13a to 13c and the barcode that was successfully read. This improves the recognition process by the object recognition process. Furthermore, as a result, learning results are accumulated each time a customer pays for an item during the normal operation of a store, etc., so the accuracy of item recognition can be improved without the store operator having to perform any special processing.

[0111] Furthermore, when learning is performed during the recognition process for the purpose of payment, the item indicated by the successfully read barcode may be output as the recognition result. In this way, it will appear to customers and other users for payment purposes that the item has been successfully purchased, thus reducing customer anxiety.

[0112] If the information output unit 33 successfully reads the barcode using readers 12a to 12f but fails to perform recognition processing using object recognition processing, it may output item information corresponding to the barcode read by readers 12a to 12f.

[0113] Furthermore, if the purpose is solely learning, the turntable 16 may be placed in place of the mounting base 11, or on top of the mounting base 11, in order to train the system with images of items taken from various angles. The learning unit 34 may rotate the turntable 16 and read the barcode of the item placed on the rotating turntable 16 using readers 12a to 12f. The learning unit 34 may also take images of the item placed on the rotating turntable 16 from multiple angles using cameras 13a to 13c and acquire images. The learning unit 34 may train the recognition engine with images of the item taken from multiple angles as images of the item indicated by the barcode. In this case, the operation of the recognition system may be switched to a learning operation triggered by the installation of the turntable 16. Alternatively, the operation of the recognition system may be switched to an object recognition operation triggered by the removal of the turntable 16. Whether the turntable 16 has been installed or removed can be determined, for example, by detecting whether the turntable 16 is connected to the recognition system. Furthermore, when the turntable 16 is placed on the mounting base 11, a barcode or the like can be written on the turntable 16, and the determination can be made based on whether or not the barcode has been detected.

[0114] The information output unit 33 may output item information corresponding to the recognition result from the object recognition process, or item information corresponding to the barcode read by the readers 12a to 12c. When the control device 20 (AI scanner 1) is operating in recognition (item registration) mode, it may output information using the information output unit 33, and when it is operating in learning mode, it may perform learning using the learning unit 34. When the learning unit 34 is operating in recognition mode and detects that the turntable 16 has been connected, it may switch the operation of the control device 20 to learning mode. The learning unit 34 may perform learning using images obtained when performing item recognition in the object recognition process.

[0115] If the learning unit 34 is operating in recognition mode and detects that the turntable 16 has been connected, the control device 20 may switch to learning mode.

[0116] If the learning unit 34 is operating in learning mode and the connection to the turntable 16 is disconnected, the control device 20 may be switched to recognition mode. If the learning unit 34 detects that the connection to the turntable 16 has been disconnected and the mounting base 11 has been installed, the control device 20 may be switched to recognition mode.

[0117] The learning unit 34 may have a recognition engine, or another external device may have a recognition engine. The learning engine's functions may be realized by the processor 21 of the control device 20, or by an external device including a cloud. The learning unit 34 may instruct the recognition engine to learn.

[0118] The learning unit 34 may extract the portion of the images from cameras 13a to 13c in which an item is photographed, and train the recognition engine with the extracted image as the image of the item indicated by the barcode.

[0119] The above example of a learning scenario is just one example; learning can also be performed without using a turntable if the sole purpose is learning. Furthermore, learning may not be performed during customer checkout; instead, store staff or other personnel may replicate the checkout process for learning purposes.

[0120] Figure 13 is a flowchart illustrating the product learning process. A user, such as a store employee, places the item to be learned on the turntable 16. The user initiates the learning process using the input device. The learning unit 34 starts processing the flowchart in Figure 13 in response to the user's learning initiation operation.

[0121] The learning unit 34 starts rotating the turntable 16 (S1).

[0122] The learning unit 34 photographs the object placed on the turntable 16 each time the turntable 16 rotates by a certain angle (S2).

[0123] The learning unit 34 extracts an image of the product from the image taken in S2 (S3). In other words, it extracts an image of the product by removing the background portion included in the image. Removing the background portion can be achieved, for example, by taking the background difference between the image taken of the turntable 16 when nothing is placed on it and the image taken of the product portion.

[0124] The learning unit 34 stores the image cropped in S3 in the storage unit 22 (S4).

[0125] The learning unit 34 stops the rotation of the turntable 16 after it has completed one full rotation (S5).

[0126] The learning unit 34 determines whether or not an operation to photograph the product from a different orientation has been performed (S6). For example, if the user has a bag-shaped product with only two sides or a product with six sides, the user flips the top and bottom sides of the product over and places them on the turntable 16, and then performs an operation on the input device to photograph the product from a different orientation.

[0127] If the learning unit 34 performs an operation to photograph the product from a different orientation (Yes in S6), it proceeds to process S1.

[0128] If no operation is performed to photograph the product from another orientation (No. in S6), the learning unit 34 terminates the shooting (S7).

[0129] The learning unit 34 displays the image stored in the storage unit 22 in S4 on the display 14a (S8). This allows the user to confirm whether the product placed on the turntable 16 has been properly photographed.

[0130] The learning unit 34 determines whether or not the re-shooting operation has been performed (S9). For example, the user checks whether the barcode and product name attached to the product are clearly photographed, and if they are not clearly photographed, they perform the re-shooting operation.

[0131] If the learning unit 34 performs a reshoot operation (Yes in S9), it proceeds to process S1.

[0132] If the reshooting operation is not performed (No. in S9), the learning unit 34 performs additional product recognition registration (S10).

[0133] <User Interface> Figure 14 shows an image displayed on the display 14a. As shown in image A14a of Figure 14, the display 14a displays an image instructing the user to place the products one by one on the table (placement stand 11). This allows the customer to place the products one by one on the table (placement stand 11) according to image A14a displayed on the display 14a.

[0134] As shown in image A14a, display 14a displays an image prompting the customer to touch the "Pay" button on the POS display (e.g., display 14b) after scanning the items to be purchased. This allows the customer to easily complete the payment process for their items by following the image A14a displayed on display 14a once all the items to be purchased have been scanned.

[0135] Figure 15 shows an image displayed on the display 14a. As shown in image A15a of Figure 15, the display 14a displays an image of the product recognized by the AI ​​scanner 1. This allows customers to easily determine whether the product they wish to purchase has been scanned correctly. The product image may be an image pre-registered in the AI ​​scanner 1, or an image captured by cameras 13a to 13c.

[0136] Figure 16 shows an image displayed on the display 14a. If the first item and the second item do not match, the display 14a will show both the first and second items, as shown in image A16a of Figure 16, and a button will appear allowing the customer to select an item. The AI ​​scanner 1 outputs the item information of the item selected by the customer using the input device to an external device. In Figure 16, a "Select" button for selecting an item is displayed, but the item image itself may also serve as the button for selecting an item. That is, the customer may select the item to purchase by touching the item image displayed in image A16a.

[0137] Figure 17 shows an image displayed on the display 14a. If both the barcode reading by readers 12a to 12f and the acquisition of the item recognition result by object recognition processing fail, the display 14a will show an image instructing the user to remove the item placed on the stand 11 and place it again, as shown in image A17a of Figure 17. This allows the customer to, for example, replace an item that is not within a predetermined range (on the marker 11a) with an item that is within that range.

[0138] Furthermore, if the AI ​​scanner 1 displays image A17a a predetermined number of times (including once), that is, if it fails to acquire item information even after the customer replaces the product a predetermined number of times, it may display an alternative phase image such as "We are calling a store employee. Please wait."

[0139] Figure 18 shows an image displayed on the display 14a. If an item recognized by the object recognition process is determined to be an item that cannot be paid for by the AI ​​scanner 1, an image prompting the user to pay at a manned register will be displayed on the display 14a, as shown in image A18a of Figure 18.

[0140] Figure 19 shows an image displayed on the display 14a. When the second item is included in a pre-set group of similar items, the display 14a displays an image of the group of similar items including the second item, as shown in image A19a of Figure 19. The AI ​​scanner 1 outputs the item information of the item selected by the customer using the input device from among the items in the group of similar items displayed on the display 14a to an external device. Items for which a group of similar items is set may include, for example, fresh produce such as vegetables and fruits, or small items such as screws, which are difficult to assign barcodes to. For such items, it is not possible to obtain identification information using barcodes, and there are other candidates that have a similar appearance. Therefore, it is difficult to determine whether the item indicated by the recognition result by object recognition is the correct item or whether it is a misrecognition of another item. In this embodiment, such items that are easily misrecognized as other items are recorded as a group of similar items, and the user is asked to determine which item is the correct item.

[0141] <Summary of the First Embodiment> The AI ​​scanner 1 includes a mounting table 11, a plurality of cameras 13a to 13c installed at different angles to capture images of an item placed on the mounting table 11, and an object recognition unit 31 that recognizes the item placed on the mounting table 11 by object recognition processing using the images captured by the cameras 13a to 13c. As a result, the AI ​​scanner 1 can appropriately capture characteristic parts of an item.

[0142] The AI ​​scanner 1 acquires a first item indicated by a barcode read by readers 12a to 12f and a second item indicated by the recognition result from object recognition processing. Based on the first and second items, it determines information indicating the item placed on the display stand 11 and has an information output unit 33 that outputs this information to an external device. This allows the AI ​​scanner 1 to output appropriate item information to an external device.

[0143] The AI ​​scanner 1 uses the images captured by cameras 13a to 13c to train its recognition engine, which performs object recognition processing, as images of the items indicated by their barcodes. This makes it easy to link images with the barcodes of the items.

[0144] (Second Embodiment) In the second embodiment, the AI ​​scanner 1 is equipped with a reader that uses a different barcode reading method than the readers 12a to 12f described in the first embodiment. The differences from the first embodiment will be described below.

[0145] <Reader Arrangement> Figure 20 is a view of the top plate portion 1c according to the second embodiment, seen from below. As shown in Figure 20, the top plate portion 1c has a reader 17 in the center of its lower surface. The reader 17 is a tag reader that reads, for example, a tag (a code written on the tag) attached to an item by short-range wireless communication. The tag has, for example, the same code as a barcode that identifies the item written on it. Readers 12a to 12f read the barcode optically, while reader 17 reads the code (barcode value) by wireless communication. Hereinafter, readers 12a to 12f may be referred to as the first reader, and reader 17 may be referred to as the second reader.

[0146] In the second embodiment, the AI ​​scanner 1 comprehensively judges the object recognition result, the barcode reading result, and the tag reading result to determine the recognition result of the product placed on the display stand 11. Figure 35 shows an example of the rules for determining the output result.

[0147] This rule prioritizes the output results in the following order: object recognition results with scores exceeding a predetermined threshold > barcode reading results > tag reading results > object recognition results with scores below a predetermined threshold. The reason for this is as follows:

[0148] First, if the object recognition score is sufficiently high, it is highly likely that the system can correctly recognize products even if they lack barcodes or have incorrectly attached tags. Therefore, it can be assumed that the recognition results obtained are more accurate than those indicated by barcode or tag readings.

[0149] Furthermore, barcodes are often printed on items and are usually inseparable, making it less likely that they will be misassociated with items. However, tags are usually attached to items with stickers or similar adhesives, increasing the possibility of misassociation due to incorrect attachment. Therefore, it can be assumed that barcode reading results provide more accurate recognition results than tag reading results.

[0150] Furthermore, a low object recognition score suggests a high probability of misrecognition, and if other results are available, they can be presumed to be more reliable.

[0151] <Operation of the Control Device> Figure 21 is a flowchart showing an example of the operation of the control device 20 according to the second embodiment. The control device 20 displays, for example, the image A14a shown in Figure 14 on the display 14a. When a customer places a product on the display stand 11 according to the contents of image A14a, the control device 20 executes the process shown in the flowchart of Figure 21.

[0152] The control device 20 receives the barcode of the product read by the first reader from the first reader (S21a). The control device 20 receives the images taken by cameras 13a to 13c from cameras 13a to 13c (S21b). The control device 20 receives the barcode read by the second reader from the second reader (S21c).

[0153] The control device 20 detects an object from the image received in S21b (S22).

[0154] The control device 20 determines whether the outer circumference of the object detected in S22 is within a predetermined range (S23).

[0155] If the control device 20 determines in S23 that the outer circumference of the object is not within a predetermined range (No. in S23), it displays an image on the display 14a instructing the product to be repositioned (S24). For example, the control device 20 displays the image A17a shown in Figure 17 on the display 14a.

[0156] On the other hand, if the control device 20 determines in S23 that the outer circumference of the object is within a predetermined range (Yes in S23), it determines in S22 whether or not the center of the object detected is within a predetermined range (S25). Note that the predetermined range in S24 is within the predetermined range in S23, but smaller than the predetermined range in S23.

[0157] If the control device 20 determines in S25 that the center of the object is not within a predetermined range (No. in S25), it proceeds to S24.

[0158] On the other hand, if the control device 20 determines in S25 that the center of the object is within a predetermined range (Yes in S25), it obtains the recognition result of the item by object recognition processing using the received image in S21b (S26).

[0159] The control device 20 calculates a recognition score (S27) that indicates the likelihood of the item recognition result obtained by the object recognition process in S26.

[0160] The control device 20 acquires the barcode of the product placed on the display stand 11 based on the recognition score calculated in S27 (S28). For example, if the recognition score is above a predetermined value, the control device 20 acquires the barcode of the product placed on the display stand 11.

[0161] After processing in S21a, S21c, and S28, the control device 20 executes an integrated determination process (S29). That is, the control device 20 can obtain barcodes acquired using the first reader (S21a), barcodes acquired using object recognition processing (S28), and barcodes acquired using the second reader (S21b). Then, the control device 20 moves on to the integrated determination process. The integrated determination process will be described in detail using the flowcharts in Figures 22 and 23.

[0162] Figures 22 and 23 are flowcharts of the integrated judgment process in Figure 21. In the flowcharts shown in Figures 22 and 23, the process continues at "1" circled in the figures.

[0163] The control device 20 determines whether there is a difference between the three results (S31). That is, the control device 20 determines whether there is a difference between the barcode of the product acquired by the first reader, the barcode of the product acquired using object recognition processing, and the barcode of the product acquired by the second reader.

[0164] If the control device 20 determines in S31 that there is no difference between the three results (No. in S31), it adopts the result (barcode) of the first reader (S32). In other words, if the barcode of the product acquired by the first reader, the barcode of the product acquired using object recognition processing, and the barcode of the product acquired by the second reader are the same, the control device 20 adopts the barcode of the first reader. Note that since the three results are the same, the control device 20 may also adopt the barcode of the product acquired using object recognition processing or the barcode of the product acquired by the second reader. In other words, the control device 20 may adopt any one of the three results.

[0165] On the other hand, if the control device 20 determines in S31 that there is a difference between the three results (Yes in S31), it determines whether or not the barcode of the first reader is included in the acquired barcode (the barcode acquired in the flowchart process in Figure 21) (S33).

[0166] If the control device 20 determines in S33 that the acquired barcode includes the barcode of the first reader (Yes in S33), it determines whether or not multiple types of barcodes are included (S34). In other words, the control device 20 determines whether or not a product that may contain multiple types of barcodes, such as a 6-pack of beer, has been placed on the display stand 11.

[0167] If the control device 20 determines in S34 that multiple types of barcodes are included (Yes in S34), it determines whether or not the barcode of the second reader is included among the acquired barcodes (S35).

[0168] In S35, if the control device 20 determines that the acquired barcode includes a barcode from the second reader (Yes in S35), it adopts the result (barcode) from the second reader (S36). In other words, when a product that may contain multiple types of barcodes, such as a 6-pack of beer, is placed on the display stand 11, the control device 20 adopts the barcode from the second reader if the acquired barcode includes a barcode from the second reader.

[0169] Here, the barcode obtained by the second reader is based on a tag attached to the product. Therefore, the barcode on the tag can represent the barcode of a product that combines multiple items into one, such as a six-pack of beer. On the other hand, the barcode obtained by the first reader may include multiple barcodes (for example, the barcode on the six-pack and the barcodes on the beer cans inside the six-pack). Furthermore, it is assumed that the second reader, which obtains barcodes via wireless communication, has a higher barcode accuracy rate than the object recognition process. Therefore, if the obtained barcode includes multiple types of barcodes, the control device 20 can appropriately select the barcode of a single product that combines multiple items (for example, the barcode of the six-pack in the case of a six-pack of beer) by adopting the barcode of the second reader instead of the barcodes of the first reader and the object recognition process.

[0170] If the control device 20 determines in S35 that the acquired barcode does not contain a barcode from the second reader (No. in S35), it determines whether the acquired barcode contains a barcode for object recognition processing (S37).

[0171] In S37, the control device 20 determines that the acquired barcodes include a barcode for object recognition processing (Yes in S37), and then adopts the result of the object recognition processing (barcode) (S38). In other words, when a product that may contain multiple types of barcodes, such as a 6-pack of beer, is placed on the display stand 11, the control device 20 adopts the barcode for object recognition processing if the acquired barcodes do not include a barcode from the second reader but do include a barcode for object recognition processing.

[0172] In the object recognition process, since products are recognized by images, it is possible to recognize products that are bundled together and obtain the barcode of such bundled products. On the other hand, the barcode obtained by the first reader may contain multiple barcodes. Therefore, if the obtained barcode contains multiple types of barcodes and does not include the barcode from the second reader, the control device 20 can appropriately select the barcode of a single bundled product (for example, the barcode of a 6-pack of beer) by adopting the barcode from the object recognition process instead of the barcode from the first reader.

[0173] If the control device 20 determines in S37 that the acquired barcodes do not include a barcode for object recognition processing (No. in S37), it adopts the multiple results (barcodes) from the first reader (S39).

[0174] In the process of S33, if the control device 20 determines that the acquired barcode does not contain a barcode from the first reader (No. in S33), it determines whether or not the acquired barcode contains a barcode from the second reader (S40).

[0175] If the control device 20 determines in S40 that the acquired barcode includes a barcode from the second reader (Yes in S40), it adopts the result (barcode) from the second reader (S36).

[0176] In this case, if both the first and second barcodes are recognized by the first reader, the recognition result from the first reader is given priority. The reason for this decision is as follows: Barcodes are generally printed integrally on product packaging, so they are generally unlikely to be attached to the wrong product. In contrast, tags are often attached to product packaging with stickers or other means in a removable manner, so they may be attached to the wrong product. Therefore, if the acquired barcode does not include a barcode from the first reader, it is determined whether a barcode from the second reader is included as a complement, and if a barcode from the second reader is included, the barcode from the second reader is adopted.

[0177] If the control device 20 determines in S40 that the acquired barcode does not contain a barcode from the second reader (No. in S40), it determines in S41 whether or not the acquired barcode contains a barcode for object recognition processing.

[0178] If the control device 20 determines in S41 that the barcode for object recognition processing is included (Yes in S41), it adopts the barcode for object recognition processing (S38).

[0179] On the other hand, if the control device 20 determines in S41 that the barcode for object recognition processing is not included (No. in S41), it determines that the product cannot be recognized (S42).

[0180] As shown in Figure 23, the control device 20 displays images on the display 14a according to the results of S39, S32, S36, S38, and S42 (see Figure 22).

[0181] For example, if the control device 20 selects multiple results (barcodes) from the first reader in S39, it displays images of the products indicated by the recognized barcodes on the display 14a. For example, the control device 20 displays image A16a shown in Figure 16 on the display 14a.

[0182] For example, if the control device 20 adopts the result of the first reader in S32, adopts the result of the second reader in S36, or adopts the result of the object recognition process in S38, it displays an image of the product indicated by the adopted barcode on the display 14a. For example, the control device 20 displays the image A15a shown in Figure 15 on the display 14a.

[0183] For example, if the control device 20 accepts the result of the object recognition process in S38, it displays the image of the product indicated by the accepted barcode on the display 14a. For example, the control device 20 displays the image A19a shown in Figure 19 on the display 14a.

[0184] For example, if the control device 20 determines in S39 that the product cannot be recognized, it displays an image on the display 14a instructing the user to read the barcode with a hand scanner.

[0185] For example, if the control device 20 determines in S39 that it cannot recognize the product, it displays an image on the display 14a instructing the device to remove the product placed on the stand 11 and place it again. For example, the control device 20 displays the image A17a shown in Figure 17 on the display 14a.

[0186] For example, if the control device 20 determines in S39 that it cannot recognize the product, it displays an image on the display 14a instructing the user to use a manned register. For example, the control device 20 displays the image A18a shown in Figure 18 on the display 14a.

[0187] After the image is displayed in S43, the control device 20 performs an action in accordance with the customer's operation (S44).

[0188] For example, if image A15a shown in Figure 15 is displayed on the display 14a, the customer checks whether the displayed product is appropriate. If the product on the display stand 11 is appropriate, the customer removes the product from the display stand 11 and places the next product on the display stand 11. The control device 20 outputs the item information of the product removed from the display stand 11 to an external device and terminates the integrated judgment process shown in Figures 22 and 23. Then, the control device 20 returns to the flowchart process shown in Figure 21 and obtains the item information of the next product.

[0189] For example, if image A16a shown in Figure 16 is displayed on the display 14a, the customer selects a product. The control device 20 outputs the product information of the selected product to an external device. The control device 20 then completes the integrated judgment process shown in Figures 22 and 23, returns to the flowchart process shown in Figure 21, and obtains the product information of the next product.

[0190] For example, if image A17a shown in Figure 17 is displayed on the display 14a, the customer removes the product placed on the display stand 11 and places it back on the display stand 11. The control device 20 terminates the integrated judgment process shown in Figures 22 and 23, returns to the flowchart process shown in Figure 21, and obtains the item information of the replaced product.

[0191] For example, if image A19a shown in Figure 19 is displayed on the display 14a, the customer selects a product. The control device 20 outputs the product information of the selected product to an external device. The control device 20 then completes the integrated judgment process shown in Figures 22 and 23, returns to the flowchart process shown in Figure 21, and obtains the product information of the next product.

[0192] After processing in S39, S32, S36, S38, and S42 (see Figure 22), the control device 20 determines whether the barcode of the first reader or the second reader is the same as the barcode of the object recognition process (S45).

[0193] If the control device 20 determines that the barcode of the first or second reader is the same as the barcode of the object recognition process (Yes in S45), it terminates the integrated determination process shown in Figures 22 and 23 and returns to the flowchart process shown in Figure 21.

[0194] On the other hand, if the barcode of the first or second reader and the barcode of the object recognition process are not the same (No. in S45), the control device 20 executes a learning process (S46). That is, even if object recognition fails during normal operation, if the barcode reading by the first or second reader is successful, the control device 20 proceeds with the process as if the barcode had been recognized normally, and performs additional learning using the image at the time of settlement and the barcode of the first or second reader. The control device 20 finishes the integrated judgment process shown in Figures 22 and 23 and returns to the flowchart process shown in Figure 21.

[0195] <Summary of the second embodiment> The AI ​​scanner 1 can output appropriate item information to an external device using barcodes acquired by the first and second readers of different types. The AI ​​scanner 1 can output appropriate item information to an external device through the integrated judgment process described above.

[0196] (Third Embodiment) In the third embodiment, the field of view of the leaders 12a to 12f (first leaders), cameras 13a to 13c, and leader 17 (second leader) will be described.

[0197] <Field of View of the First Reader> Figure 24 shows the field of view of readers 12a and 12b when the AI ​​scanner 1 is viewed from the front. Figure 25 shows the field of view of readers 12a and 12b when the AI ​​scanner 1 is viewed from the front right diagonal direction. Figure 26 shows the field of view of readers 12c to 12f when the AI ​​scanner 1 is viewed from the front right upper diagonal direction.

[0198] As shown in Figures 24 and 25, the field of view A24a of leader 12a has a conical shape with leader 12a as its apex and its base widening from the upper left rear of the mounting base 11 toward the center of the mounting base 11. As shown in Figures 24 and 25, the field of view A24b of leader 12b has a conical shape with leader 12b as its apex and its base widening from the upper right rear of the mounting base 11 toward the center of the mounting base 11.

[0199] As shown in Figure 26, the field of view A26a of leader 12c has a conical shape with leader 12c as its apex, and its base widening from the lower left front side of the mounting platform 11 toward the center of the mounting platform 11. The field of view A26b of leader 12d has a conical shape with leader 12d as its apex, and its base widening from the lower right front side of the mounting platform 11 toward the center of the mounting platform 11. The field of view A26c of leader 12e has a conical shape with leader 12e as its apex, and its base widening from the front center of the mounting platform 11 toward the center of the mounting platform 11. The field of view A26d of leader 12f has a conical shape with leader 12f as its apex, and its base widening from the rear center of the mounting platform 11 toward the center of the mounting platform 11.

[0200] The above field of view can cover the entire surface (all directions) of the item placed on the mounting platform 11. Therefore, regardless of the orientation of the item placed on the mounting platform 11, readers 12a to 12f can read the barcode of the item placed on the mounting platform 11.

[0201] <Camera Field of View> Figure 27 shows the field of view of camera 13a when the AI ​​scanner 1 is viewed from the front. Figure 28 shows the field of view of camera 13b when the AI ​​scanner 1 is viewed from the front. Figure 29 shows the field of view of camera 13c when the AI ​​scanner 1 is viewed from the front. Figure 30 shows the field of view of cameras 13a to 13c when the AI ​​scanner 1 is viewed from the front. Figure 31 shows the field of view of cameras 13a to 13c when the AI ​​scanner 1 is viewed from the right side.

[0202] As shown in Figures 27, 30, and 31, the field of view A27a of camera 13a has a conical shape with camera 13a as its apex and its base widening from the upper left of the mounting base 11 toward the center of the mounting base 11. As shown in Figures 28, 30, and 31, the field of view A28a of camera 13b has a conical shape with camera 13b as its apex and its base widening from the upper right of the mounting base 11 toward the center of the mounting base 11. As shown in Figures 29, 30, and 31, the field of view A29a of camera 13c has a conical shape with camera 13c as its apex and its base widening from the upper center of the mounting base 11 toward the center of the mounting base 11.

[0203] The above field of view can cover the entire upper surface (all directions except the bottom surface) of the item placed on the mounting platform 11. Therefore, regardless of the orientation of the item placed on the mounting platform 11, the control device 20 can read the barcode of the item placed on the mounting platform 11 from the images captured by cameras 13a to 13c.

[0204] Furthermore, when handling a variety of items, it is difficult to predict whether the angle of an item placed on the mounting platform includes any distinctive features. Therefore, in this embodiment, cameras 13a and 13b are configured to photograph the mounting platform from an oblique direction. This allows the distinctive features to be included within the field of view of camera 13a or camera 13b, even if the side of the item has distinctive features.

[0205] Furthermore, with cameras 13a and 13b, if a tall object is placed on the mounting platform, the highest part of the object may not be included in the field of view. Therefore, this embodiment includes a camera 13c that photographs the object from above. This allows the object to be recognized even if a characteristic part of the object is located at a height outside the field of view of cameras 13a and 13b. Note that since camera 13c is a camera for capturing tall objects in the field of view, camera 13c may be omitted if only short objects are being handled.

[0206] Furthermore, in this embodiment, the camera is not positioned on the underside of the mounting base 11. This is because, when a user places an item on the mounting base, it is unlikely that they would intentionally position it with its distinctive features facing downwards. However, the camera may also be positioned on the underside of the mounting base 11 so that the camera's field of view covers the entire surface of the item.

[0207] Alternatively, a wide-angle camera such as a fisheye camera may be used as camera 13 to cover the entire field of view with a single camera. However, when performing object recognition, there is a risk that information may be lost or incorrect corrections may be made due to image distortion correction. Therefore, rather than using a camera with a very wide field of view as camera 13, it is better to use multiple cameras with relatively narrow fields of view so that the combined field of view of those cameras covers the entire upper surface of the object, thereby improving the accuracy of object recognition.

[0208] Furthermore, the field of view of cameras 13a to 13c does not need to cover the entire upper surface of the object with respect to the entire mounting platform 11. As described above, object recognition is performed within a predetermined range on the mounting platform 11, so it is sufficient that the field of view of cameras 13a to 13c is configured to cover at least the entire upper surface (all directions excluding the bottom surface) of the object within this range.

[0209] Even if the barcode printed on the item is placed in contact with the surface of the mounting table 11, cameras 13a to 13c can still capture characteristic display parts of the item, such as the product name or images of its contents. Therefore, the control device 20 can obtain the barcode of the item placed on the mounting table 11 from the images captured by cameras 13a to 13c (see the description of "4. Camera" in the first embodiment).

[0210] <Field of View of the Second Reader> Figure 32 shows the field of view of the second reader (reader 17) when the AI ​​scanner 1 is viewed from the front. As shown in Figure 32, the field of view A32a of the reader 17 has a conical shape with the reader 17 as its apex, and its base widening from approximately above the center of the mounting table 11 toward approximately the center of the mounting table 11.

[0211] The above field of view allows the reader 17 to receive radio waves from tags attached to items placed on the mounting platform 11. Therefore, regardless of the orientation of the item when it is placed on the mounting platform 11, the reader 17 can read the barcode of the item placed on the mounting platform 11.

[0212] <Summary of the third embodiment> The AI ​​scanner 1 can appropriately acquire item information of an item placed on the mounting table 11 based on the field of view of the first reader, the field of view of the camera, and the field of view of the second reader as described above.

[0213] <Modifications> The embodiments have been described above. Modifications of the above embodiments will be described below.

[0214] In the embodiment described above, a configuration was described in which a total of three cameras 13 for capturing images are mounted on the left, right, and center of the mounting platform. However, the number of cameras can be more or fewer. In this embodiment, the cameras are positioned so as to capture any object placed on the mounting platform without creating blind spots except for the bottom surface. Therefore, if the field of view of the cameras sufficiently includes the mounting platform and the space above the mounting platform, two cameras, one on the left and one on the right, may suffice. However, if the object is tall, the top of the item may fall outside the field of view of either the left or right camera. Therefore, if it is necessary to handle tall objects, a central camera may also be installed. Furthermore, the number of cameras can be reduced by using a fisheye lens. However, with a camera using a fisheye lens, image distortion correction is required before object recognition. Therefore, characteristic information of the object may be lost or incorrectly corrected during the correction process. Consequently, in order to ensure the accuracy of object recognition, it is better to arrange multiple cameras with sufficiently narrow fields of view at different angles.

[0215] In the embodiment described above, a configuration was described in which the camera is placed only above the mounting base 11. However, a camera may also be placed below the mounting base 11. This allows the object to be recognized by image recognition even when its distinctive parts are facing downwards.

[0216] In the embodiment described above, the marker 11a was configured as a substantially transparent plate. However, the marker 11a may be a mark or the like printed on the mounting base 11. The mark or the like may be colored, but it is preferable that it be transparent so as not to interfere with the reading of barcodes, etc.

[0217] In the embodiment described above, the mounting platform 11 was described as being transparent. However, if it is not necessary to read barcodes or the like written on the underside of the item placed on the mounting platform 11, an opaque mounting platform 11 may be used.

[0218] In the above-described embodiment, the POS device 4 was located outside the AI ​​scanner 1. However, the POS device 4 may be housed inside the AI ​​scanner 1. For example, it may be housed in the housing portion shown in Figures 9 and 10 together with the control device of the AI ​​scanner 1, or in place of the control device. In the latter case, the control device may be housed in another part of the AI ​​scanner 1. In this case, in order to reduce the size of the POS device 4 to be housed, the display and barcode scanner of the POS device 4 may be omitted. In the above-described embodiment, since the AI ​​scanner 1 outputs the recognition results of items in the same format and interface as the barcode scanner, the POS device 4 can be used in place of the barcode scanner by connecting it to the AI ​​scanner 1. In addition, the output of the POS device 4 and the display of the AI ​​scanner 1 may be connected inside the housing in which the POS device 4 is housed so that the output of the POS device 4 is displayed on the display of the AI ​​scanner 1. For example, display 14b may be used as a display to show the output of the POS device 4.

[0219] In the embodiment described above, the predetermined area where the item is to be placed is assumed to be approximately in the center of the mounting table 11, but it may be in another position. In this case, the camera's field of view should be set to match the position of the predetermined area. In the embodiment described above, the barcode reader's field of view is set to cover the entire mounting table 11, so no special settings are required regardless of whether the predetermined area is in the center or not. However, if necessary to improve reading accuracy, the barcode reader's field of view may also be set to match the position of the predetermined area. Even if the predetermined area is not approximately in the center of the mounting table 11, if it is narrower than the entire surface of the mounting table 11, the processing load for object recognition can be reduced by limiting the area in which object recognition is performed to the predetermined area.

[0220] In the embodiments described above, an example of applying the recognition system to a self-checkout system was explained. However, the recognition system of this disclosure can also be applied to inventory management in warehouses, etc. When the recognition system is applied to inventory management in warehouses, etc., instead of a POS device, an inventory management device may be connected to the AI ​​scanner 1, or placed inside the base portion 1a of the AI ​​scanner 1, or placed inside the housing of the control device 20. The inventory management device may be a device that manages or registers inventory of goods, etc., and may be called a registration device.

[0221] In the first embodiment described above, the AI ​​scanner 1 is equipped with readers 12a to 12f that optically read barcodes. However, instead of readers 12a to 12f, it may be equipped with a reader that reads barcodes wirelessly, such as a tag reader. In this case, the reader that reads barcodes may be the reader 17 located on the top plate portion 1c, as shown in Figure 20.

[0222] In the second embodiment described above, an integrated barcode determination process involving three steps—barcode reading by the first reader, barcode reading by the second reader, and barcode acquisition by object recognition processing—was explained, but the invention is not limited thereto. The control device 20 may execute two of the integrated barcode determination processes among barcode reading by the first reader, barcode reading by the second reader, and barcode acquisition by object recognition processing.

[0223] While embodiments have been described above with reference to the drawings, this disclosure is not limited to such examples. It will be apparent to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims. Such modifications or alterations are also understood to fall within the technical scope of this disclosure. Furthermore, the components in the embodiments may be combined in any way without departing from the spirit of this disclosure.

[0224] In the embodiments described above, the notation "...part" used for each component may be replaced with other notations such as "...circuitry," "...assembly," "...device," "...unit," or "...module."

[0225] This disclosure can be implemented in software, hardware, or software in conjunction with hardware. Each functional block used in the description of the above embodiments may be implemented in part or in whole as an integrated circuit (LSI), and each process described in the above embodiments may be controlled in part or in whole by a single LSI or a combination of LSIs. An LSI may consist of individual chips, or it may consist of a single chip that includes some or all of the functional blocks. An LSI may have data inputs and outputs. Depending on the degree of integration, LSIs may be referred to as ICs, system LSIs, super LSIs, or ultra LSIs.

[0226] The integrated circuit implementation method is not limited to LSIs; it may also be implemented using dedicated circuits, general-purpose processors, or dedicated processors. Furthermore, a Field Programmable Gate Array (FPGA) that can be programmed after LSI manufacturing, or a reconfigurable processor that allows for the reconfiguration of the connections and settings of circuit cells within the LSI, may also be used. This disclosure may be implemented as digital or analog processing.

[0227] Furthermore, if advancements in semiconductor technology or related technologies lead to the emergence of integrated circuit technologies that can replace LSIs, then naturally, these technologies can be used to integrate functional blocks. The application of biotechnology, for example, is a possibility.

[0228] Analysis 1 As described in Patent Document 1, the technology for recognizing items based on image features is useful when it is difficult to determine the location of barcodes or other labels attached to items. However, the features of an item do not necessarily appear when the item is photographed at a specific angle. For example, it is difficult to distinguish items such as bottles from images taken from directly above. Furthermore, in environments where a wide variety of items are handled, such as retail stores, the characteristic parts differ for each item, making it difficult to adjust the angle at which items should be photographed. Therefore, there is a need for a recognition device that can appropriately photograph the characteristic parts of an item.

[0229] Furthermore, the need to appropriately photograph the distinctive features of items is not limited to retail store systems such as self-checkout counters. For example, it is used in warehouses and factories to register information about items such as parts and work-in-progress, and in distribution sites to register information about packages to manage goods. In these cases as well, there is a need to be able to appropriately photograph the distinctive features of items in order to recognize them based on images.

[0230] Non-limiting embodiments of this disclosure contribute to the provision of recognition devices, recognition methods, and programs that can appropriately capture characteristic parts when recognizing various articles by image.

[0231] <Note 1> A recognition device comprising: a mounting platform; a plurality of cameras installed at different angles to capture images of an item placed on the mounting platform; and an object recognition unit that recognizes the item placed on the mounting platform by object recognition processing using the images captured by the cameras.

[0232] <Note 2> The recognition device as described in Note 1, wherein the multiple cameras are arranged such that, when the multiple angles of view are combined, no blind spots are created on the upper surface of a predetermined area where the item is to be placed.

[0233] <Note 3> The recognition device according to Note 2, wherein at least two of the multiple cameras are set above the base described above, and the at least two cameras have angles of view that capture the predetermined range from different oblique directions.

[0234] <Note 4> The recognition device according to Note 3, wherein the plurality of cameras further include a camera having a field of view that captures the predetermined range substantially vertically.

[0235] <Note 5> The recognition device further comprises a detection unit for detecting the position of an article placed on the stand described above, the object recognition unit obtains the recognition result from the object recognition process when the article is located within the predetermined range of the stand described above, and does not obtain the recognition result from the object recognition process when the article is located outside the predetermined range of the stand described above, regardless of whether the image contains the article or not, the recognition device as described in Note 2.

[0236] <Note 6> The recognition device further comprises one or more readers for reading identification information of an article placed on the stand described above, wherein the one or more readers read the identification information regardless of whether the article is located within the predetermined range or not, as described in Note 5.

[0237] <Note 7> The recognition device according to Note 6, comprising: a recognition determination unit that determines an article placed on the stand described above based on at least one of the identification information read by the one or more readers and the recognition result obtained by the object recognition process.

[0238] <Note 8> The recognition device according to Note 7, wherein the mounting base is substantially transparent, and the one or more readers include a plurality of readers installed at positions opposite each other across the mounting base described above.

[0239] <Note 9> The recognition device described in Note 8, wherein the mounting base is provided with a structure indicating the predetermined range.

[0240] <Note 10> The recognition device described in Note 9, wherein the structure indicating the predetermined range is a transparent plate or a substantially transparent marker attached to the stand described above.

[0241] <Note 11> The recognition device according to Note 8, wherein the recognition device comprises a light-emitting device that emits light toward the aforementioned base from above and below the base, and the light-emitting device is positioned on either side of the aforementioned base, not facing the reader.

[0242] <Note 12> The recognition device as described in Note 7, wherein if only one of the following is successful, the recognition determination unit determines that the item indicated by the successful one is the item placed on the stand described above.

[0243] <Note 13> The recognition device described in Note 6, which operates by switching between a recognition mode in which the object recognition unit recognizes the article, a shooting mode in which the object recognition unit does not recognize the article but photographs the article, and a learning mode in which the recognition engine that performs the object recognition processing learns the article contained in the image.

[0244] <Note 14> The recognition device described in Note 13, which operates in the learning mode when it detects that the mounting base has been replaced with a rotating base.

[0245] <Note 15> If the reader successfully reads the identification information, and the object recognition unit fails to obtain the recognition result of the article, the recognition device operates in the learning mode and causes the object recognition engine that performs the object recognition processing to learn the image as an image of the article indicated by the identification information, as described in Note 13.

[0246] <Note 16> The recognition device described in Note 15, wherein if the recognition device succeeds in reading the identification information by the reader and fails to obtain the recognition result of the article by the object recognition unit, it outputs the article indicated by the reading result of the identification information by the reader as the recognition result of the article.

[0247] <Note 17> The recognition device according to Note 1, further comprising a first display device positioned behind the mounting base and displaying information indicating an item recognized by the object recognition unit.

[0248] <Note 18> The recognition device described in Note 1 further comprises a camera that monitors the user's operation on the mounting platform.

[0249] <Note 19> The recognition device described in Note 1, wherein the recognition result of an item placed on the aforementioned stand is connected to a barcode reader and is output to a terminal that manages the sale of the item in the same format as the output of the barcode reader.

[0250] <Note 20> The recognition device according to Note 19, further comprising a storage unit capable of storing the terminal and a second display device for displaying the output of the terminal stored in the storage unit.

[0251] <Note 21> A recognition device comprising a mounting platform and a plurality of cameras installed at different angles to capture images of an item placed on the mounting platform, wherein the recognition method involves recognizing an item placed on the mounting platform by object recognition processing using images captured by the cameras.

[0252] <Note 22> A program that causes a computer, which includes a mounting platform and a plurality of cameras installed at different angles to take images of an item placed on the mounting platform, to perform an object recognition process using the images taken by the cameras to recognize the item placed on the mounting platform.

[0253] Analysis 2: There are cases where the device cannot properly read identification information such as barcodes attached to items. For example, customers unfamiliar with self-checkout systems may not be able to properly get the barcode reader to read it. A recognition device is desired that can properly acquire identification information of items and output information indicating the items to an external device such as a POS system, even when operated by customers unfamiliar with self-checkout systems.

[0254] Furthermore, the need to appropriately acquire information identifying items is not limited to retail store systems such as self-checkout counters. For example, information about items such as parts and work-in-progress is registered to manage items in warehouses and factories, and information about packages is registered to manage items such as packages in distribution sites. In such cases, inexperienced workers may have difficulty getting barcodes to be read correctly by a reader, which can lead to delays in the work. Therefore, there is a need for recognition devices that can appropriately output information identifying items to external devices in a variety of situations.

[0255] Non-limiting embodiments of this disclosure contribute to the provision of recognition devices, information output methods, programs, and recognition systems that can appropriately output information indicating an article.

[0256] <Note 1> Recognition device comprising: a mounting platform; a reader for reading identification information attached to an article placed on the mounting platform; a camera for taking an image of the article placed on the mounting platform; an object recognition unit for acquiring the recognition result of the article by object recognition processing using the image taken by the camera; and an output unit for acquiring a first article indicated by the identification information read by the reader and a second article indicated by the recognition result of the object recognition processing, determining information indicating the article placed on the mounting platform based on the first and second articles, and outputting it to an external device.

[0257] <Note 2> The recognition device as described in Note 1, wherein the output unit outputs information indicating the first item to an external device when the identification information read by the reader matches the second item indicated by the recognition result obtained by the object recognition process.

[0258] <Note 3> The recognition device as described in Note 1, wherein the object recognition unit further acquires a recognition score indicating the certainty of the recognition result of the object recognition process for the article, and the output unit, if the first article and the second article do not match, outputs information indicating the second article to an external device if the recognition score of the second article is above a predetermined threshold, and outputs information indicating the first article to an external device if the recognition score of the second article is below a predetermined threshold.

[0259] <Note 4> The recognition device as described in Note 1, wherein the output unit outputs information indicating the first item to the external device when the first item and the second item do not match and the object recognition process results in only one candidate for the first item.

[0260] <Note 5> The recognition device described in Note 4, wherein the output unit outputs information indicating the second item to the external device when the identification information of an item read by the reader includes identification information for multiple items in a single item.

[0261] <Note 6> The recognition device described in Note 4, wherein the output unit outputs information indicating the second item to the external device when the identification information of an item read by the reader includes identification information for multiple items.

[0262] <Note 7> The recognition device described in Note 4, wherein the output unit notifies the customer to move the item if the identification information of the item read by the reader contains identification information for multiple items and information indicating the second item cannot be obtained.

[0263] <Note 8> The recognition device further comprises a display unit with an input function, and the output unit, when the first item and the second item do not match, displays the first item and the second item on the display unit and outputs information indicating the item selected by the user from among the first item and the second item to an external device, as described in Note 1.

[0264] <Note 9> The recognition device described in Note 1, wherein the output unit notifies an external device that the first article and the second article do not match when the first article and the second article do not match.

[0265] <Note 10> The output unit is the recognition device described in Note 9, which, when a customer presses a call button on a display unit equipped with an input, notifies the customer that they need the assistance of a store employee.

[0266] <Note 11> The external device is a lighting device, and the output unit provides the notification by outputting a control signal that controls the lighting of the lighting device, as described in Note 9.

[0267] <Note 12> The recognition device according to Note 9, wherein the external device is a terminal device, and the output unit transmits a notification to the terminal device including the image, information indicating the first article, and information indicating the second article.

[0268] <Note 13> The external device is a terminal device, and the output unit causes the terminal device to display the information received by the terminal device to the owner of the terminal device, as described in Note 9.

[0269] <Note 14> The recognition device further comprises a display unit equipped with an input function, and the output unit, when the second item is included in a preset group of similar items, displays the group of similar items including the second item on the display unit, and outputs information to an external device indicating the item selected by the user from among the items included in the group of similar items including the second item, as described in Note 1.

[0270] <Note 15> The recognition device described in Note 1, wherein the output unit outputs the identification information of the article as information indicating the article to a registration device that registers the article as a processing target.

[0271] <Note 16> The recognition device as described in Note 10, wherein the output unit outputs the identification information read by the reader when outputting information indicating the first item, and converts the recognition result of the item into the identification information and outputs it when outputting information indicating the second item.

[0272] <Note 17> The identification information is the JAN (Japan Article Number) code of the article, as described in Note 10.

[0273] <Note 18> The recognition device described in Note 10, wherein the output unit outputs additional information along with the identification information.

[0274] <Note 19> The recognition device according to Note 1, wherein the object recognition unit performs object recognition processing using the image captured by the camera when the article is located within a predetermined range of the stand described above, and does not perform object recognition processing using the image captured by the camera when the article is located outside the predetermined range of the stand described above, regardless of whether the image includes the article or not.

[0275] <Note 20> The recognition device according to Note 14, wherein the output unit notifies the object to move into the predetermined area when the object is located outside the predetermined range.

[0276] <Note 21> The recognition device according to Note 14, wherein the output unit outputs information indicating the first item indicated by the identification information read by the reader when the object recognition unit does not perform the object recognition processing and the reader reads the identification information.

[0277] <Note 22> The recognition device as described in Note 1, wherein, when a turntable is connected to the recognition device, the object recognition unit photographs an item placed on the turntable with the camera, and the output unit outputs an image of the photographed item.

[0278] <Note 23> The recognition device described in Note 1, wherein, when a turntable is connected to the recognition device, the object recognition unit photographs an item placed on the turntable with the camera, extracts only the item portion from the image, and performs the object recognition process on the item to be recognized.

[0279] <Note 24> An information output method for a recognition device comprising: a mounting platform; a reader for reading identification information attached to an article placed on the mounting platform; and a camera for taking an image of the article placed on the mounting platform, wherein the method obtains the recognition result of the article by object recognition processing using the image taken by the camera; obtains a first article indicated by the identification information read by the reader and a second article indicated by the recognition result of the object recognition processing; and determines information indicating the article placed on the mounting platform based on the first article and the second article, and outputs it to an external device.

[0280] <Note 25> A program that causes a computer, which comprises a mounting platform, a reader for reading identification information attached to an article placed on the mounting platform, and a camera for taking an image of the article placed on the mounting platform, to execute the following: a process for obtaining the recognition result of the article by object recognition processing using the image taken by the camera; a process for obtaining a first article indicated by the identification information read by the reader and a second article indicated by the recognition result of the object recognition processing; and a process for determining information indicating the article placed on the mounting platform based on the first article and the second article and outputting it to an external device.

[0281] <Note 26> Recognition device comprising: a registration device for registering information of an item; a platform; a reader for reading identification information attached to an item placed on the platform; a camera for taking an image of the item placed on the platform; an object recognition unit for acquiring the recognition result of the item by object recognition processing using the image taken by the camera; and an output unit for acquiring a first item indicated by the identification information read by the reader and a second item indicated by the recognition result of the object recognition processing, determining information indicating the item placed on the platform based on the first and second items, and outputting it to the registration device.

[0282] <Note 27> The recognition system according to Note 26, wherein the recognition device comprises a storage unit capable of storing the registration device and a display unit that displays the output of the settlement device stored in the storage unit.

[0283] Analysis 3 As disclosed in Patent Document 1, information about an item can be obtained from an image. In this case, it is necessary to link the image of the item with the item's information and record it in advance. However, stores and other establishments handle a large number of items, so linking items with their information places a heavy burden on them. Therefore, it is desirable that the linking of images with item identification information can be easily performed.

[0284] Furthermore, the linking of images to item identification information is not limited to retail store systems such as self-checkout counters. For example, when managing items such as parts and work-in-progress in warehouses and factories, or when managing goods such as packages in distribution sites, images and item identification information may be linked to use for item recognition.

[0285] Non-limiting embodiments of this disclosure contribute to the provision of recognition devices, recognition methods, and programs that can easily perform the linking of images with identification information of articles.

[0286] <Note 1> A recognition device comprising: a mounting platform; a reader for reading identification information attached to an item placed on the mounting platform; a camera for taking an image of the item placed on the mounting platform; an object recognition unit for acquiring the recognition result of the item by object recognition processing using the image taken by the camera; and a learning unit for training the recognition engine that performs the object recognition processing to learn the image as an image of the item indicated by the identification information.

[0287] <Note 2> The recognition device as described in Note 1, wherein if the reader successfully reads the identification information and the object recognition process fails, the learning unit causes the recognition engine that performs the object recognition process to learn the image as an image of the item indicated by the identification information.

[0288] <Note 3> The recognition device according to Note 2, further comprising: an output unit that outputs information indicating an item corresponding to the identification information if the reader successfully reads the identification information but the object recognition process fails.

[0289] <Note 4> The recognition device described in Note 1, wherein a rotating platform can be installed in place of or on the aforementioned mounting platform, the reader reads the identification information from an article placed on the rotating platform and rotating, the camera generates images of the article placed on the rotating platform and rotating from multiple angles, and the learning unit trains the recognition engine to recognize the images of the article taken from the multiple angles as images of the article indicated by the identification information.

[0290] <Note 5> The recognition device further comprises an output unit that outputs information indicating an item corresponding to the recognition result by the object recognition unit, or information indicating an item corresponding to the identification information read by the reader, and the recognition device performs output by the output unit when operating in recognition mode, and performs learning by the learning unit when operating in learning mode, as described in Note 4.

[0291] <Note 6> The recognition device as described in Note 5, wherein the learning unit switches the recognition device to learning mode when it detects that the rotating platform has been connected while the recognition device is operating in the recognition mode.

[0292] <Note 7> The recognition device described in Note 5, wherein the learning unit switches to the recognition mode when it detects that the connection to the rotating platform has been disconnected and the aforementioned mounting platform has been installed while it is operating in the learning mode.

[0293] <Note 8> The recognition device as described in Note 1, wherein the recognition engine is located in an external image device including a processor or cloud provided by the recognition device, and the learning unit instructs the recognition engine to learn.

[0294] <Note 9> The recognition device according to Note 1, wherein the learning unit extracts the portion of the image in which the item is photographed, and trains the recognition engine to recognize the extracted image as the image of the item indicated by the identification information.

[0295] <Note 10> A learning method for a recognition device comprising: a mounting platform; a reader for reading identification information attached to an article placed on the mounting platform; and a camera for taking an image of the article placed on the mounting platform, the learning method comprising: obtaining the recognition result of the article by object recognition processing using the image taken by the camera; and training the recognition engine that performs the object recognition processing with the image as an image of the article indicated by the identification information.

[0296] <Note 11> A program that causes a computer, which comprises a mounting platform, a reader for reading identification information attached to an item placed on the mounting platform, and a camera for taking an image of the item placed on the mounting platform, to execute: a process for acquiring the recognition result of the item by object recognition processing using the image taken by the camera, and a process for training the recognition engine that performs the object recognition processing with the image as the image of the item indicated by the identification information.

[0297] The disclosures of the specifications, drawings, and abstracts contained in the Japanese applications 2024-166728, 2024-166731, and 2024-166734, filed on September 25, 2024, are all incorporated herein by reference.

[0298] This disclosure is useful for reading barcodes attached to items in self-checkout systems.

[0299] 1 AI scanner 1a Base unit 1b Back panel unit 1c Top panel unit 2 Handheld scanner 3 Payment terminal 4 POS device 11 Mounting platform 12a-12f Reader 13a-13c Camera 14a, 14b Display 15a-15g Light-emitting device 16 Turntable 17 Reader 20 Control device

Claims

1. A recognition device comprising: an acquisition unit that acquires images of an item placed on a platform from multiple cameras that photograph the item from different angles; an object recognition unit that recognizes the item placed on the platform by object recognition processing using the images; and an output unit that outputs information indicating the recognized item.

2. The recognition device according to claim 1, wherein the acquisition unit further acquires the identification information from a reader that reads the identification information of an article placed on the aforementioned stand, and the object recognition unit determines the article placed on the aforementioned stand based on the recognition result from the object recognition process and the result of reading the identification information by the reader.

3. The recognition device according to claim 2, wherein the object recognition unit acquires the recognition result of the object recognition process when the article is located within a predetermined range of the stand described above, and does not acquire the recognition result of the object recognition process when the article is located outside the predetermined range of the stand described above, regardless of whether the image contains the article or not.

4. The recognition device according to claim 3, wherein the object recognition unit reads the identification information by the reader regardless of whether the article is located within the predetermined range of the stand described above.

5. If the reader successfully reads the identification information, and the object recognition unit fails to obtain the recognition result of the article, the recognition device causes the object recognition engine that performs the object recognition processing to learn the image as an image of the article indicated by the identification information, according to claim 2.

6. The recognition device according to claim 5, wherein the object recognition engine is located in an external device including a processor or cloud provided by the recognition device, and the external device instructs the object recognition engine to learn.

7. The recognition device according to claim 2, wherein if the reader successfully reads the identification information and the object recognition unit fails to obtain the recognition result of the article, the output unit outputs the article indicated by the reader's reading result as the recognition result of the article.

8. The recognition device according to claim 2, wherein the output unit outputs information indicating the article as the recognition result of the article when the first article indicated by the identification information read by the reader matches the second article indicated by the recognition result of the object recognition process.

9. The recognition device according to claim 6, wherein the output unit outputs information indicating the first item to the external device when the first item indicated by the identification information read by the reader does not match the second item indicated by the recognition result of the object recognition process, and the object recognition process recognizes that there is one item placed on the stand described above.

10. The recognition device according to claim 6, wherein the output unit outputs to the external device information indicating the second item indicated by the recognition result of the object recognition process when the identification information of an item read by the reader includes identification information of multiple items.

11. The recognition device according to claim 10, wherein the output unit notifies the customer to move the item if the identification information of the item read by the reader includes identification information for multiple items and information indicating the second item cannot be obtained.

12. The recognition device according to claim 2, wherein the output unit outputs a statement indicating that the first item and the second item do not match when the identification information of the item read by the reader does not match the second item indicated by the recognition result of the object recognition process.

13. The recognition device according to claim 12, wherein, if the first article and the second article do not match, the output unit outputs information indicating the article selected by the user from among the first article and the second article as information indicating the recognized article.

14. The recognition device according to claim 12, wherein the output unit transmits a notification to a terminal device including the image, information indicating the first article, and information indicating the second article, when the first article and the second article do not match.

15. The recognition device according to claim 2, wherein the output unit outputs the identification information read by the reader when outputting information indicating a first article indicated by the identification information read by the reader, and converts the recognition result of the article into the identification information and outputs it when outputting information indicating a second article indicated by the recognition result of the object recognition process.

16. The recognition device according to claim 15, wherein the identification information is the JAN (Japan Article Number) code of the article.

17. The recognition device according to claim 2, wherein the output unit outputs information indicating the article indicated by either the reading of identification information by the reader or the acquisition of the recognition result of the article by the object recognition unit, if only one of these is successful.

18. The recognition device according to claim 1, wherein the output unit, when the item recognized by the object recognition process is included in a pre-set group of similar items, displays the group of similar items including the item on the display unit and outputs information indicating the item selected by the user from among the items included in the group of similar items including the item.

19. A recognition method comprising: a recognition device that acquires images of an item placed on a platform from multiple cameras that photograph the item from different angles; an object recognition process using the images to recognize the item placed on the platform described above; and an output of information indicating the recognized item.

20. A program that causes a computer to perform a process that involves acquiring images of an item placed on a platform from multiple cameras that photograph the item from different angles, recognizing the item placed on the platform using the images through object recognition processing, and outputting information indicating the recognized item.

Images