Information processing device, image display method, and image display program

Abstract

To provide an information processing apparatus configured to immediately respond to addition of a new article, an image display method, and an image display program.SOLUTION: An information processing apparatus 10 includes: an acquisition unit 11A which acquires a card image obtained by imaging a card medium having a unique characteristic part; a recognition unit 11B which recognizes the unique characteristic part from the acquired card image; a generation unit 11C which generates an optimal article image for an article, for the card image with the recognized unique characteristic part, using a trained model 15C generated by machine-learning a training data group including a pair of card image obtained by imaging a card medium in advance and an article image obtained by imaging an article in advance; and a display control unit 11D which controls a display unit to display the generated article image in place of the acquired card image.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an information processing device, an image display method, and an image display program. [Background technology]

[0002] Smart cash registers using object detection technology are becoming more and more popular. At smart cash registers, a user brings in the item they want to purchase and takes a picture of it with a camera installed in the smart cash register, which captures an image of the item. Then, by applying object detection technology (YOLO, etc.) to the image of the item, the item can be automatically recognized and payment can be made.

[0003] For example, Patent Document 1 describes a technology for performing machine learning by augmenting data on images captured from multiple angles to determine the size and orientation of an object. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-148980 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, when performing object detection in a smart register, if machine learning is performed by expanding the data of images taken from multiple angles to determine the size and orientation of the object, the amount of learning becomes large. Therefore, it is difficult to perform machine learning every time a new item is added, and it is not possible to immediately respond to the addition of a new item.

[0006] The present invention has been made in consideration of the above circumstances, and an object of the present invention is to provide an information processing device, an image display method, and an image display program that can immediately respond to the addition of a new item. [Means for solving the problem]

[0007] In order to achieve the above object, the information processing device described in claim 1 includes an acquisition unit that acquires a card image obtained by photographing a card medium having a unique characteristic portion; a recognition unit that recognizes the unique characteristic portion from the card image acquired by the acquisition unit; a generation unit that generates an optimal item image for the item for the card image in which the unique characteristic portion has been recognized by the recognition unit using a trained model generated by machine learning a group of learning data that pairs a card image obtained in advance by photographing the card medium with an item image obtained in advance by photographing an item; and a display control unit that controls the display of the item image generated by the generation unit on a display unit instead of the card image acquired by the acquisition unit.

[0008] According to the invention described in claim 1, a trained model is used that uses a card image as input and generates and outputs an optimal product image. Compared to the case where various shapes of products themselves are photographed from multiple angles and recognized, it is sufficient to associate the product with a simple card medium that is easy to recognize, so the amount of learning can be reduced and it is possible to immediately respond to the addition of new products. In addition, since the user can purchase the product while visually checking the product image, it is possible to prevent purchasing mistakes.

[0009] In addition, the information processing device of claim 2 is the information processing device of claim 1, in which the recognition unit recognizes the unique characteristic parts from the card images acquired by the acquisition unit using a feature recognition model generated by machine learning by matching a group of card images obtained in advance by photographing multiple types of card media with different unique characteristic parts with labels that represent the unique characteristic parts.

[0010] According to the invention described in claim 2, by using a feature recognition model that is generated by machine learning including a group of card images and that uses a card image as input and recognizes and outputs its unique features, it is possible to immediately respond to the addition of new items.

[0011] In addition, the information processing device of claim 3 is the information processing device of claim 2, further comprising a modification unit that changes a label representing the unique characteristic portion recognized by the recognition unit to a label representing the item of the item image generated by the generation unit, and the display control unit controls the display of the label representing the item changed by the modification unit together with the item image on the display unit.

[0012] According to the invention described in claim 3, the addition of a new item can be immediately handled by simply changing the label representing the unique characteristic portion of the card image to a label representing the item of the item image.

[0013] In addition, the information processing device of claim 4 is the information processing device of claim 3, wherein the display control unit further controls the display unit to display item information including the item name and price of the item based on the label representing the item changed by the change unit.

[0014] According to the invention as recited in claim 4, by displaying the product information corresponding to the label representing the product of the changed product image, the user can grasp the product information about the product.

[0015] In addition, the information processing device described in claim 5 is an information processing device described in any one of claims 1 to 3, further comprising a detection unit that detects items by applying object detection technology to the item image generated by the generation unit, and the display control unit controls the display of item information including the item name and price of the item on the display unit based on the detection result by the detection unit.

[0016] According to the invention of claim 5, by displaying item information corresponding to an item detected from an item image using object detection technology, a user can grasp item information about the item.

[0017] In addition, the information processing device described in claim 6 is the information processing device described in claim 5, in which the detection unit detects the items from the item images generated by the generation unit using an item detection model generated by machine learning by matching a group of item images obtained in advance by photographing multiple types of items with labels representing the items.

[0018] According to the invention described in claim 6, by using an item detection model that is generated by machine learning including a group of item images and that uses an item image as input to detect and output an item, it is possible to accommodate the addition of new items without using a card medium.

[0019] In addition, an information processing device according to a seventh aspect of the present invention is the information processing device according to the sixth aspect, in which the article detection model is a model that has been re-trained by adding an article image included in the training data group.

[0020] According to the invention described in claim 7, by using an item detection model that has been retrained by adding item images included in the training data group, it is possible to accommodate the addition of new items without using card media.

[0021] Moreover, an information processing device according to claim 8 is the information processing device according to any one of claims 1 to 7, wherein the trained model is a model of a generative adversarial network.

[0022] According to the invention described in claim 8, by using a generative adversarial network as a trained model to generate optimal item images for card images, the amount of learning can be reduced and the addition of new items can be responded to immediately.

[0023] Further, in order to achieve the above object, the image display method according to claim 9 comprises: The computerA card image is obtained by photographing a card medium having a unique characteristic part, the unique characteristic part is recognized from the acquired card image, and an optimal item image of the item is generated for the card image in which the unique characteristic part is recognized using a trained model generated by machine learning a group of learning data that pairs a card image obtained in advance by photographing the card medium with an item image obtained in advance by photographing an item, and control is performed to display the generated item image on a display unit instead of the acquired card image.

[0024] According to the invention described in claim 9, as in claim 1, a trained model is used that uses a card image as input and generates and outputs an optimal product image. Compared to the case where various shapes of products themselves are photographed from multiple angles and recognized, it is sufficient to associate the product with a simple card medium that is easy to recognize, so the amount of learning can be reduced and it is possible to immediately respond to the addition of new products. In addition, since the user can purchase the product while visually checking the product image, it is possible to prevent purchasing mistakes.

[0025] Furthermore, in order to achieve the above object, the image display program described in claim 10 causes a computer to execute the following steps: acquire a card image obtained by photographing a card medium having a unique characteristic portion, recognize the unique characteristic portion from the acquired card image, and use a trained model generated by machine learning a group of learning data that pairs a card image obtained in advance by photographing the card medium with an item image obtained in advance by photographing an item to generate an optimal item image for the item for the card image in which the unique characteristic portion has been recognized; and control the display of the generated item image on a display unit instead of the acquired card image.

[0026] According to the invention described in claim 10, as in claim 1, a trained model is used that uses a card image as input and generates and outputs an optimal article image. Compared to the case where articles of various shapes are photographed from multiple angles and recognized, it is sufficient to associate the article with a simple card medium that is easy to recognize, so the amount of learning can be reduced and it is possible to immediately respond to the addition of new articles. In addition, since the user can purchase while visually checking the article image, it is possible to prevent purchasing mistakes. Effect of the Invention

[0027] As described above, according to the present invention, it is possible to obtain an effect that the addition of a new article can be immediately dealt with. [Brief description of the drawings]

[0028] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an image display system according to an embodiment. [Diagram 2] 1A to 1C are diagrams illustrating an example of a plurality of types of card media according to an embodiment. [Diagram 3] 2 is a block diagram showing an example of an electrical configuration of the information processing apparatus according to the embodiment; [Figure 4] FIG. 2 is a block diagram showing an example of a functional configuration of the information processing device according to the embodiment. [Diagram 5] FIG. 1 is a diagram showing an example of a training data group used for training a trained model according to an embodiment. [Figure 6] FIG. 2 is a diagram showing an example of an item information DB according to the embodiment. [Figure 7] 5A to 5C are diagrams illustrating an example of a screen transition of a display unit according to the embodiment. [Figure 8] 10 is a flowchart showing an example of a flow of an image display process by an image display program according to the embodiment. [Figure 9] 11 is a flowchart showing an example of a flow of a learning process by an image display program according to an embodiment, illustrating a process of generating a feature recognition model. [Figure 10]11 is a flowchart showing an example of the flow of another learning process by the image display program according to the embodiment, showing a process for generating a trained model. [Figure 11] 1A to 1C are diagrams illustrating a method for photographing an article according to an embodiment. [Figure 12] 13 is a flowchart showing an example of the flow of yet another learning process by the image display program according to the embodiment, illustrating a re-learning process for the item detection model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Hereinafter, an example of a mode for implementing the technology of the present disclosure will be described in detail with reference to the drawings. Note that components and processes that perform the same operation, action, and function may be given the same reference numerals throughout all the drawings, and duplicated explanations may be omitted as appropriate. Each drawing is merely shown in a schematic manner to allow a sufficient understanding of the technology of the present disclosure. Therefore, the technology of the present disclosure is not limited to only the illustrated examples. In addition, in this embodiment, explanations of configurations that are not directly related to the present invention or well-known configurations may be omitted.

[0030] FIG. 1 is a diagram showing an example of the configuration of an image display system 100 according to this embodiment.

[0031] 1, an image display system 100 according to this embodiment includes an information processing device 10, a camera 30, a reader 40, a feature recognition model 15B, a trained model 15C, an item detection model 15D, and an item information DB (DataBase) 15E. Note that although the feature recognition model 15B and the item detection model 15D are shown as separate models, they may be realized as a single object detection model such as YOLO.

[0032] FIG. 2 is a diagram showing an example of a plurality of types of card media according to the present embodiment.

[0033] As shown in FIG. 2, each of the card media Ac1, Ac2, Ac3, ... according to this embodiment is a plurality of types of card-shaped media having different unique characteristic parts. The unique characteristic parts are represented, for example, by pictures, characters, etc., and in the example of FIG. 2, the card medium Ac1 is a "blue rectangular" card, the card medium Ac2 is a "red circular" card, and the card medium Ac3 is a "yellow star" card. Also, each of the card media Ac1, Ac2, Ac3, ... is assigned a label representing the unique characteristic part. Specifically, the card medium Ac1 is assigned a label "Visitor001" representing a "blue rectangular", the card medium Ac2 is assigned a label "Visitor002" representing a "red circular", and the card medium Ac3 is assigned a label "Visitor003" representing a "yellow star". When there is no need to particularly distinguish between these plurality of types of card media Ac1, Ac2, Ac3, ..., they are simply referred to as card media Ac.

[0034] The image display system 100 according to the present embodiment is shown as being applied to a smart register installed in a purchasing store or the like used by employees in a certain business establishment. In the image display system 100, a plurality of card media Ac corresponding to newly arrived items Bp are prepared in advance, and a corresponding plurality of card media Ac are placed in the sales floor of the items Bp. When there are a plurality of newly arrived items Bp, card media Ac with different unique characteristic parts are prepared for each of the plurality of items Bp. In the example of FIG. 1, an item Bp representing a rag and a plurality of card media Ac representing a "blue rectangle" are placed in the sales floor of the rags.

[0035] When purchasing an item Bp, a user brings the item Bp and card medium Ac to the smart register. If multiple items Bp are to be purchased, multiple card media Ac can be brought along. When the user holds the card medium Ac over the camera 30, the camera 30 captures an image of the card medium Ac and transmits a card image Ag to the information processing device 10. The camera 30 is a camera capable of capturing still and moving images.

[0036] The information processing device 10 is connected to each of the camera 30 and the reader 40. For example, a general-purpose server computer, a personal computer (PC), or the like is applied to the information processing device 10. The information processing device 10 uses the feature recognition model 15B to recognize a unique feature part from the card image Ag acquired from the camera 30, and uses the trained model 15C to generate an optimal article image Bg for the card image Ag in which the unique feature part has been recognized, and displays the generated article image Bg on the display unit 16. The "optimal article image" here means an image in which the shape of the article is easily detected and an image in which the user can easily grasp the article, for example, an image in which the article is viewed from the front at an appropriate size. At this time, the position of the unique feature part of the card image Ag is recognized, and the recognized position is replaced with the article Bp of the article image Bg and displayed. In addition, the label representing the unique feature part of the card image Ag is changed to a label representing the article Bp of the article image Bg. In the example of FIG. 1, the label "Visitor001" of the card image Ag is changed to the label "Waste" of the item image Bg.

[0037] Then, the information processing device 10 refers to the item information DB 15E based on the changed label (the label "Waste" in the example of FIG. 1), and displays the item information (item name, price, etc.) regarding the item Bp obtained from the item information DB 15E on the display unit 16. The user checks the item information and holds his / her employee ID card over the reader 40.

[0038] The reader 40 reads the employee ID card held up by the user, and transmits the employee information obtained by reading it to the information processing device 10. The information processing device 10 uses the employee information obtained from the reader 40 to make payment for the detected item Bp.

[0039] On the other hand, when the user purchases an existing item other than the item Bp (hereinafter referred to as an "existing item"), the user brings the existing item to the smart register. The existing item is previously learned by the item detection model 15D in association with a label representing the existing item. When the user places the existing item in a predetermined position where it can be photographed by the camera 30, the camera 30 photographs the existing item and transmits the existing item image to the information processing device 10. The information processing device 10 uses the item detection model 15D to detect the existing item from the existing item image acquired from the camera 30, and based on the detected existing item, refers to the item information DB 15E, and displays item information (item name, price, etc.) related to the existing item obtained from the item information DB 15E on the display unit 16. Note that in the image display system 100 according to this embodiment, it is also possible to simultaneously purchase a new item Bp and an existing item other than the item Bp.

[0040] According to this embodiment, when a new item Bp is purchased, the item Bp itself is not recognized, but the card medium Ac associated with the item Bp is recognized. The characteristic parts of the card medium Ac are simple, so it is easy to recognize and does not require much learning. In addition, a generative adversarial network (GAN) described later is used for machine learning to generate an optimal item image Bg from a card image Ag. This generative adversarial network (GAN) also does not require much learning. Therefore, compared to the case where the item itself is recognized using YOLO, which is one of the object detection technologies, the amount of learning is reduced, and it is possible to immediately respond to the addition of new items without stopping the service of the smart register.

[0041] In this embodiment, for new items, the trained model 15C is used to generate an optimal item image for the card image. On the other hand, for existing items already on sale, the item detection model 15D such as YOLO is used to detect the item from the item image. Here, the training data group used for training the trained model 15C includes item images of new items. These item images of new items are reflected in the item detection model 15D, and the item images of new items are added as training data to retrain the item detection model 15D. When the item detection model 15D is retrained, it becomes possible to recognize the new item itself, so the card medium may be unnecessary. For example, when the card medium of the new item runs out (or there are only a few remaining), the new item may be switched to the retrained item detection model 15D for the new item, and the corresponding item may be detected from the item image obtained by photographing the item itself instead of the card medium.

[0042] FIG. 3 is a block diagram showing an example of the electrical configuration of the information processing device 10 according to the present embodiment.

[0043] As shown in FIG. 3, the information processing device 10 according to this embodiment includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an input / output interface (I / O) 14, a memory unit 15, a display unit 16, an operation unit 17, a communication unit 18, and an external interface (external I / F) 19.

[0044] The CPU 11, ROM 12, RAM 13, and I / O 14 are connected to each other via a bus. The I / O 14 is connected to various functional units including a storage unit 15, a display unit 16, an operation unit 17, a communication unit 18, and an external I / F 19. These functional units are capable of communicating with the CPU 11 via the I / O 14.

[0045] The control unit is configured with the CPU 11, the ROM 12, the RAM 13, and the I / O 14. The control unit may be configured as a sub-control unit that controls part of the operation of the information processing device 10, or may be configured as a part of the main control unit that controls the entire operation of the information processing device 10. For example, an integrated circuit such as an LSI (Large Scale Integration) or an IC (Integrated Circuit) chip set is used for part or all of the blocks of the control unit. An individual circuit may be used for each of the above blocks, or a circuit in which some or all of the blocks are integrated may be used. The above blocks may be provided integrally, or some of the blocks may be provided separately. In addition, a part of each of the above blocks may be provided separately. The integration of the control unit is not limited to an LSI, and a dedicated circuit or a general-purpose processor may be used.

[0046] For example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. are used as the storage unit 15. An image display program 15A according to the present embodiment is stored in the storage unit 15. Note that the image display program 15A may be stored in the ROM 12.

[0047] The image display program 15A may be pre-installed in the information processing device 10, for example. The image display program 15A may be realized by storing it in a non-volatile storage medium or distributing it via a network and appropriately installing it in the information processing device 10. Note that examples of the non-volatile storage medium include a CD-ROM (Compact Disc Read Only Memory), a magneto-optical disk, a HDD, a DVD-ROM (Digital Versatile Disc Read Only Memory), a flash memory, a memory card, and the like.

[0048] For example, a liquid crystal display (LCD) or an organic EL (Electro Luminescence) display is used for the display unit 16. The display unit 16 has an integrated touch panel and receives operation input from the user via the touch panel. The operation unit 17 is provided with input devices such as a keyboard and a mouse.

[0049] The communication unit 18 is a communication interface for connecting to a network such as the Internet, a local area network (LAN), or a wide area network (WAN), and is capable of communicating with external devices via the network.

[0050] The external I / F 19 is an interface for connecting to each of the camera 30 and the reader 40, and for example, a USB (Universal Serial Bus) is used.

[0051] The CPU 11 of the information processing device 10 according to this embodiment writes an image display program 15A stored in the storage unit 15 into the RAM 13 and executes it, thereby functioning as each unit shown in FIG.

[0052] FIG. 4 is a block diagram showing an example of a functional configuration of the information processing device 10 according to the present embodiment.

[0053] As shown in FIG. 4, the CPU 11 of the information processing device 10 according to this embodiment functions as an acquisition unit 11A, a recognition unit 11B, a generation unit 11C, a display control unit 11D, a change unit 11E, a detection unit 11F, and a learning unit 11G.

[0054] A feature recognition model 15B, a trained model 15C, an article detection model 15D, and an article information DB 15E are stored in the storage unit 15. Note that the feature recognition model 15B, the trained model 15C, the article detection model 15D, and the article information DB 15E may be stored in an accessible external storage device.

[0055] The feature recognition model 15B is a model generated by machine learning by associating a group of card images obtained by photographing a plurality of types of card media Ac1, Ac2, Ac3, ... (see FIG. 2 above) having different unique feature parts with labels representing the unique feature parts. For the feature recognition model 15B, various models such as well-known CNN (Convolutional Neural Network), R-CNN (Region-based-CNN), YOLO, SSD (Single Shot multibox Detector), etc. are used, but other object detection techniques may also be used. For example, deep learning is used as a learning method for machine learning. When a card image Ag is input, the feature recognition model 15B outputs a label (e.g., Visitor001) representing a unique feature part (e.g., a blue rectangle) of the card image Ag.

[0056] FIG. 5 is a diagram showing an example of a group of training data used for training the trained model 15C according to this embodiment.

[0057] As shown in FIG. 5, the trained model 15C is a model generated by machine learning a group of training data in which a card image Ag obtained by photographing a card medium Ac in advance and an item image Bg obtained by photographing an item Bp in advance are paired. For example, a model of a generative adversarial network (GAN) is used for the trained model 15C. More specifically, pix2pix, which is a type of conditional GAN, is used. This pix2pix is ​​a type of image generation algorithm using a GAN, and is a method of generating a pair of images by interpolating from one image while taking into account the relationship between images by learning the relationship between two paired images. The card medium Ac is, for example, a card with a "blue rectangle", and the item Bp is, for example, a "rag".

[0058] The card image Ag as the learning data is subjected to a predetermined data expansion (e.g., enlargement, reduction, rotation, noise addition, etc.), and many images are generated from several captured still images. Similarly, the item image Bg as the learning data is subjected to a predetermined data expansion, and many images are generated from several captured still images. In this way, many pairs of the card image Ag and the item image Bg are generated as a learning data group. The trained model 15C receives the card image Ag (i.e., a label representing a unique characteristic part of the card image Ag) as an input, and generates and outputs an optimal item image Bg. For example, for the label "Visitor001" representing a "blue rectangle", it outputs an optimal image of a cloth. For example, deep learning is used as a learning method of machine learning.

[0059] The item detection model 15D is a model generated by machine learning by associating a group of item images obtained by photographing a plurality of types of items (e.g., rags, tape, batteries, etc.) in advance with labels representing the items. For the item detection model 15D, various models such as well-known CNN, R-CNN, YOLO, SSD, etc. are used, but other object detection techniques may also be used. For example, deep learning is used as a learning method for machine learning. When an item image is input, the item detection model 15D outputs a label representing the item in the item image.

[0060] FIG. 6 is a diagram showing an example of the product information DB 15E according to the present embodiment.

[0061] In the product information DB 15E shown in FIG. 6, a label representing the product, a product name, a model number, and a price are registered as product information related to the product.

[0062] The acquisition unit 11A acquires a card image Ag obtained by photographing the card medium Ac with the camera 30.

[0063] The recognition unit 11B recognizes a unique feature part from the card image Ag acquired by the acquisition unit 11A. Specifically, the card image Ag is input to the feature recognition model 15B, and a label (e.g., Visitor001) representing the unique feature part (e.g., a blue rectangle) is output from the feature recognition model 15B.

[0064] The generation unit 11C uses the trained model 15C to generate an optimal item image Bg for the card image Ag whose unique feature portion has been recognized by the recognition unit 11B. Specifically, the card image Ag (e.g., an image showing a blue rectangle) is input to the trained model 15C, and the trained model 15C outputs the item image Bg (e.g., an image of a rag).

[0065] The display control unit 11D performs control to display on the display unit 16 the item image Bg generated by the generation unit 11C instead of the card image Ag acquired by the acquisition unit 11A.

[0066] The change unit 11E changes the label representing the unique characteristic part recognized by the recognition unit 11B to a label representing the item Bp of the item image Bg generated by the generation unit 11C. For example, the label "Visitor001" of the card image Ag is changed to the label "Waste" of the item image Bg. In this case, the display control unit 11D controls to display the label representing the item Bp changed by the change unit 11E on the display unit 16 together with the item image Bg. In other words, the item Bp is identified by simply changing the label of the item Bp without performing item detection.

[0067] The detection unit 11F detects the item Bp by applying an object detection technique to the item image Bg generated by the generation unit 11C. Specifically, the detection unit 11F detects the item Bp from the item image Bg generated by the generation unit 11C using the item detection model 15D, which is an example of the object detection technique. The item detection model 15D is a model that has been retrained by adding an item image included in the learning data group used to train the trained model 15C. In other words, when the item detection model 15D is retrained, it is possible to recognize the new item itself, so the card medium may be unnecessary. For example, when the card medium of the new item runs out (or there are only a few remaining), the new item may be switched to the retrained item detection model 15D, and the corresponding item may be detected from the item image obtained by photographing the item itself instead of the card medium. In this case, the display control unit 11D controls the display unit 16 to display item information including the item name and amount of the item based on the detection result by the detection unit 11F.

[0068] The display control unit 11D refers to the product information DB 15E based on the label representing the product Bp changed by the change unit 11E or the label representing the product Bp detected by the detection unit 11F, acquires product information including the product name and price corresponding to the label representing the product Bp, and controls display of the acquired product information on the display unit 16. The number of products Bp may be determined by the number of card media Ac, or may be determined by input by the user.

[0069] FIG. 7 is a diagram showing an example of screen transition on the display unit 16 according to the present embodiment.

[0070] As shown in Fig. 7, when an optimal article image Bg is generated for a card image Ag whose unique characteristic portion has been recognized using the trained model 15C, an article image display screen 16A is displayed on the display unit 16. The generated article image Bg is displayed on the article image display screen 16A, along with a message "Do you want to present article information?", a "Yes" button, and a "No" button. Note that the label of the article image Bg has been changed from a label representing the unique characteristic portion (e.g., Visitor001) to a label representing the article Bp (e.g., rag).

[0071] Next, when the user presses the "Yes" button on the item image display screen 16A, the item information DB 15E shown in FIG. 6 above, for example, is referenced based on the label representing the item Bp, and item information corresponding to the item Bp is obtained. As a result, the item information display screen 16B is displayed on the display unit 16. The item information display screen 16B displays the item image Bg and item information, as well as a message saying "Please check the item information and hold your employee ID card if you wish to make payment."

[0072] Next, when the user holds his / her employee ID card over the reader 40 while the item information display screen 16B is displayed, the employee information is read from the employee ID card. As a result, the payment screen 16C is displayed on the display unit 16. The payment screen 16C displays the item image Bg, item information, and employee information, as well as a "Done" button and a "Cancel" button. When the user presses the "Done" button, the payment for the item Bp is completed.

[0073] The learning unit 11G performs machine learning to generate the feature recognition model 15B, the trained model 15C, and the item detection model 15D. In this embodiment, the learning unit 11G is shown as being included in the information processing device 10, but the learning unit 11G may be included in another information processing device. In this case, the information processing device 10 acquires the feature recognition model 15B, the trained model 15C, and the item detection model 15D from the other information processing device.

[0074] Next, the operation of the information processing device 10 according to the present embodiment will be described with reference to FIG.

[0075] FIG. 8 is a flowchart showing an example of the flow of image display processing by the image display program 15A according to this embodiment.

[0076] First, when a new item (item Bp) is purchased, a card medium Ac corresponding to the new item is held up to the camera 30, an image display process is instructed to the information processing device 10, the image display program 15A is started, and the following processes are executed.

[0077] In step S101 of FIG. 8, the CPU 11 captures an image of the card medium Ac with the camera 30 to obtain a card image Ag.

[0078] In step S102, CPU 11 uses feature recognition model 15B to recognize a unique feature part from card image Ag acquired in step S101. Specifically, when card image Ag is input, feature recognition model 15B outputs a label (e.g., Visitor001) representing a unique feature part (e.g., a blue rectangle) of card image Ag.

[0079] In step S103, the CPU 11 uses the trained model 15C to generate an optimal article image Bg for the card image Ag whose unique feature portion was recognized in step S102. As described above, the "optimal article image" means an image in which the shape of the article is easily detected and the article is easily recognized by the user, for example, an image of an appropriate size in which the article is viewed from the front. Specifically, for example, when a card image Ag recognized as a blue rectangular card is input, an article image Bg representing a rag is output.

[0080] In step S104, the CPU 11 changes the label representing the unique characteristic portion recognized in step S102 to a label representing the item Bp of the item image Bg generated in step S103. For example, the label "Visitor001" of the card image Ag is changed to the label "Waste" of the item image Bg.

[0081] In step S105, the CPU 11 controls to display the item image Bg generated in step S103 on the display unit 16 together with the label changed in step S104, instead of the card image Ag (for example, see the item image display screen 16A shown in FIG. 7 described above). At this time, the position of the unique characteristic part of the card image Ag is recognized, and the item Bp of the item image Bg is replaced with the recognized position. Note that the label of the item image Bg has been changed from the label representing the unique characteristic part (for example, Visitor001) to the label representing the item Bp (for example, rag).

[0082] In step S106, the CPU 11 refers to the product information DB 15E shown in Fig. 6 described above, for example, based on the label representing the product Bp changed in step S104, acquires product information including the product name, model number, and price corresponding to the label from the product information DB 15E, and performs control to display the acquired product information on the display unit 16 (for example, see the product information display screen 16B shown in Fig. 7 described above). Note that the number of products Bp may be determined by the number of card media Ac, or may be determined by a user's input.

[0083] In step S107, the CPU 11 judges whether or not the user's employee ID card has been read by the reader 40. If it is judged that the user's employee ID card has been read by the reader 40 (if the judgment is positive), the process proceeds to step S108, and if it is judged that the user's employee ID card has not been read by the reader 40 (if the judgment is negative), the process waits in step S107.

[0084] In step S108, the CPU 11 performs control to display the employee information obtained by reading the user's employee ID card in step S107 on the display unit 16 (for example, see the payment screen 16C shown in FIG. 7 above). When the user presses the "Complete" button on the payment screen 16C, the payment process is completed.

[0085] In step S109, the CPU 11 stores the article image Bg included in the learning data group used to learn the learned model 15C in the storage unit 15 as learning data for re-learning the article detection model 15D, and ends the image display process by the image display program 15A. In this way, the article image Bg of the newly received article Bp is reflected in the article detection model 15D, and the article detection model 15D is re-learned by adding the article image Bg as learning data. When the article detection model 15D is re-learned, it is possible to recognize the article Bp itself, so the card medium Ac may be unnecessary. For example, when the card medium Ac of the article Bp runs out (or there are only a few left), the re-learned article detection model 15D for the article Bp may be switched to the re-learned article detection model 15D for the article Bp, and the corresponding article may be detected from the article image obtained by photographing the article Bp itself instead of the card medium Ac.

[0086] Next, the learning process of the information processing device 10 according to the present embodiment will be described with reference to FIGS.

[0087] FIG. 9 is a flowchart showing an example of the flow of a learning process by the image display program 15A according to this embodiment, and shows a process for generating a feature recognition model 15B.

[0088] In step S111 of FIG. 9, the CPU 11 acquires a group of card images obtained by photographing a plurality of types of card media Ac1, Ac2, Ac3, . . . (see FIG. 2 above) having different unique characteristic portions.

[0089] In step S112, CPU 11 performs machine learning on the group of card images acquired in step S111 by associating them with labels that represent unique characteristic parts. As a learning model for performing machine learning, various models such as well-known CNN, R-CNN, YOLO, SSD, etc. are used. Note that, in the machine learning, it is desirable to increase the learning data by performing data expansion (enlarging, reducing, rotating, adding noise, etc.) on the card images.

[0090] In step S113, CPU 11 generates, by the machine learning performed in step S112, feature recognition model 15B that receives card image Ag as input and outputs labels that represent unique feature parts.

[0091] In step S114, the CPU 11 stores the feature recognition model 15B generated in step S113 in the storage unit 15, and ends the learning process by the image display program 15A.

[0092] FIG. 10 is a flowchart showing an example of the flow of another learning process by the image display program 15A according to the present embodiment, and shows a process for generating a trained model 15C.

[0093] In step S121 of FIG. 10, the CPU 11 acquires a plurality (several) card images Ag obtained by photographing a card medium Ac having a unique characteristic portion.

[0094] In step S122, the CPU 11 acquires a plurality (several) of article images Bg obtained by photographing the corresponding article Bp.

[0095] FIG. 11 is a diagram for explaining a method for photographing an article Bp according to this embodiment.

[0096] As shown in Fig. 11, a mounting table 31 is provided directly below the camera 30. The mounting table 31 is a table on which an article Bp is placed. The mounting table 31 has a built-in light source (not shown) and can irradiate light from behind the placed article Bp. When the camera 30 photographs the article Bp placed on the mounting table 31, it is possible to obtain three-dimensional feature amounts by adjusting the shooting angle of the camera 30 while irradiating light from behind the article Bp to emphasize its contour.

[0097] In step S123, CPU 11 performs a predetermined data expansion process (e.g., enlargement, reduction, rotation, noise addition, etc.) on the multiple card images Ag acquired in step S121, and generates many images from several captured still images. Similarly, CPU 11 performs a predetermined data expansion process on the multiple item images Bg acquired in step S122, and generates many images from several captured still images. In this way, many pairs of card images Ag and item images Bg are generated as a learning data group.

[0098] In step S124, the CPU 11 performs machine learning on the group of learning data (pairs of card images and item images) generated in step S123. For example, pix2pix, which is a type of Conditional GAN, is used as a learning model for performing the machine learning.

[0099] In step S125, the CPU 11 generates a trained model 15C that receives the card image Ag as an input and generates and outputs an optimal item image Bg, by the machine learning performed in step S124.

[0100] In step S126, the CPU 11 stores the trained model 15C generated in step S125 in the storage unit 15, and ends the learning process by the image display program 15A.

[0101] FIG. 12 is a flowchart showing an example of the flow of yet another learning process by the image display program 15A according to the present embodiment, and shows the re-learning process of the item detection model 15D.

[0102] In step S131 in FIG. 12, the CPU 11 adds an article image Bg (a new article image) included in the learning data group used for learning the trained model 15C as learning data for re-learning the article detection model 15D.

[0103] In step S132, the CPU 11 associates the item image Bg added in step S131 with a label representing the item and re-learns the item detection model 15D. For the item detection model 15D, various models such as well-known CNN, R-CNN, YOLO, SSD, etc. are used.

[0104] In step S133, the CPU 11 stores the object detection model 15D re-trained in step S132 in the storage unit 15, and ends the re-learning process by the image display program 15A.

[0105] As described above, according to this embodiment, when a new item is purchased, the card medium associated with the new item is recognized instead of the new item itself. The characteristic parts of the card medium are simple, so it is easy to recognize and does not require much learning. In addition, a generative adversarial network (GAN) is used for machine learning to generate an optimal item image from a card image, so this also does not require much learning. Therefore, compared to when the item itself is recognized using YOLO, which is one of the object detection technologies, the amount of learning is reduced, and it is possible to immediately respond to the addition of a new item without stopping the service of the smart register.

[0106] In each of the above embodiments, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0107] In addition, the operations of the processor in each of the above embodiments may be performed not only by one processor, but also by multiple processors located at physically separate locations working together. The order of the operations of the processor is not limited to the order described in each of the above embodiments, and may be changed as appropriate.

[0108] The information processing device according to the embodiment has been described above as an example. The embodiment may be in the form of a program for causing a computer to execute the functions of the information processing device. The embodiment may be in the form of a non-transitory storage medium that stores the program and is readable by a computer.

[0109] Furthermore, the configuration of the information processing device described in the above embodiment is merely an example, and may be changed according to circumstances without departing from the spirit of the invention.

[0110] Furthermore, the processing flow of the program described in the above embodiment is also one example, and unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0111] In the above embodiment, the processing according to the embodiment is realized by a software configuration using a computer by executing a program, but the present invention is not limited to this. The embodiment may be realized by, for example, a hardware configuration or a combination of a hardware configuration and a software configuration. [Explanation of symbols]

[0112] 10. Information processing device 11 CPU 11A Acquisition Department 11B Recognition part 11C generation section 11D Display control unit 11E Changes 11F Detection section 11G Learning Department 12 ROM 13 RAM 14 I / O 15 Storage section 15A Image display program 15B Feature Recognition Model 15C Pre-trained model 15D object detection model 15E Goods information DB 16 Display 17 Control section 18 Communications Department 19 External I / F 30 Camera 31 Placement table 40 Reader 100 Image display system

Claims

1. an acquisition unit that acquires a card image obtained by photographing a card medium having a unique characteristic portion; a recognition unit that recognizes the unique characteristic portion from the card image acquired by the acquisition unit; a generation unit that uses a trained model generated by machine learning a group of training data in which a card image obtained by photographing the card medium in advance and an article image obtained by photographing an article in advance are paired, and generates an optimal article image for the article for the card image in which the unique characteristic portion is recognized by the recognition unit; A display control unit that controls displaying the item image generated by the generation unit on a display unit instead of the card image acquired by the acquisition unit; An information processing device comprising:

2. The recognition unit recognizes the unique characteristic portion from the card image acquired by the acquisition unit using a feature recognition model generated by machine learning a group of card images obtained by photographing a plurality of types of card media having different unique characteristic portions and associating the group with a label representing the unique characteristic portion. The information processing device according to claim 1 .

3. A change unit changes a label representing the unique feature portion recognized by the recognition unit to a label representing the article of the article image generated by the generation unit, The display control unit controls to display a label representing the item changed by the change unit on the display unit together with the item image. The information processing device according to claim 2 .

4. The display control unit further controls the display unit to display product information including a product name and a price of the product based on the label representing the product changed by the change unit. The information processing device according to claim 3 .

5. A detection unit that detects an article by applying an object detection technique to the article image generated by the generation unit, The display control unit controls displaying, on the display unit, item information including an item name and a price of the item based on a detection result by the detection unit.

4. The information processing device according to claim 1.

6. The detection unit detects the items from the item images generated by the generation unit using an item detection model generated by machine learning a group of item images obtained by photographing a plurality of types of items in advance, the group being associated with labels representing the items. The information processing device according to claim 5 .

7. The item detection model is a model that has been retrained by adding item images included in the training data group. The information processing device according to claim 6.

8. The trained model is a model of a generative adversarial network. The information processing device according to any one of claims 1 to 7.

9. A computer comprising: Acquiring a card image obtained by photographing a card medium having a unique characteristic portion; Recognizing the unique feature portion from the acquired card image; using a trained model generated by machine learning a group of training data in which a card image obtained by photographing the card medium in advance and an article image obtained by photographing an article in advance are paired, an optimal article image for the article is generated for the card image in which the unique characteristic portion is recognized; and performing control to display the generated item image on a display unit instead of the acquired card image. Image display method.

10. Acquiring a card image obtained by photographing a card medium having a unique characteristic portion; Recognizing the unique feature portion from the acquired card image; using a trained model generated by machine learning a group of training data in which a card image obtained by photographing the card medium in advance and an article image obtained by photographing an article in advance are paired, an optimal article image for the article is generated for the card image in which the unique characteristic portion is recognized; Controlling to display the generated item image on a display unit instead of the acquired card image, An image display program for execution by a computer.

Images