Information processing system, information processing method, and program

Abstract

To enable customers to view the price and product information of each product using a mobile terminal, without giving price tags on the product shelves.SOLUTION: In an information processing system, imaging means generates a captured image of the appearance of a product, product recognition means performs image recognition of the captured image and recognizes the product contained in the captured image, price acquiring means acquires the price of the recognized product, and display control means displays a price-superimposed screen in which the price is superimposed on the captured image on a display unit.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to displaying product information using a mobile terminal. [Background technology]

[0002] A system has been proposed for reading products in stores using mobile terminals such as smartphones owned by customers. For example, Patent Document 1 proposes a method for reading barcodes attached to products using a mobile information terminal with a camera and displaying the product price. Patent Document 2 describes a method for taking images of products at a cash register where the payment for products is made, and displaying images of the purchased products, the product name, unit price, quantity, total price, etc. of each product. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2004-310274 A [Patent Document 2] JP 2012-043371 A Summary of the Invention [Problem to be solved by the invention]

[0004] In a typical store, price tags are attached to the products displayed on the shelves, so customers can know the price of each product by looking at the price tag. However, it takes a lot of effort for the store to create price tags for each product and attach price tags to the product shelves. In particular, when new products are displayed, the price of a product changes, or the tax rate changes, it is necessary to add or update the price tags of the applicable products, which places a heavy burden on the store.

[0005] An object of the present invention is to provide an information processing system that makes it possible to realize a store in which customers can view product prices and product information using mobile terminals without attaching price tags to product shelves. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing system, An imaging means for generating an image of an external appearance of a product; a product recognition means for performing image recognition on the captured image and recognizing a product included in the captured image; a price obtaining means for obtaining a price of the recognized item; a display control means for displaying on a display unit a price superimposition screen in which the price is superimposed on the captured image, and then, when one product is selected on the price superimposition screen, displaying on the display unit a product detail screen including product detail information of the one product and a mark for informing the customer of the existence of a coupon that can be used for the one product; Equipped with.

[0007] Another aspect of the present invention is a computer-implemented information processing method, comprising: receiving an image of the exterior of the product from a terminal device; Performing image recognition on the captured image, and recognizing the product included in the captured image; Get the price of the recognized item, A price superimposition screen in which the price is superimposed on the captured image is generated and transmitted to the terminal device, and then when an item is selected on the price superimposition screen, a product details screen is generated that includes product information about the item and a mark for informing the customer of the existence of a coupon that can be used for the item, and transmitted to the terminal device.

[0008] Another aspect of the present invention is a program, receiving an image of the exterior of the product from a terminal device; Performing image recognition on the captured image, and recognizing the product included in the captured image; Get the price of the recognized item, The computer is caused to execute a process of generating a price superimposition screen in which the price is superimposed on the captured image and transmitting the price superimposition screen to the terminal device, and then, when an item is selected on the price superimposition screen, generating a product details screen including product information for the item and a mark for informing the customer of the existence of a coupon that can be used for the item, and transmitting the product details screen to the terminal device. Effect of the Invention

[0009] According to the present invention, it is possible to realize a store in which price tags are not attached to product shelves and customers can view product prices and product information using mobile terminals. [Brief description of the drawings]

[0010] [Figure 1] 1 shows a configuration of an information processing system according to an embodiment. [Diagram 2] 2 shows a schematic configuration of an image processing server. [Diagram 3] 1 shows an example of a data structure of product information. [Figure 4] 13 shows another example of the data structure of product information. [Diagram 5] 2 shows a schematic configuration of a mobile terminal. [Figure 6] A method for obtaining price information using captured images is shown. [Figure 7] An example of a product details screen is shown. [Figure 8] 13 shows an example of a product details screen for comparing multiple products. [Figure 9] This shows a method for obtaining product information using a barcode. [Figure 10] 13 is a flowchart of a display process. [Figure 11] 13 shows an example of a message display when a captured image contains a large number of products. [Figure 12] 13 shows another example of the product details screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. [Overall system configuration] FIG. 1 shows the overall configuration of an information processing system 100 according to this embodiment. The information processing system 100 includes an image processing server 2 and a mobile terminal 3. A customer purchases a product in a store 8. In the store 8, the products are displayed on a product shelf 4 and sold. In a normal store, the product shelf has a price tag, and a customer can know the price of the product by looking at the price tag. In contrast, in this embodiment, it is assumed that the product shelf 4 in the store 8 does not have a price tag. That is, each product is simply displayed on the product shelf 4. Note that in this embodiment, a description will be given using medicine as an example of a product. As shown in the figure, different types of medicines A to H are displayed on the product shelf 4.

[0012] In this embodiment, since there are no price tags on the product shelves 4, customers use the mobile terminal 3 to obtain price information and detailed information about the products (hereinafter referred to as "product details information"). Note that "product details information" refers to information about product specifications, and includes at least one of the following: product name, manufacturer name, price, ingredients, place of origin, capacity, usage, efficacy, and images of the product or product packaging. Specifically, customers can view price information and product details information by displaying the product on the mobile terminal 3 within the store 8. Note that customers can also view product details information by using the mobile terminal 3 to read a barcode attached to the product.

[0013] The image processing server 2 can communicate with the mobile terminal 3 via a network or the like, and the mobile terminal 3 stores information necessary for display control, and transmits display information necessary for the mobile terminal 3 to display various screens to the mobile terminal 3. The image processing server 2 may acquire information necessary for the display control of the mobile terminal 3 from an external device every time the mobile terminal performs display control, or may periodically acquire the information from the external device and store it internally. The image processing server 2 may internally store only a portion of the information necessary for the display control of the mobile terminal, and acquire the remaining information from an external device. The image processing server 2 of this embodiment is described on the assumption that the mobile terminal 3 periodically acquires information necessary for display control from an external device, or that the mobile terminal 3 stores information necessary for display control internally in advance.

[0014] The mobile terminal 3 is a terminal used by customers who shop at the store 8, and has functions such as capturing images of products displayed on the product shelves 4, reading product barcodes attached to the products, and acquiring and displaying price information and detailed product information by communicating with the image processing server 2. For ease of explanation, only one mobile terminal 3 is shown in FIG. 1, but in reality there are as many mobile terminals as there are customers in the store 8. The mobile terminal 3 may be a terminal such as a smartphone owned by the customer, or may be a store-owned terminal that is temporarily lent to the customer when shopping at the store 8.

[0015] [Device configuration] Next, the configurations of the image processing server 2 and the mobile terminal 3 will be described with reference to FIGS. (Image processing server) 2 shows a schematic configuration of the image processing server 2. The image processing server 2 includes a storage unit 23, a communication unit 24, and a control unit 25.

[0016] The storage unit 23 is configured by a memory such as a hard disk or a flash memory. The storage unit 23 stores a program executed by the control unit 25, and information necessary for the control unit 25 to execute a predetermined process by executing the program. When the storage unit 23 is provided in the image processing server 2, it is connected to the communication unit 24 and the control unit 25 via the bus line 20. Instead of providing the storage unit 23 in the image processing server 2, information used by the image processing server 2 may be stored in an external device outside the image processing server 2, and the control unit 25 may acquire the information from the external device and perform the process. In this case, the external device is connected to the communication unit 24 and the control unit 25 via a fiber channel, a LAN cable, or the like.

[0017] Specifically, the storage unit 23 stores customer information, product information, and image feature information. "Customer information" is information about an individual customer. In this embodiment, in order to receive a service of displaying product information using a mobile terminal 3 at one store 8 belonging to a chain store, the customer performs user registration with the chain headquarters in advance. The storage unit 23 stores information about the registered customer, such as the customer ID, the terminal ID of the mobile terminal 3 used by the customer, the customer's past purchase history, and member rank, as customer information. The member rank is set based on the customer's past purchase history at that store and the entire chain of stores, and generally, the higher the past purchase amount, the higher the member rank.

[0018] "Product information" is information about each product, and includes price information and product details information for each product identification information such as a product ID. "Price information" indicates the price of each product. Product prices are determined for each store, and the price information includes the price for each product for each store. "Product details information" is information about product specifications, as described above, and includes at least one of the following: product name, manufacturer name, price, ingredients, place of origin, capacity, usage, efficacy, and images of the product or product packaging. The product details information is associated with product identification information. Product identification information is information for identifying the type of product, and is, for example, the JAN code or product name indicated by a barcode attached to the product.

[0019] FIG. 3 shows an example of the data structure of product information. In this example, product information linking product identification information, price information, and product detail information is prepared for each store and stored in the storage unit 23. FIG. 4 shows another example of the data structure of product information. In this example, product information is divided into price information and product detail information and stored in the storage unit 23. As shown in FIG. 4(A), the price information indicates the price of each product for each store. Meanwhile, as shown in FIG. 4(B), the product detail information is stored for each product.

[0020] "Image feature information" is information on features used for image recognition of products. Specifically, the image feature information is information indicating features extracted by image processing from an image of the exterior of each product sold at the store 8. The image feature information is stored in association with the product identification information. Note that instead of the image feature information, the image of the product itself may be stored.

[0021] The communication unit 24 communicates with the mobile terminal 3 by wired or wireless communication. The control unit 25 includes a CPU, a ROM, a RAM, etc. (not shown), and performs various controls on the components in the image processing server 2. The processing executed by the control unit 25 will be described later.

[0022] In the above configuration, the image processing server 2 is an example of an information processing device, the communication unit 24 is an example of a receiving means and a transmitting means, the memory unit 23 is an example of a price memory unit, and the control unit 25 is an example of a product recognition means and a price acquisition means.

[0023] (Mobile devices) 5 shows a schematic configuration of the mobile terminal 3. The mobile terminal 3 includes a display unit 31, an input unit 32, a storage unit 33, a communication unit 34, a control unit 35, and a camera 36. These elements are connected to each other via a bus line 30.

[0024] The display unit 31 displays product images captured by the camera 36, ​​product prices, detailed product information, etc., under the control of the control unit 35. The input unit 32 is an interface that accepts user input, and corresponds to, for example, a touch panel, a button, a voice input device, etc.

[0025] The storage unit 33 is composed of a memory such as a hard disk or a flash memory. The storage unit 33 stores programs executed by the control unit 35 and information necessary for the control unit 35 to execute predetermined processes by executing the programs. For example, the storage unit 33 stores a dedicated application program (hereinafter referred to as a "product information display application") that is activated when shopping at the store 8 and controls the display of price information and detailed product information.

[0026] The communication unit 34 communicates with the image processing server 2, for example, via a WiFi (registered trademark) device installed in the store. The control unit 35 includes a CPU, ROM, RAM, etc. (not shown), and performs various controls on the components in the mobile terminal 3. The controls performed by the control unit 35 will be described later. In the above configuration, the camera 36 is an example of an imaging means, a reading means, and a store identification information acquisition means, and the control unit 35 is an example of a display control means and a customer identification information acquisition means.

[0027] The camera 36 is used to capture images of products. When a customer sets the camera 36 to a shooting mode and captures an image of a product displayed on a product shelf, the captured image of the product is displayed on the display unit 31. Here, "capturing an image" refers to a state in which an image of a product is captured within the shooting range of the camera 36, ​​and it is not necessary to store or record the image as image data in the mobile terminal 3. In other words, when an image of a product on a product shelf is captured by the camera 36, ​​the product is displayed on the display unit 31, but when the customer changes the position or orientation of the mobile terminal 3, a different product is captured and the captured product is displayed on the display unit 31. Hereinafter, an image captured by the mobile terminal 3 is referred to as a "captured image."

[0028] The camera 36 also functions as a barcode reader. When the captured image includes a barcode, the above-mentioned product information display application operates as a barcode reader, obtains product identification information such as a JAN code from the barcode captured by the camera 36, ​​and outputs the information to the control unit 35.

[0029] [Product information display method] Next, a method for displaying product information will be described. (Method using captured images) (1) Display of price overlay screen In this embodiment, as described above, the product shelves 4 in the store 8 do not have price tags, and customers obtain product information using the mobile terminal 3. FIG. 6 shows a method of obtaining price information using a captured image. The customer operates the product information display application installed on the mobile terminal 3 to start the camera 36, ​​and captures an image of a product of interest with the mobile terminal 3. As shown in the example of FIG. 6, when the customer captures an image of the range indicated by the dashed line 42 on the product shelf 4 with the mobile terminal 3, the mobile terminal 3 displays the captured image of the range of the dashed line 42 as shown on the display screen 61, and transmits the captured image to the image processing server 2. When the image processing server 2 receives the captured image from the mobile terminal 3, it performs image recognition of each product included in the captured image, obtains price information of each product, and transmits it to the mobile terminal 3. The mobile terminal 3 displays the received price information by superimposing it on the corresponding product. As a result, a screen (called a "price superimposed screen") in which the price 51 of the product is displayed on each product as shown on the display screen 62 is displayed on the mobile terminal 3. In this way, when a customer takes an image of a product that interests them with the mobile terminal 3, the price of the product included in the captured image is displayed superimposed on each product. Therefore, even if there are no price tags on the product shelves 4, the customer can use the mobile terminal 3 to find out the price of the product that interests them.

[0030] (2) Display of product details screen When a predetermined time has elapsed with the same product displayed on the price superimposition screen, the mobile terminal 3 displays a message 52 informing the customer that a product can be selected, as exemplified in FIG. 7(A). When the customer taps on one of the products on the screen to select it, a product details screen for the selected product is displayed. Specifically, when the customer selects a product, the mobile terminal 3 transmits product identification information of the selected product to the image processing server 2. The image processing server 2 acquires product details information of the product corresponding to the received product identification information from the storage unit 23 and transmits it to the mobile terminal 3. The mobile terminal 3 generates a product details screen including the received product details information, and displays it on the display unit 31.

[0031] FIG. 7(B) is an example of a product details screen. This example is an example in which medicine B is selected on the price superimposed screen. The product details screen 63 includes a package image 63a of the product and detailed information 63b about the product. Note that the package image 63a is not an image captured by the mobile terminal 3, but an image that is stored in advance as part of the product details information in the storage unit 23 of the image processing server 2. In this way, a customer can view detailed information about a product in which he or she is interested by selecting the product on the price superimposed screen.

[0032] 7(A), when a customer selects multiple products while message 52 is displayed, the mobile terminal 3 displays a product details screen that comparatively displays product detailed information about the multiple products. Specifically, the mobile terminal 3 transmits product identification information of the multiple products selected by the customer to the image processing server 2. The image processing server 2 acquires product detailed information of the product corresponding to the received product identification information from the storage unit 23 and transmits it to the mobile terminal 3. The mobile terminal 3 generates a product details screen including the received product detailed information of the multiple products, and displays it on the display unit 31.

[0033] Fig. 8 shows an example of a product details screen for comparing multiple products. Fig. 8 shows a product details screen 64 when medicine B and medicine G are selected as multiple products. In this way, when a customer selects multiple products, a product details screen that displays detailed product information for those products in a comparative manner is displayed on the mobile terminal 3. In this case, it is preferable to set a limit on the number of products that can be displayed for comparison at one time so that the product details screen does not become difficult to view.

[0034] If only one product is included in the captured image by the mobile terminal 3, it can be considered that the product has been selected, and the mobile terminal 3 may display a product details screen for that product without displaying a price superimposition screen. Specifically, the mobile terminal 3 transmits the captured image to the image processing server 2, and the image processing server 2 performs image recognition of the product included in the captured image. The image processing server 2 then determines whether the captured image contains only one product, and if the captured image contains only one product, obtains product details information for that product from the storage unit 23 and transmits it to the mobile terminal 3. The mobile terminal 3 generates a product details screen including the received product details information, and displays it on the display unit 31.

[0035] (Method using barcodes) In this embodiment, in addition to the method of acquiring product information using a captured image as described above, product information can also be acquired using a barcode. FIG. 9 shows a method of acquiring product information using a barcode. A customer operates a product information display application on the mobile terminal 3 to start the camera 36 and captures a barcode 43 printed or attached to a package 41 of a product. When the mobile terminal 3 determines that the captured image includes a barcode, it functions as a barcode reader, reads the barcode, acquires product identification information, and transmits it to the image processing server 2. Based on the product identification information, the image processing server 2 acquires product details information of the corresponding product from the storage unit 23 and transmits it to the mobile terminal 3. The mobile terminal 3 generates a product details screen 65 including the received product details information, and displays it on the display unit 31. Note that the barcode read from the product may be a one-dimensional barcode exemplified in FIG. 9, or a two-dimensional barcode such as a QR code (registered trademark). The barcode is an example of code information.

[0036] [Display processing] Next, the display process by the information processing system 100 will be described. The display process is a process in which a customer uses the mobile terminal 3 in the store 8 to display product information. FIG. 10 is a flowchart of the display process. The display process is realized by the image processing server 2 and the mobile terminal 3 executing programs prepared in advance. It is assumed that a customer has previously registered as a user with the chain headquarters to which the store 8 belongs in order to receive a product display service using the mobile terminal 3 in the store 8, and that the customer's information is registered as customer information in the storage unit 23 of the image processing server 2. It is also assumed that the customer has installed a product information display application for displaying products in the store 8 in the mobile terminal 3.

[0037] First, when a customer enters the store 8, the customer starts a product information display application installed on the mobile terminal 3 and reads a store barcode using the mobile terminal 3 (step S101). The store barcode is attached to a predetermined location in the store 8, for example, a wall near the entrance of the store or a card used for shopping. When the customer reads the store barcode, the mobile terminal 3 obtains a store ID from the read store barcode and transmits it to the image processing server 2 (step S102). When transmitting any information to the image processing server 2, the mobile terminal 3 transmits a customer ID or a terminal ID of the mobile terminal 3 to identify itself as necessary. The image processing server 2 receives the store ID from the mobile terminal 3 and identifies the store (step S103).

[0038] In the above step S101, the mobile terminal 3 identifies the store by reading the store barcode, but instead, the mobile terminal 3 may identify the store using GPS information indicating the position of the mobile terminal 3 and transmit the store ID to the image processing server 2. Also, instead of the mobile terminal 3 transmitting store information to the image processing server 2, the image processing server 2 may identify the store in which the customer is located by identifying the store's WiFi from which the information from the mobile terminal 3 is transmitted to the image processing server 2.

[0039] Next, the mobile terminal 3 determines whether or not the product barcode attached to the product is read using the camera 36 (step S201). If the product barcode is read (step S201: Yes), the mobile terminal 3 transmits product identification information of the read product to the image processing server 2 based on the read product barcode (step S202). The image processing server 2 receives the product identification information from the mobile terminal 3 (step S203), acquires product details of the product by referring to the storage unit 23, and transmits the product details to the mobile terminal 3 (step S204). If the product details of each product are stored in the storage unit 23 for each store, the image processing server 2 acquires the store specified in step S103 and the product details associated with the product identification information received in step S203 from the storage unit 23 in step S204, and transmits the product details to the mobile terminal 3. Following step S204, the mobile terminal 3 receives the product details information and displays a product details screen on the display unit 31 (step S205). In this manner, as described with reference to Fig. 9, the product information is displayed using the barcode, and the process proceeds to step S206. If the product barcode has not been read (step S201: No), the process also proceeds to step S206.

[0040] In step S206, the mobile terminal 3 determines whether or not a captured image of the product has been acquired using the camera 36. If a captured image has been acquired (step S206: Yes), the mobile terminal 3 transmits the captured image to the image processing server 2 (step S207). The image processing server 2 receives the captured image from the mobile terminal 3 (step S208), and acquires product identification information of the product included in the captured image using the image feature information in the storage unit 23.

[0041] Next, the image processing server 2 refers to the product information corresponding to the store identified in step S103 among the product information in the storage unit 23, acquires the price of each product corresponding to the product identification information acquired in step S208, and transmits it to the mobile terminal 3 (step S209). The mobile terminal 3 receives the price of each product included in the captured image, and displays it on the display unit 31 (step S210). In this way, as described with reference to FIG. 6, a price superimposition screen is displayed on the mobile terminal 3 using the captured image of the product. Then, the process proceeds to step S211. If the captured image is not displayed (step S206: No), the process also proceeds to step S211. Note that, if the captured image received in step S208 contains only one product, the image processing server 2 acquires the product identification information of the product based on the captured image, acquires the product information corresponding to the store identified in step S103, and transmits the product information to the mobile terminal 3. In this case, the mobile terminal 3 displays a product detail screen including the product detailed information of the product.

[0042] In step S211, the mobile terminal 3 determines whether or not a product has been selected by the customer on the price superimposition screen. If a product has been selected (step S211: Yes), the mobile terminal 3 transmits information identifying the selected product (hereinafter referred to as "selected product information") to the image processing server 2 (step S212). Note that, since the image processing server 2 has already identified each product at the time the price superimposition screen is displayed, the mobile terminal 3 can transmit product identification information of the selected product as selected product information. Note that, instead of product identification information, the mobile terminal 3 may use, for example, position information of the selected product on the captured screen as the selected product information.

[0043] The image processing server 2 receives the selected product information from the mobile terminal 3 (step S213), refers to the storage unit 23, acquires product detail information of the selected product, and transmits it to the mobile terminal 3 (step S214). If product detail information of each product for each store is stored in the storage unit 23, the image processing server 2 acquires the product detail information in step S214 by referring to the product information in the storage unit 23 corresponding to the store identified in step S103. If multiple products are selected, the image processing server 2 transmits the product detail information of the multiple products to the mobile terminal 3.

[0044] Next, the mobile terminal 3 receives the product detail information of the selected product and displays the product detail screen on the display unit 31 (step S215). In this way, the product detail screen of the product selected by the customer is displayed on the price superimposed screen, as described with reference to Figs. 7 and 8.

[0045] Next, the mobile terminal 3 determines whether the product information display application has been terminated by the customer's operation (step S216). If the product information display application has been terminated (step S216: Yes), the process ends. On the other hand, if the product information display application has not been terminated (step S216: No), the process returns to step S201.

[0046] As described above, according to the product information display process of this embodiment, a customer can obtain price information, detailed product information, and the like by reading a barcode or capturing an image of a product using the mobile terminal 3. Therefore, even if there is no price tag on the product shelf in the store, the customer can easily obtain the information necessary to purchase the product.

[0047] [Variations] Next, modified examples of the present embodiment will be described. The following modified examples can be implemented in appropriate combination. (1) Processing when a captured image contains a large number of products If the mobile terminal 3 captures a large number of products at one time, the area of ​​each product on the price superimposition screen becomes smaller, which may make the price superimposition screen difficult to see. For this reason, when the captured image contains a large number of products, the mobile terminal 3 displays a message requesting the user to reduce the number of products captured before displaying the price superimposition screen. FIG. 11(A) shows an example of a message display when the captured image contains a large number of products. Assume that the upper limit of the number of products that can be displayed at one time on the price superimposition screen (hereinafter referred to as the "upper limit number") is six. In the example of FIG. 11(A), the mobile terminal 3 captures products that exceed the upper limit number. In this case, the mobile terminal 3 displays a message 56 requesting the user to reduce the number of products.

[0048] As a specific process, the mobile terminal 3 transmits an image captured by the camera 36 to the image processing server 2. The image processing server 2 recognizes products from the captured image, and if the number of recognized products exceeds the upper limit, notifies the mobile terminal 3 of that fact. Upon receiving the notification, the mobile terminal 3 displays a message 56 requesting the user to reduce the number of products, as exemplified in Fig. 11(A).

[0049] Instead of using the upper limit number of products that can be displayed at one time, an upper limit size of the area that each product occupies in the captured image may be determined. For example, the upper limit size of each product may be set to 1 / 6 of the area of ​​the entire captured image. In this case, if the size of each product recognized from the captured image is equal to or larger than the upper limit size, the image processing server 2 notifies the mobile terminal 3 of this fact. Upon receiving this notification, the mobile terminal 3 displays a message 56 requesting the user to reduce the number of products, as exemplified in FIG. 11(A).

[0050] In addition, when the number of products included in the captured image is large, instead of requesting the customer to reduce the number of products as described above, the customer may be requested to select one of the products. FIG. 11(B) is an example of displaying a message requesting the customer to select one of the products. In this example, since the number of products included in the captured image exceeds the upper limit of six, a message 57 is displayed requesting the customer to select one of the products included in the captured image. In this case, as a specific process, the mobile terminal 3 transmits the captured image to the image processing server 2. The image processing server 2 recognizes products from the captured image, and if the number of recognized products exceeds the upper limit or if the size of each product exceeds the upper limit, the mobile terminal 3 is notified of this to the mobile terminal 3. Upon receiving the notification, the mobile terminal 3 displays a message 57 requesting the customer to select one of the products as exemplified in FIG. 11(B). When the customer selects one of the products while the message 57 is displayed, the mobile terminal 3 displays a product details screen for the selected product as exemplified in FIG. 7(B). Instead of requesting the customer to select one of the products, the customer may be prompted to approach the product shelf, and the number of products displayed on the mobile terminal 3 may be reduced to or below the upper limit. In this case, the mobile terminal 3 may display a message such as "Please move closer to the products to reduce the number of products displayed on the screen to six or less."

[0051] (2) Price changes for individual customers In this embodiment, a customer who uses the mobile terminal 3 to display product information at a store 8 has previously registered as a user with the chain headquarters to which the store belongs. Therefore, the price of a product may be changed for each customer. In this case, the storage unit 23 of the image processing server 2 stores different price information for each store and for each customer or for each customer rank. For example, in addition to a standard price list applied to general customers, a special price list for good customers who have a high purchase history at the store may be stored. Also, a member rank may be set for a customer based on the purchase history, and a price list for each member rank may be stored. Note that instead of preparing a price list for good customers and each member rank, a discount rate corresponding to good customers and each customer rank may be stored.

[0052] In actual processing, when the image processing server 2 receives a customer ID from the customer's mobile terminal 3, it refers to the customer information in the storage unit 23 to determine whether the customer is a general customer or a preferred customer, or the customer's membership rank. Then, the image processing server 2 refers to the product information corresponding to the store and the customer stored in the storage unit 23, determines the price of each product, and transmits it to the mobile terminal 3.

[0053] (3) Display information on the product details screen The product details screens exemplified in FIG. 7(B) and FIG. 8 are screens that display detailed product information for each product, but sales information may be added to them. For example, as shown in FIG. 12(A), a coupon mark 53 may be added to the product details screen to inform the customer of the existence of a coupon that can be used for that product. The coupon that can be applied to each product may be determined according to the customer's purchase history and membership rank. In another example, sales information for a predetermined period of time and information such as the number of units sold may be added to the product details screen. In this case, the image processing server 2 may access a POS server that manages the sales of each product in the entire chain or for each store, and obtain information such as the sales and number of units sold of the product. Information such as the sales and number of units sold of each product may be periodically transmitted from the POS server to the image processing server 2, and the information may be used. In the example of FIG. 12(B), recent sales information 54 of the product is displayed.

[0054] In addition, some or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following notes.

[0055] (Appendix 1) An information processing system, An imaging means for generating an image of an external appearance of a product; a product recognition means for performing image recognition on the captured image and recognizing a product included in the captured image; a price obtaining means for obtaining a price of the recognized item; a display control means for displaying on a display unit a price superimposition screen in which the price is superimposed on the captured image; An information processing system comprising:

[0056] (Appendix 2) The information processing system described in Appendix 1, wherein the display control means, when a product is selected on the price overlay screen, obtains product details information for the selected product and displays a product details screen including the product details information on the display unit.

[0057] (Appendix 3) The information processing system according to claim 2, wherein the display control means, when multiple products are selected on the price overlay screen, displays a product details screen on the display unit that comparatively displays product information about the selected multiple products.

[0058] (Appendix 4) A reading means for reading code information attached to the product is provided, The information processing system according to any one of appendices 1 to 3, wherein the display control means obtains product details information for a product corresponding to the read code information, and displays a product details screen including the product details information on the display unit.

[0059] (Appendix 5) The information processing system according to any one of appendices 2 to 4, wherein the product details screen includes coupon information determined for each customer or for each customer rank.

[0060] (Appendix 6) The information processing system according to any one of appendixes 2 to 5, wherein the product details screen includes sales information or sales quantity of the product for a specified period.

[0061] (Appendix 7) The information processing system described in any one of Appendices 1 to 6, wherein the display control means displays a message on the display unit requesting the user to reduce the number of products included in the captured image if the size of each product included in the captured image is equal to or larger than a predetermined size, or if the number of products included in the captured image is equal to or larger than a predetermined number.

[0062] (Appendix 8) The information processing system of any one of appendices 1 to 6, wherein the display control means displays a message on the display unit requesting the selection of a product included in the captured image when the size of each product included in the captured image is equal to or larger than a predetermined size, or when the number of products included in the captured image is equal to or larger than a predetermined number, and when a product is selected, displays a product details screen on the display unit including product information about the selected product.

[0063] (Appendix 9) A customer identification information acquisition means for acquiring customer identification information; a price storage unit that stores different prices for each of the customers or for each rank of the customers, 9. The information processing system according to any one of claims 1 to 8, wherein the price acquisition means acquires a price corresponding to the customer from the price memory unit based on the customer's identification information.

[0064] (Appendix 10) A store identification information acquisition means for acquiring store identification information; A price storage unit that stores different prices for each store, 10. The information processing system according to any one of claims 1 to 9, wherein the price acquisition means acquires a price corresponding to a store from the price storage unit based on identification information of the store.

[0065] (Appendix 11) The commodity recognition means detects the number of commodities included in the captured image, An information processing system as described in any one of Appendices 1 to 10, wherein the display control means displays a product details screen including product information about the product on the display unit when the number of products is one, and displays the price overlay screen on the display unit when the number of products is multiple.

[0066] (Appendix 12) An information processing device capable of communicating with a terminal device, A receiving means for receiving an image of an external appearance of a product from the terminal device; a product recognition means for performing image recognition on the captured image and recognizing a product included in the captured image; a price obtaining means for obtaining a price of the recognized item; a transmission means for generating a price superimposed screen in which the price is superimposed on the captured image and transmitting the price superimposed screen to the terminal device; An information processing device comprising:

[0067] (Appendix 13) receiving an image of the exterior of the product from a terminal device; Performing image recognition on the captured image, and recognizing the product included in the captured image; Get the price of the recognized item, An information processing method for generating a price superimposed screen in which the price is superimposed on the captured image, and transmitting the price superimposed screen to the terminal device.

[0068] (Appendix 14) receiving an image of the exterior of the product from a terminal device; Performing image recognition on the captured image, and recognizing the product included in the captured image; Get the price of the recognized item, A program that causes a computer to execute a process of generating a price superimposition screen in which the price is superimposed on the captured image and transmitting the price superimposition screen to the terminal device.

[0069] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated herein by reference. [Explanation of symbols]

[0070] 2. Image Processing Server 3. Mobile devices 4 Product shelves 8 stores 25 Control Unit 36 Camera 100 Information Processing Systems

Claims

1. an imaging means for generating an image of the exterior of a product; a product recognition means for performing image recognition on the captured image and recognizing a product included in the captured image; a price obtaining means for obtaining a price of the recognized product; a display control means for displaying on a display unit a price superimposition screen in which the price is superimposed on the captured image, and then, when one product is selected on the price superimposition screen, displaying on the display unit a product detail screen including product detail information of the one product and a mark for informing the customer of the existence of a coupon that can be used for the one product; An information processing system comprising:

2. 2. The information processing system according to claim 1, wherein the display control means determines the coupon to be applied to the one product based on the customer's purchase history or the customer's rank.

3. The information processing system according to claim 1, wherein the display control means displays the product details screen on the display unit, the product details screen further including information relating to sales of the one product for a specified period of time or information relating to the number of units sold of the one product.

4. The information processing system of claim 1, wherein the display control means displays a string of characters on the price superimposition screen informing the customer that one of the products included in the price superimposition screen can now be selected when a predetermined time has elapsed while the same product is displayed on the price superimposition screen, and when one of the products is selected while the string of characters is displayed, displays the product details screen on the display unit.

5. 1. A computer-implemented information processing method, comprising: receiving an image of the exterior of the product from the terminal device; Performing image recognition on the captured image, recognizing the product included in the captured image; Get the price of the recognized item, An information processing method that generates a price superimposition screen in which the price is superimposed on the captured image and transmits it to the terminal device, and then, when a product is selected on the price superimposition screen, generates a product detail screen that includes product detail information for the product and a mark for informing the customer of the existence of a coupon that can be used for the product and transmits it to the terminal device.

6. receiving an image of the exterior of the product from the terminal device; Performing image recognition on the captured image, recognizing the product included in the captured image; Get the price of the recognized item, A program that causes a computer to execute a process of generating a price superimposition screen in which the price is superimposed on the captured image and transmitting it to the terminal device, and then, when a product is selected on the price superimposition screen, generating a product details screen that includes product details information for the product and a mark to inform the customer of the existence of a coupon that can be used for the product and transmitting this to the terminal device.

Images