Information processing system, information processing method, and program

Abstract

Instead of attaching price tags to product shelves, customers will be able to view prices and product information for each item using their mobile devices. [Solution] In the information processing system, the imaging means generates an image of the product's appearance. The product recognition means performs image recognition on the image and recognizes the product included in the image. The price acquisition means acquires the price of the recognized product. The display control means displays a price overlay screen on the display unit, with the price superimposed on the image.

Description

Technical Field

[0004] , , ,

[0005] , , ,

[0001] The present invention relates to the display of product information using a mobile terminal.

Background Art

[0002] In stores, systems have been proposed for reading products using mobile terminals such as smartphones owned by customers. For example, Patent Document 1 proposes a method of reading a barcode attached to a product using a portable information terminal with a camera and displaying the price of the product. Further, Patent Document 2 describes a method of taking a picture of a product at a cash register for settlement of the product and displaying the picture of the purchased product, the product name, unit price, quantity, total amount, etc. of each product.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In ordinary stores, for products displayed on product shelves, price tags are attached to the product shelves, so customers can know the price of each product by looking at the price tags. However, for the store side, the labor of creating price tags for individual products or attaching price tags to product shelves is significant. In particular, when new products are displayed, product prices are changed, or tax rates are changed, it is necessary to add or update the price tags of the target products, which is a great burden for the store.

[0005] The objective of the present invention is to provide an information processing system that enables the realization of a store where price tags are not attached to product shelves, and customers can view product prices and product information using their mobile devices. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing system, A reading means that reads a first code information corresponding to the store visited by the customer, and a second code information attached to the products displayed in the store, A display control means that, when the first code information and the second code information are read, displays a first screen containing detailed information of the product on the customer's terminal device, It is equipped with.

[0007] Another aspect of the present invention is an information processing method performed by a computer, The system reads a first code information corresponding to the store the customer visited, and also reads a second code information attached to the products displayed in that store. When the first code information and the second code information are read, a first screen containing detailed information about the product is displayed on the customer's terminal device.

[0008] Another aspect of the present invention is a program, The system reads a first code information corresponding to the store the customer visited, and also reads a second code information attached to the products displayed in that store. When the first code information and the second code information are read, the computer is instructed to perform a process to display a first screen containing detailed information about the product on the customer's terminal device. [Effects of the Invention]

[0009] According to the present invention, it is possible to realize a store where price tags are not attached to product shelves, and customers can view product prices and product information using their mobile devices. [Brief explanation of the drawing]

[0010] [Figure 1] The configuration of the information processing system according to the embodiment is shown. [Figure 2] The following shows the general configuration of the image processing server. [Figure 3] An example of a product information data structure is shown. [Figure 4] Here are some other examples of product information data structures. [Figure 5] This shows the general configuration of a mobile device. [Figure 6] This document describes a method for obtaining price information using captured images. [Figure 7] An example of a product details screen is shown. [Figure 8] This shows an example of a product details screen that compares multiple products. [Figure 9] This demonstrates how to obtain product information using barcodes. [Figure 10] This is a flowchart of the display process. [Figure 11] This shows an example of a message displayed when there are many items in the captured image. [Figure 12] Here are some other examples of the product details screen. [Modes for carrying out the invention]

[0011] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. [Overall System Configuration] FIG. 1 shows the overall configuration of an information processing system 100 according to this embodiment. The information processing system 100 includes an image processing server 2 and a mobile terminal 3. A customer purchases a product at a store 8. At the store 8, products are displayed and sold on a product shelf 4. Here, in a normal store, price tags are attached to the product shelves, and customers can know the prices of the products by looking at the price tags. In contrast, in this embodiment, it is assumed that no price tags are attached to the product shelves 4 in the store 8. That is, only the respective products are simply displayed on the product shelves 4. In this embodiment, medicine is used as an example of a product for explanation. As shown in the figure, different types of medicines A to H and the like are displayed on the product shelf 4.

[0012] In this embodiment, since no price tags are attached to the product shelves 4, customers use the mobile terminal 3 to obtain price information and detailed information about the products (hereinafter referred to as "product detailed information"). The "product detailed information" is information on product specifications and includes, for example, at least any one of the product name, manufacturer name, price, raw materials, place of origin, capacity, usage method, efficacy, and images of the product or product package. Specifically, the customer can view the price information and product detailed information by displaying the product on the mobile terminal 3 within the store 8. In addition, the customer can also view the product detailed information by using the mobile terminal 3 to read the barcode attached to the product.

[0013] The image processing server 2 can communicate with the mobile terminal 3 via a network or the like, stores information necessary for the mobile terminal 3 for display control, and transmits display information necessary for the mobile terminal 3 to display various screens to the mobile terminal 3. The image processing server 2 may obtain information necessary for the mobile terminal 3 for display control from an external device each time the mobile terminal performs display control, or may periodically obtain it from an external device and store it internally. Also, the image processing server 2 may store only a part of the information necessary for the mobile terminal for display control internally and obtain the remaining information from an external device. The image processing server 2 in this embodiment is described on the premise that it periodically obtains information necessary for the mobile terminal 3 for display control from an external device or has previously stored information necessary for the mobile terminal 3 for display control internally.

[0014] The mobile terminal 3 is a terminal used by customers who shop at the store 8. It has functions such as imaging products displayed on the product shelf 4, reading product barcodes attached to products, obtaining and displaying price information and product details by communicating with the image processing server 2. In FIG. 1, for convenience of explanation, only one mobile terminal 3 is shown, but in reality, there are as many as the number of customers who have entered the store 8. The mobile terminal 3 may be a terminal such as a smartphone owned by the customer, or may be a store-owned terminal temporarily lent to the customer when shopping at the store 8.

[0015]

[0015] [Device Configuration] Next, the configurations of the image processing server 2 and the mobile terminal 3 will be described with reference to FIGS. 2 to 3. (Image Processing Server) FIG. 2 shows the schematic configuration of the image processing server 2. The image processing server 2 includes a storage unit 23, a communication unit 24, and a control unit 25.

[0016] The storage unit 23 is composed of a memory such as a hard disk or a flash memory. The storage unit 23 stores the programs executed by the control unit 25 and the information necessary for the control unit 25 to execute a predetermined process by executing the programs. When the storage unit 23 is provided inside the image processing server 2, it is connected to the communication unit 24 and the control unit 25 via the bus line 20. Instead of providing the storage unit 23 inside the image processing server 2, the information used by the image processing server 2 may be stored in an external device outside the image processing server 2, and the control unit 25 may obtain the information from the external device and perform processing. In that case, the external device is connected to the communication unit 24 and the control unit 25 via a fiber channel, a LAN cable, or the like.

[0017] Specifically, the memory unit 23 stores customer information, product information, and image feature information. "Customer information" refers to information about individual customers. In this embodiment, in order to receive a service that displays product information using a mobile terminal 3 at one of the stores 8 belonging to the chain, customers register as users with the chain headquarters in advance. The memory unit 23 stores information such as the customer ID, the terminal ID of the mobile terminal 3 used by the customer, the customer's past purchase history, and membership rank as customer information for the registered customer. The membership rank is set based on the customer's past purchase history at that store or the entire chain, and generally, the higher the past purchase amount, the higher the membership rank.

[0018] "Product information" refers to information about each product, including price information and detailed product information for each product identification information such as product ID. "Price information" shows the price of each product. Note that product prices are determined by each store, and the price information includes the store's price for each product. "Detailed product information" is information about the product specifications, as mentioned above, and includes at least one of the following: product name, manufacturer name, price, raw materials, place of origin, volume, usage, efficacy, and images of the product or product packaging. Detailed product information is associated with product identification information. Product identification information is information used to identify the type of product, such as the JAN code shown by the barcode attached to the product or the product name.

[0019] Figure 3 shows an example of a product information data structure. In this example, product information linking product identification information, price information, and product details is prepared for each store and stored in the storage unit 23. Figure 4 shows another example of a product information data structure. In this example, product information is divided into price information and product details and stored in the storage unit 23. As shown in Figure 4(A), the price information shows the price of each product for each store. On the other hand, as shown in Figure 4(B), product details are stored for each product.

[0020] "Image feature information" refers to information about the features used for image recognition of products. Specifically, image feature information is information that indicates the features extracted through image processing from images of the appearance of each product sold at store 8. Image feature information is stored in association with product identification information. Alternatively, the image of the product itself may be stored instead of image feature information.

[0021] The communication unit 24 communicates with the mobile terminal 3 via wired or wireless communication. The control unit 25 includes a CPU, ROM, RAM, etc. (not shown) and performs various controls on each component within the image processing server 2. The processing performed by the control unit 25 will be described later.

[0022] In the above configuration, the image processing server 2 is an example of an information processing device, the communication unit 24 is an example of a receiving means and a transmitting means, the storage unit 23 is an example of a price storage unit, and the control unit 25 is an example of a product recognition means and a price acquisition means.

[0023] (Mobile device) Figure 5 shows the schematic configuration of the mobile terminal 3. The mobile terminal 3 comprises a display unit 31, an input unit 32, a storage unit 33, a communication unit 34, a control unit 35, and a camera 36. These elements are interconnected via a bus line 30.

[0024] The display unit 31 displays images of products captured by the camera 36, ​​product prices, product details, etc., based on the control of the control unit 35. The input unit 32 is an interface that accepts user input, and may include, for example, a touch panel, buttons, or a voice input device.

[0025] The storage unit 33 is composed of memory such as a hard disk or flash memory. The storage unit 33 stores the program executed by the control unit 35, and the information necessary for the control unit 35 to execute predetermined processes by executing the program. For example, the storage unit 33 stores a dedicated application program (hereinafter referred to as the "product information display application") that is activated when shopping at store 8 and controls the display of price information and product details.

[0026] The communication unit 34 communicates with the image processing server 2, for example, via communication through a WiFi (registered trademark) device installed in the store. The control unit 35 includes a CPU, ROM, RAM, etc. (not shown) and performs various controls on each component within the mobile terminal 3. The controls performed by the control unit 35 will be described later. In the above configuration, the camera 36 is an example of an imaging means, a reading means, and a store identification information acquisition means, and the control unit 35 is an example of a display control means and a customer identification information acquisition means.

[0027] Camera 36 is used to image products. When a customer sets camera 36 to shooting mode and images products displayed on the shelves, the imaged products are displayed on the display unit 31. Here, "image capture" refers to the state in which an image of a product is captured within the shooting range of camera 36, ​​and it is not necessary to store or record that image as image data on the mobile terminal 3. In other words, when a product on the shelves is imaged by camera 36, ​​that product is displayed on the display unit 31, but if the customer changes the position or orientation of the mobile terminal 3, a different product will be imaged and that product will be displayed on the display unit 31. Hereinafter, images captured by the mobile terminal 3 will be referred to as "imaged images".

[0028] Furthermore, camera 36 also functions as a barcode reader. The aforementioned product information display application operates as a barcode reader when the captured image contains a barcode, and acquires product identification information such as a JAN code from the barcode captured by camera 36 and outputs it to the control unit 35.

[0029] [How to display product information] Next, we will explain how to display product information. (Method using captured images) (1) Display of price overlay screen In this embodiment, as described above, the product shelves 4 in store 8 do not have price tags, and customers obtain product information using a mobile terminal 3. Figure 6 shows a method for obtaining price information using captured images. The customer activates the camera 36 by operating the product information display application installed on the mobile terminal 3 and takes a picture of a product of interest with the mobile terminal 3. As shown in the example in Figure 6, when the customer takes a picture of the area indicated by the dashed line 42 on the product shelf 4 with the mobile terminal 3, the mobile terminal 3 displays the captured image of the area indicated by the dashed line 42, as shown in the display screen 61, and also transmits the captured image to the image processing server 2. When the image processing server 2 receives the captured image from the mobile terminal 3, it recognizes each product included in the captured image, obtains the price information for each product, and transmits it to the mobile terminal 3. The mobile terminal 3 displays the received price information superimposed on the corresponding product. As a result, as shown in the display screen 62, a screen (referred to as the "price superimposed screen") is displayed on the mobile terminal 3 with the price 51 of each product displayed on it. In this way, when a customer takes a picture of an item they are interested in with their mobile device 3, the price of the item included in the captured image is displayed overlaid on each item. Therefore, even if there are no price tags on the shelves 4, customers can use their mobile device 3 to find out the price of the item they are interested in.

[0030] (2) Display of product details screen If the same product is displayed on the price overlay screen for a predetermined period of time, the mobile terminal 3 displays a message 52 to inform the customer that the product can be selected, as illustrated in Figure 7(A). When the customer taps on any product on the screen to select it, the product details screen for the selected product is displayed. Specifically, when the customer selects a product, the mobile terminal 3 sends the product identification information of the selected product to the image processing server 2. The image processing server 2 retrieves the product details of the product corresponding to the received product identification information from the storage unit 23 and sends it to the mobile terminal 3. The mobile terminal 3 generates a product details screen including the received product details and displays it on the display unit 31.

[0031] Figure 7(B) shows an example of a product details screen. This example shows the case where drug B is selected on the price overlay screen. The product details screen 63 includes the product's package image 63a and detailed information about the product 63b. Note that the package image 63a is not an image captured by the mobile terminal 3, but an image that has been stored in advance as part of the product details in the storage unit 23 of the image processing server 2. In this way, customers can view detailed information about a product by selecting the product they are interested in on the price overlay screen.

[0032] Furthermore, as shown in Figure 7(A), when message 52 is displayed and the customer selects multiple products, the mobile terminal 3 displays a product details screen that compares and displays the product details of the multiple products. Specifically, the mobile terminal 3 sends the product identification information of the multiple products selected by the customer to the image processing server 2. The image processing server 2 retrieves the product details of the products corresponding to the received product identification information from the storage unit 23 and sends them to the mobile terminal 3. The mobile terminal 3 generates a product details screen containing the product details of the multiple products received and displays it on the display unit 31.

[0033] Figure 8 shows an example of a product details screen that compares multiple products. Figure 8 shows product details screen 64 when drug B and drug G are selected as multiple products. In this way, when a customer selects multiple products, the mobile device 3 displays a product details screen that compares the product details of those products. In this case, it is preferable to limit the number of products that can be displayed for comparison at one time so that the product details screen does not become difficult to read.

[0034] If the image captured by the mobile terminal 3 contains only one product, it can be assumed that the product has been selected, and the mobile terminal 3 can display the product details screen for that product without displaying the price overlay screen. Specifically, the mobile terminal 3 sends the captured image to the image processing server 2, and the image processing server 2 recognizes the products contained in the captured image. The image processing server 2 then determines whether or not there is only one product in the captured image, and if there is only one product in the captured image, it retrieves the product details for that product from the storage unit 23 and sends it to the mobile terminal 3. The mobile terminal 3 generates a product details screen containing the received product details and displays it on the display unit 31.

[0035] (Using barcodes) In this embodiment, in addition to the method of acquiring product information using captured images as described above, product information can also be acquired using barcodes. Figure 9 shows the method of acquiring product information using barcodes. The customer operates the product information display application on the mobile terminal 3 to activate the camera 36 and captures a barcode 43 printed or affixed to the product package 41. When the mobile terminal 3 determines that the captured image contains a barcode, it functions as a barcode reader, reads the barcode, acquires product identification information, and transmits it to the image processing server 2. Based on the product identification information, the image processing server 2 acquires product details of the corresponding product from the storage unit 23 and transmits it to the mobile terminal 3. The mobile terminal 3 generates a product details screen 65 including the received product details and displays it on the display unit 31. Note that the barcode to be read from the product may be a one-dimensional barcode as exemplified in Figure 9, or a two-dimensional barcode such as a QR code (registered trademark). A barcode is an example of code information.

[0036] [Display Processing] Next, the display processing by the information processing system 100 will be described. The display processing is the process by which a customer displays product information using the mobile terminal 3 at store 8. Figure 10 is a flowchart of the display processing. The display processing is realized by the image processing server 2 and the mobile terminal 3 each executing pre-prepared programs. As a prerequisite, it is assumed that the customer has registered as a user with the chain headquarters to which store 8 belongs in advance in order to receive the product display service using the mobile terminal 3 at store 8, and that the customer's information is registered as customer information in the storage unit 23 of the image processing server 2. It is also assumed that the customer has installed a product information display application on the mobile terminal 3 for displaying products at store 8.

[0037] First, upon entering store 8, the customer launches the product information display application installed on mobile terminal 3 and uses mobile terminal 3 to scan the store barcode (step S101). The store barcode is affixed to a designated location within store 8, for example, on a wall near the store entrance or on a card used for shopping. When the customer scans the store barcode, mobile terminal 3 obtains the store ID from the scanned barcode and transmits it to image processing server 2 (step S102). When transmitting any information to image processing server 2, mobile terminal 3 also transmits the customer ID or the terminal ID of mobile terminal 3 to identify itself as needed. Image processing server 2 receives the store ID from mobile terminal 3 and identifies the store (step S103).

[0038] In step S101 above, the mobile terminal 3 identifies the store by reading the store's barcode. Alternatively, the store may be identified using GPS information indicating the location of the mobile terminal 3, and the store ID may be sent to the image processing server 2. Furthermore, instead of the mobile terminal 3 sending store information to the image processing server 2, the image processing server 2 may identify the store where the customer is located by identifying which store's Wi-Fi is being used to transmit the information from the mobile terminal 3 to the image processing server 2.

[0039] Next, the mobile terminal 3 uses the camera 36 to determine whether or not the product barcode attached to the product has been read (step S201). If the product barcode has been read (step S201: Yes), the mobile terminal 3 transmits the product identification information of the scanned product to the image processing server 2 based on the scanned product barcode (step S202). The image processing server 2 receives the product identification information from the mobile terminal 3 (step S203), refers to the storage unit 23 to obtain the product details of that product, and transmits them to the mobile terminal 3 (step S204). If the product details of each product are stored in the storage unit 23 for each store, in step S204, the image processing server 2 obtains the product details associated with the store identified in step S103 and the product identification information received in step S203 from the storage unit 23 and transmits them to the mobile terminal 3. Following step S204, the mobile terminal 3 receives the product details and displays the product details screen on the display unit 31 (step S205). Thus, as explained with reference to Figure 9, product information is displayed using the barcode, and the process proceeds to step S206. Note that even if the product barcode is not read (step S201: No), the process still proceeds to step S206.

[0040] In step S206, the mobile terminal 3 determines whether or not an image of the product has been captured using the camera 36. If an image has been captured (step S206: Yes), the mobile terminal 3 sends the captured image to the image processing server 2 (step S207). The image processing server 2 receives the captured image from the mobile terminal 3 (step S208) and uses the image feature information in the storage unit 23 to obtain product identification information of the product contained in the captured image.

[0041] Next, the image processing server 2 refers to the product information in the storage unit 23 that corresponds to the store identified in step S103, obtains the price of each product corresponding to the product identification information obtained in step S208, and transmits it to the mobile terminal 3 (step S209). The mobile terminal 3 receives the price of each product included in the captured image and displays it on the display unit 31 (step S210). In this way, as explained with reference to Figure 6, a price overlay screen is displayed on the mobile terminal 3 using the captured image of the products. Then the process proceeds to step S211. Even if no captured image is displayed (step S206: No), the process proceeds to step S211. Note that if the image processing server 2 contains only one product in the captured image received in step S208, it only needs to obtain the product identification information of that product based on the captured image, refer to the product information corresponding to the store identified in step S103 to obtain the product details of that product, and transmit them to the mobile terminal 3. In this case, the mobile terminal 3 displays a product details screen that includes the product details of that product.

[0042] In step S211, the mobile terminal 3 determines whether a product has been selected by the customer on the price overlay screen. If a product is selected (step S211: Yes), the mobile terminal 3 sends information identifying the selected product (hereinafter referred to as "selected product information") to the image processing server 2 (step S212). Since the image processing server 2 has already identified each product when the price overlay screen is displayed, the mobile terminal 3 can send the product identification information of the selected product as the selected product information. Alternatively, the mobile terminal 3 may use information such as the location of the selected product on the image screen as the selected product information.

[0043] The image processing server 2 receives the selected product information from the mobile terminal 3 (step S213), refers to the storage unit 23 to obtain the product details of the selected product, and transmits them to the mobile terminal 3 (step S214). If the product details of each product are stored in the storage unit 23 for each store, in step S214, the image processing server 2 refers to the product information in the storage unit 23 corresponding to the store identified in step S103 to obtain the product details. If multiple products are selected, the image processing server 2 transmits the product details of all of those products to the mobile terminal 3.

[0044] Next, the mobile terminal 3 receives the product details of the selected product and displays the product details screen on the display unit 31 (step S215). Thus, as explained with reference to Figures 7 and 8, the product details screen of the product selected by the customer is displayed on the price overlay screen.

[0045] Next, the mobile terminal 3 determines whether the product information display application has been terminated by the customer's operation (step S216). If the product information display application has been terminated (step S216: Yes), the process ends. On the other hand, if the product information display application has not been terminated (step S216: No), the process returns to step S201.

[0046] As described above, with the product information display processing of this embodiment, customers can obtain price information, product details, etc., by reading a barcode using the mobile terminal 3 or by capturing an image of the product. Therefore, even if there are no price tags on the product shelves in the store, customers can easily obtain the information necessary to purchase a product.

[0047] [Differentiation] Next, modifications of this embodiment will be described. The following modifications can be implemented in appropriate combinations. (1) Processing when there are many products in the captured image If a large number of products are captured at once by the mobile terminal 3, the area for each product on the price overlay screen becomes smaller, which may make the price overlay screen difficult to view. Therefore, if there are many products in the captured image, the mobile terminal 3 will display a message requesting that the number of products to be captured be reduced before displaying the price overlay screen. Figure 11(A) shows an example of a message displayed when there are many products in the captured image. Let's assume that the maximum number of products that can be displayed at once on the price overlay screen (hereinafter referred to as the "maximum number") is 6. In the example in Figure 11(A), the mobile terminal 3 has captured more products than the maximum number. In this case, the mobile terminal 3 displays a message 56 requesting that the number of products be reduced.

[0048] Specifically, the mobile terminal 3 transmits the image captured by the camera 36 to the image processing server 2. The image processing server 2 recognizes the products from the captured image, and if the number of recognized products exceeds the upper limit, it notifies the mobile terminal 3 of this fact. Upon receiving this notification, the mobile terminal 3 displays a message 56 requesting that the number of products be reduced, as illustrated in Figure 11(A).

[0049] Alternatively, instead of using the maximum number of items that can be displayed at once, a maximum size for the area occupied by each item in the captured image may be set. For example, the maximum size for each item may be set to 1 / 6 of the total area of ​​the captured image. In this case, if the size of an individual item recognized from the captured image exceeds the maximum size, the image processing server 2 will notify the mobile terminal 3 accordingly. Upon receiving this notification, the mobile terminal 3 will display a message 56 requesting a reduction in the number of items, as illustrated in Figure 11(A).

[0050] Furthermore, if the number of products in the captured image is large, instead of asking the customer to reduce the number of products as described above, the customer may be asked to select one of the products. Figure 11(B) shows an example of displaying a message asking the customer to select one of the products. In this example, since the number of products in the captured image exceeds the upper limit of 6, message 57 is displayed asking the customer to select one of the products in the captured image. In this case, the specific process is as follows: the mobile terminal 3 sends the captured image to the image processing server 2. The image processing server 2 recognizes the products from the captured image, and if the number of recognized products exceeds the upper limit, or if the size of individual products exceeds the upper limit, it notifies the mobile terminal 3 accordingly. Upon receiving this notification, the mobile terminal 3 displays message 57 asking the customer to select one of the products, as illustrated in Figure 11(B). When message 57 is displayed and the customer selects one of the products, the mobile terminal 3 displays a product details screen for the selected product, as illustrated in Figure 7(B). Alternatively, instead of asking the customer to select one of the products, the customer may be prompted to approach the product shelf, and the number of products displayed on the mobile terminal 3 may be reduced to or below the upper limit. In this case, mobile device 3 should display a message such as, "Please move closer to the products to reduce the number of products displayed on the screen to six or fewer."

[0051] (2) Customer-specific price changes In this embodiment, customers who use the mobile terminal 3 to view product information at store 8 have registered as users in advance with the chain headquarters to which the store belongs. Therefore, the price of products may be changed for each customer. In this case, the storage unit 23 of the image processing server 2 stores different price information for each store and for each customer or customer rank. For example, in addition to a standard price list applicable to general customers, a special price list for valued customers with a high purchase history at that store may be stored. Alternatively, customer ranks may be set based on purchase history, and price lists for each rank may be stored. Instead of preparing price lists for valued customers and each rank, the discount rates corresponding to valued customers and each customer rank may be stored.

[0052] In actual processing, when the image processing server 2 receives a customer ID from the customer's mobile terminal 3, it refers to the customer information in the storage unit 23 to determine whether the customer is a regular customer, a premium customer, or their membership rank. Then, the image processing server 2 refers to the product information corresponding to the store and the customer stored in the storage unit 23, determines the price of each product, and transmits it to the mobile terminal 3.

[0053] (3) Information displayed on the product details screen The product details screens illustrated in Figures 7(B) and 8 display detailed product information for each item, but sales information may also be added. For example, as shown in Figure 12(A), a coupon mark 53 can be added to the product details screen to inform customers of the existence of coupons that can be used for that product. The coupons applicable to each product can be determined based on the customer's purchase history and membership rank. In another example, sales information for a predetermined period and information such as the number of units sold for that product may be added to the product details screen. In this case, the image processing server 2 can access a POS server that manages sales for the entire chain or individual stores for each product, for example, to obtain information such as sales and the number of units sold for that product. Alternatively, sales information and the number of units sold for each product may be periodically sent from the POS server to the image processing server 2 and used. In the example in Figure 12(B), recent sales information 54 for that product is displayed.

[0054] In addition, some or all of the above embodiments (including modifications, the same applies hereinafter) may also be described as follows, but are not limited to the following.

[0055] (Note 1) An information processing system, An imaging means for generating an image of the product's appearance, Product recognition means that performs image recognition on the captured image and recognizes the products included in the captured image, A price acquisition means for obtaining the price of a recognized product, A display control means that displays a price superimposed screen, in which the aforementioned price is superimposed on the captured image, on the display unit, An information processing system equipped with the following features.

[0056] (Note 2) The information processing system described in Appendix 1, wherein the display control means obtains product details of the selected product when a product is selected on the price overlay screen, and displays a product details screen including the product details on the display unit.

[0057] (Note 3) The information processing system described in Appendix 2, wherein when multiple products are selected on the price overlay screen, the display control means displays a product details screen on the display unit that compares and displays the product details of the multiple selected products.

[0058] (Note 4) Equipped with a reading means for reading code information attached to the product, The information processing system according to any one of the appendices 1 to 3, wherein the display control means obtains product details of a product corresponding to the read code information and displays a product details screen including the product details on the display unit.

[0059] (Note 5) The aforementioned product details screen is an information processing system as described in any one of the appendices 2 to 4, which includes coupon information determined for each customer or for each customer rank.

[0060] (Note 6) The aforementioned product details screen is an information processing system described in any one of the items in Appendix 2 to 5, which includes sales information or the number of units sold for the product for a specified period.

[0061] (Note 7) The information processing system according to any one of the appendices 1 to 6, wherein the display control means displays a message on the display unit requesting a reduction in the number of items included in the captured image if the size of each item included in the captured image is greater than or equal to a predetermined size, or if the number of items included in the captured image is greater than or equal to a predetermined number.

[0062] (Note 8) The information processing system according to any one of the appendices 1 to 6, wherein the display control means displays a message on the display unit requesting the selection of products included in the captured image when the size of individual products included in the captured image is greater than or equal to a predetermined size, or when the number of products included in the captured image is greater than or equal to a predetermined number, and when a product is selected, displays a product details screen on the display unit including product details of the selected product.

[0063] (Note 9) A means for obtaining customer identification information, The system includes a price storage unit that stores different prices for each customer or for each customer rank, The price acquisition means is an information processing system according to any one of the appendices 1 to 8, which acquires the price corresponding to the customer from the price storage unit based on the customer's identification information.

[0064] (Note 10) A means for obtaining store identification information, The system includes a price storage unit that stores different prices for each store, The price acquisition means is an information processing system according to any one of the appendices 1 to 9, which acquires the price corresponding to the store from the price storage unit based on the store identification information.

[0065] (Note 11) The product recognition means detects the number of products included in the captured image, The information processing system according to any one of the appendices 1 to 10, wherein the display control means displays a product details screen including product details of the product on the display unit when there is one product, and displays a price overlay screen on the display unit when there are multiple products.

[0066] (Note 12) An information processing device capable of communicating with a terminal device, A receiving means for receiving an image of the product's appearance captured from the terminal device, Product recognition means that performs image recognition on the captured image and recognizes the products included in the captured image, A price acquisition means for obtaining the price of a recognized product, A transmission means for generating a price superimposed screen by superimposing the aforementioned price onto the captured image and transmitting it to the terminal device, An information processing device equipped with the following features.

[0067] (Note 13) The terminal device receives an image of the product's appearance, The image recognition of the captured image is performed to recognize the products included in the captured image. Get the price of the recognized product, An information processing method for generating a price superimposition screen by superimposing the aforementioned price onto the captured image, and transmitting it to the terminal device.

[0068] (Note 14) The terminal device receives an image of the product's appearance, The image recognition of the captured image is performed to recognize the products included in the captured image. Get the price of the recognized product, A program that causes a computer to execute a process to generate a price superimposition screen by superimposing the aforementioned price onto the captured image and transmit it to the terminal device.

[0069] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made that are understandable to those skilled in the art within the scope of the present invention. That is, the present invention naturally includes the full disclosure, including the claims, and various modifications and alterations that those skilled in the art could make in accordance with the technical idea. Furthermore, each disclosure of the above-mentioned patent documents and other references is incorporated herein by reference. [Explanation of symbols]

[0070] 2 Image processing server 3 Mobile devices 4 Product shelves 8 stores 25 Control Unit 36 Cameras 100 Information Processing Systems

Claims

1. An information processing system, A reading means that reads a first code information corresponding to the store visited by the customer, and a second code information attached to the products displayed in the store, A display control means that, when the first code information and the second code information are read, displays a first screen containing detailed information of the product on the customer's terminal device, An information processing system equipped with the following features.

2. The information processing system according to claim 1, wherein the display control means, after the first code information and the second code information have been read, obtains an image of the appearance of the product, and then displays a second screen on the terminal device that further includes the price of the product at the store.

3. The information processing system according to claim 2, wherein when a product other than the product is selected on the second screen, the display control means displays a third screen containing detailed information of the other product on the terminal device instead of the first screen and the second screen.

4. A method of information processing performed by a computer, The system reads a first code information corresponding to the store the customer visited, and also reads a second code information attached to the products displayed in that store. An information processing method that, when the first code information and the second code information are read, displays a first screen containing detailed information about the product on the customer's terminal device.

5. The system reads a first code information corresponding to the store the customer visited, and also reads a second code information attached to the products displayed in that store. A program that causes a computer to perform a process to display a first screen containing detailed information of the product on the customer's terminal device when the first code information and the second code information are read.

Images