Product purchase support device and product purchase support method

Abstract

This system provides support to enable people with limited mobility, such as wheelchair users, to shop smoothly in physical stores. [Solution] A product purchase support device comprising: a filtering information acquisition unit that acquires either location information within the sales floor or a photograph of the scenery or products within the sales floor as filtering information from a portable information processing device carried by a customer in a store; an image recognition unit that, when a photograph is acquired, narrows down the range of the sales floor or products based on matching it with a scenery image or product image; a product list generation unit that determines one or more products based on the location information and the narrowed range of the sales floor or product image, generates a product list relating to the determined products, and provides information relating to each product in a viewable format; and a purchase request product notification unit that accepts the selection of products that the customer wishes to purchase from the products listed in the product list and notifies the store staff of the products the customer wishes to purchase.

Description

Technical Field

[0001] This invention relates to a product purchase support device and a product purchase support method for assisting customers in a store, particularly customers with disabilities, when purchasing products in the store.

Background Art

[0002] In Japan, the revised Act on Elimination of Discrimination against Persons with Disabilities was implemented on April 1, 2024, and businesses are obligated to provide reasonable accommodation to people with disabilities. For example, a business operator operating a physical store, when there is an indication from a disabled customer that they "want to remove social barriers" in individual situations during the operation of the business, is required to take necessary and reasonable measures within the scope where the burden associated with its implementation is not excessive. On the other hand, a shortage of manpower in the operation of physical stores has become a social problem, and it is not easy to provide reasonable accommodation to disabled people with manpower. There is a desire for a new mechanism that aims to provide reasonable accommodation to disabled people while minimizing the human burden. Regarding this, in order to make shopping in a physical store smoother, a product purchase system has been proposed that directly connects from obtaining product information to comparison, purchase, and settlement using an application program (app) of a financial institution that serves as a payment institution installed on the user's mobile terminal (for example, see Patent Document 1). The app functions as an imaging means for imaging the images of products displayed in the physical store, a position information acquisition means for acquiring the position information of the mobile terminal at the time of imaging, and a product information acquisition means for acquiring the current price and product information of the imaged product based on the imaged product image and the position information of the mobile terminal. Further, the app functions as a product information display means for displaying product information including the obtained current price, and a settlement means for connecting to a financial institution server capable of executing the settlement method according to the settlement method pre-registered by the user and the position information of the mobile terminal to perform the settlement of the product.

Prior Art Documents

Patent Documents

[0003] [Patent Document 1] Japanese Patent Publication No. 2024-001579 [Overview of the project] [Problems that the invention aims to solve]

[0004] However, for example, customers with disabilities or the elderly who use wheelchairs often have difficulty reaching items placed on the upper or lower shelves, or items located at the back of the shelves, when purchasing products in a physical store. Even if they make the effort to visit a physical store to check the actual product before purchasing, there is a lack of an environment that allows them to select and purchase products that meet their expectations. This invention was made in consideration of the circumstances described above, and provides a mechanism to support wheelchair users and others who have a limited range of reach for products compared to able-bodied individuals, enabling them to shop smoothly in physical stores. [Means for solving the problem]

[0005] This invention provides a product purchase support device comprising: a filtering information acquisition unit that acquires at least one of the following as filtering information from a portable information processing device held by a customer in a store: location information within the sales floor, or a photograph of the scenery or products within the sales floor; an image recognition unit that, when a photograph is acquired as filtering information, narrows down the range of the sales floor or products based on matching it with a photograph of the scenery or products within the sales floor; a product list generation unit that determines one or more products based on at least one of the location information and the narrowed range of the sales floor, or based on the product images, generates a product list relating to the determined products, and provides information relating to each product in a viewable format; and a purchase request product notification unit that accepts the selection of products that the customer wishes to purchase from among the products listed in the product list, and notifies the store staff of the selected products as products to be purchased.

[0006] Furthermore, from a different perspective, this invention provides a product purchase support method comprising the steps of: a computer acquiring location information within the sales floor, or at least one of the scenery within the sales floor or a photograph of products within the sales floor, as filtering information from a portable information processing device held by a customer in a store; when a photograph is acquired as the filtering information, narrowing down the range of the sales floor or products based on matching it with the scenery image or product image of the sales floor; determining one or more products based on at least one of the location information and the narrowed range of the sales floor, or based on the product image, generating a product list relating to the determined products and providing information relating to each product in a viewable format; accepting the selection of products that the customer wishes to purchase from among the products listed in the product list; and notifying the store staff of the selected products as products to be purchased. [Effects of the Invention]

[0007] The product purchase support device according to this invention comprises a product list generation unit that determines one or more products based on location information and a narrowed-down sales area range, or based on product images, generates a product list relating to the determined products, and provides information relating to each product in a viewable format; and a purchase request notification unit that accepts the selection of products that the customer wishes to purchase from the products listed in the product list and notifies the store staff of the selected products as purchase request items. Therefore, it can provide a system that supports people such as wheelchair users, who have a limited range of being able to pick up products compared to able-bodied people, to shop smoothly in physical stores. The product purchase support method according to this invention also produces similar effects. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the configuration of the product purchase support device, portable information processing device, and POS terminal related to this disclosure. [Figure 2] This is an explanatory diagram showing an example of a scene inside a sales area, captured by the portable information processing device related to this disclosure. [Figure 3]This is an explanatory diagram showing an example of an image taken at a relatively close distance of multiple products lined up on a sales floor using the portable information processing device related to this disclosure. [Figure 4] This is an explanatory diagram showing an example of a close-up image of a product being captured using the portable information processing device related to this disclosure. [Figure 5] Figure 1 shows a first flowchart illustrating an example of the processing related to product purchase support performed by the control unit of the product purchase support device. [Figure 6] Figure 1 shows a second flowchart illustrating an example of the processing related to product purchase support performed by the control unit of the product purchase support device. [Figure 7] This is a third flowchart showing an example of the processing related to product purchase support performed by the control unit of the product purchase support device shown in Figure 1. [Figure 8] Figure 1 shows a fourth flowchart illustrating an example of the processing related to product purchase support performed by the control unit of the product purchase support device. [Figure 9] Figure 1 shows a fifth flowchart illustrating an example of the processing related to product purchase support performed by the control unit of the product purchase support device. [Figure 10] Unlike Figure 3, this is an explanatory diagram showing examples of photographs of multiple products of the same type but with different contents and prices. [Figure 11] Figure 2 is an explanatory diagram illustrating an example of a product list generated by the control unit based on the layout of the sales floor. [Figure 12] Figure 3 is an explanatory diagram illustrating an example of a product list generated by the control unit based on images of multiple products, as shown in the diagram. [Figure 13] Figure 10 is an explanatory diagram illustrating an example of a product list generated by the control unit based on images of multiple products. [Modes for carrying out the invention]

[0009] The present invention will be described in further detail below with reference to the drawings. The following description is illustrative in all respects and should not be interpreted as limiting the present invention. Figure 1 is a block diagram showing the configuration of a product purchase support device, a portable information processing device, and a POS terminal according to this disclosure. The product purchase support device 10 shown in Figure 1 is a device that provides product purchase support to customers who visit a physical store to purchase products, particularly customers with disabilities, the elderly, and other customers who use wheelchairs. The specific hardware configuration of the product purchase support device 10 may be a server or a personal computer. The product purchase support device 10 communicates with a portable information processing device 16 carried by the customer in the sales area, a POS terminal 18 used by store staff, and a fixed-position camera 19 permanently installed in the store, and provides services related to product purchase support.

[0010] The product purchase support device 10 comprises a control unit 11, a data storage unit 12, and a communication unit 13. The hardware of the control unit 11 may consist of a data processing circuit centered on a processor. For example, the control unit 11 may be implemented by one or more processing units (CPU (Central Processing Unit), SoC (System on a Chip), etc.). The control unit 11 may also consist of one or more circuits. The hardware of the data storage unit 12 may consist of a non-volatile storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Disk). The hardware of the communication unit 13 may consist of a communication circuit centered on a SoC (System on Chip) for wireless communication.

[0011] As shown in FIG. 1, the control unit 11 includes a narrowing information acquisition unit 11A, a product list generation unit 11L, a purchase desired product notification unit 11N, and an image recognition unit 11R. The narrowing information acquisition unit 11A performs a process of acquiring position information within the sales floor, or a photographed image of the scenery or products within the sales floor, from the portable information processing device 16. The product list generation unit 11L performs a process of generating a product list related to the narrowed-down products. The purchase desired product notification unit 11N performs a process of accepting selection of a product that a customer desires to purchase (a purchase desired product) from the products in the product list and notifying the store staff of the purchase desired product. The control unit 11 may also include, as an optional configuration, a fixed-position image acquisition unit 11F and a fixed-position image selection unit 11S. The fixed-position image acquisition unit 11F performs a process of acquiring an image (a fixed-position image) of the fixed-position camera 19 described later, and the fixed-position image selection unit 11S performs a process of accepting selection of a product using the fixed-position image.

[0012] The portable information processing device 16 is a portable information processing device held by a customer who purchases products at a store. An example of a specific aspect is a smartphone or the like. The portable information processing device 16 includes a portable photographing unit 16A that photographs an image using a camera, a portable communication unit 16C that communicates with external devices including the product purchase support device 10, a portable display unit 16D that displays characters and images, and a portable position information providing unit 16G that provides current position information. Further, it includes a portable storage unit 16M that stores data, a portable control unit 16P that executes processes such as execution, communication, and display of an application, and a portable operation detection unit 16S that detects operations by the customer. The portable control unit 16P functions to provide a user interface of a service related to product purchase support to the customer while communicating with the product purchase support device 10 according to the present disclosure. The service related to the product purchase support may be, for example, downloaded in advance to the portable storage unit 16M as a predetermined application, and the portable control unit 16P may execute the application. Also, for example, the portable control unit 16P may execute a browser, and the service related to the product purchase support may be provided by the portable control unit 16P accessing a predetermined URL provided by the product purchase support device 10.

[0013] The POS terminal 18 communicates with the product purchase support device 10 and is used by staff for payment processing, or by back-office staff to prepare items for payment, including securing items requested by customers as part of product purchase support services. The POS terminal 18 may be a stationary type, a portable type, or a combination of both. The POS terminal 18 includes a terminal communication unit 18C that communicates with external devices, including the product purchase support device 10, and a terminal display unit 18D that displays text and images. Furthermore, it includes a terminal storage unit 18M that stores data, a terminal control unit 18P that performs payment processing, communication, display, etc., and a terminal operation detection unit 18S that detects operations by customers. The fixed-position camera 19 is one or more cameras fixedly installed on the ceiling of the sales floor of the store, and can photograph not only people in the sales floor but also products placed in the sales floor. The fixed-position camera 19 may be installed for surveillance purposes and its images may be reused.

[0014] Next, a specific example of a product purchase support service provided by the product purchase support device 10 shown in Figure 1 will be described. A customer receiving the product purchase support service described herein moves around the store's sales floor in a wheelchair and makes purchases using a portable information processing device 16 that the customer possesses. As shown in Figure 1, the portable information processing device 16 is equipped with a portable location information provision unit 16G and can transmit location information indicating the customer's location within the sales floor (more precisely, the location of the portable information processing device 16) to the product purchase support device 10 at any time. The arrangement of products displayed in the sales floor is stored in the data storage unit 12 as a product arrangement database. The narrowing information acquisition unit 11A of the product purchase support device 10 compares the location information provided by the portable location information provision unit 16G with the product arrangement database and can narrow down the products displayed near the customer, that is, products that the customer is likely to consider purchasing. In this way, location information can be used as narrowing information for narrowing down products (Embodiment 1). According to this embodiment, based on location information acquired from the portable information processing device 16, a product list can be generated and provided to the customer, narrowing down the list to products located within a predetermined range from the customer's location.

[0015] In addition, the portable information processing device 16 includes a portable imaging unit 16A, and can provide an image (captured image) of the scenery inside the sales floor as seen from the customer's position or of products near the customer. The captured image may be a moving image, but a still image that can easily identify the object to be narrowed down is preferred. When receiving such captured images, the narrowing information acquisition unit 11A of the product purchase support device 10 can narrow down products displayed near the customer, that is, products that the customer may consider purchasing, using the received captured images. Thus, captured images of the scenery of the sales floor and nearby products can be used as narrowing information for narrowing down products.

[0016] FIG. 2 is an explanatory diagram showing an example of the scenery inside the sales floor captured by the portable information processing device 16. When a captured image 20 capturing a wide range inside the sales floor as shown in FIG. 2 is obtained, each product appears small and there are many of them, so it may be difficult to recognize individual products from the appearance of the product packages. However, the position inside the sales floor can be estimated from the captured image 20. Images of the scenery at various locations inside the sales floor are stored in the data storage unit 12 in advance as an image database. Then, the image recognition unit 11R of the product purchase support device 10 can collate the captured image 20 with the image database of the sales floor scenery using known image recognition techniques to estimate the position of the customer inside the sales floor. Therefore, even if the position information from the portable position information providing unit 16G cannot be obtained, the narrowing information acquisition unit 11A can narrow down products displayed near the customer, that is, products that the customer may consider purchasing, based on the position of the customer estimated by the image recognition unit 11R. Thus, an image of the scenery inside the sales floor can be used as narrowing information for narrowing down products (Embodiment 2). According to this aspect, the position of the customer can be estimated from the scenery image inside the sales floor obtained from the portable information processing device, and the product list can be generated by narrowing down to products placed within a predetermined range and direction from the estimated position and provided to the customer.

[0017] Furthermore, a narrower area than that shown in Figure 2 may be photographed, for example, a shelf displaying products. If the package images of each product in the sales area are stored in the data storage unit 12 as a product image database beforehand, the image recognition unit 11R can compare the captured image with the product image database using known image recognition techniques to narrow down the individual products shown in the image. In this way, images of products in the sales area can be used as filtering information to narrow down the products (Embodiment 3).

[0018] However, while boxed products are easily identifiable by their appearance, such as the product name and illustrations on the package, there are also products for which it is difficult to narrow down to a specific product based solely on the image. Figure 3 is an explanatory diagram showing an example of an image taken relatively close up of multiple products lined up on a sales floor. The image 30 shown in Figure 3 is a photograph of several types of bento boxes, which are products. The appearance of the bento boxes is similar, making them one of the products for which it is difficult to identify by appearance alone. For products that are difficult to identify by appearance alone, the image can be narrowed down to individual products by combining it with the product placement database described in Embodiment 1 (Embodiment 4).

[0019] Furthermore, it is possible to photograph the product even closer than the image shown in Figure 3, so that the product label for a specific product is clearly captured. Also, it is possible that not only the product itself, but also shelf tags or promotional materials (so-called product POPs) related to the product are photographed. The image 40 shown in Figure 4 is a photograph of a rice ball, and the text and barcode on the product label attached to the product are clearly captured, making it possible to recognize the content using known pattern recognition and character recognition technologies. In the example shown in Figure 4, the contents of the shelf tag are also clearly captured, and the contents of the shelf tag can also be recognized using known character recognition technologies. The contents of product labels, shelf tags, product POPs, etc. are linked to the information of each product and stored in the data storage unit 12 in advance as a product database. Then, the image recognition unit 11R can recognize the information of product labels, shelf tags, and product POPs included in the captured image and compare the recognized information with the product database. Based on the product labels, shelf tags, or product POPs shown in the image, it is possible to identify the related product (Embodiment 5). According to this embodiment, if the acquired image is determined to be a product label, shelf tag, or product POP (point of purchase) required by law or self-regulation, the information on that product label, shelf tag, or product POP can be recognized to narrow down the product selection. Furthermore, information related to the selected product can be provided to the customer.

[0020] As described above, even for products that are out of reach or difficult to reach for customers using wheelchairs, customers can use their portable information processing device 16 to select products they are interested in and display the information on the device. Furthermore, similar to online shopping, the device can accept operations such as selecting the products they wish to purchase on the screen of the portable information processing device 16 and adding the selected products to a "shopping cart." This allows customers to entrust the securing of products to store staff in the back room without having to carry the actual products themselves. In addition, payment for purchased products can be made on the screen of the portable information processing device 16.

[0021] Examples of processes performed by the product purchase support device. Next, we will describe an example of the processing performed by the product purchase support device 10 shown in Figure 1. Figures 5 to 9 are flowcharts showing examples of product purchase support processing performed by the control unit 11 of the product purchase support device 10. The processing procedure will be described according to the flowchart.

[0022] In the flowchart shown in Figure 5, the control unit 11 determines whether or not it has received location information from the customer's portable information processing device 16 via the communication unit 13 (step S11 in Figure 5). This is the location information described in Embodiment 1. If location information is acquired (Yes in step S11), the control unit 11, acting as the narrowing information acquisition unit 11A, determines the customer's location within the sales floor based on the acquired location information (step S12). Then, it compares this with the product placement database pre-stored in the data storage unit 12 and attempts to narrow down the products within a predetermined range from the customer (step S31). Then, it determines whether it was possible to narrow down the products (step S32). If the customer's location could not be determined, or if there were no products within the predetermined range, or conversely, if there were too many products, and it was not possible to narrow down the products (No in step S32), it sends a request to the customer's portable information processing device 16 to display the following message: "Products could not be narrowed down. Please move to another location or take a picture of another location" (step S34). Then, the process returns to step S11 and waits for further location information to be received.

[0023] On the other hand, if location information is not acquired in step S11 (No. of step S11), the control unit 11, acting as the refinement information acquisition unit 11A, determines whether or not a captured image has been acquired from the customer's portable information processing device 16 (step S13). If no captured image has been acquired either (No. of step S13), the control unit 11 returns the process to step S11 and waits for location information or a captured image to be sent from the portable information processing device 16.

[0024] If a captured image is acquired in step S11 (Yes in step S13), the control unit 11, acting as the filtering information acquisition unit 11A, determines whether the acquired captured image is a scene inside the sales floor (step S14). If the captured image is a scene inside the sales floor (Yes in step S14), the control unit, acting as the image recognition unit 11R, compares the captured image with an image database of scenes inside the sales floor stored in the data storage unit 12 to estimate the customer's location (step S15). This process corresponds to Embodiment 2. If the location inside the sales floor can be estimated from the captured image as a result of the comparison (Yes in step S16), the control unit 11, acting as the filtering information acquisition unit 11A, proceeds to step S31 to compare the estimated location with the product placement database and narrows down the products located within a predetermined range in the direction of the captured image from that location. On the other hand, if the location inside the sales floor cannot be estimated from the captured image (No in step S16), a request is sent to the customer's portable information processing device 16 to display the following message. The message is "Please take more pictures of the current location" (step S17), and the process returns to step S11 as described above, waiting for further images to be received.

[0025] If, as determined in step S14, the acquired image is not a scene of the sales floor (No in step S14), the control unit 11, acting as the refinement information acquisition unit 11A, then determines whether the acquired image is an image of one or more products (step S21). If the acquired image is an image of one or more products (Yes in step S21), the control unit, acting as the image recognition unit 11R, compares the acquired image with the product image database stored in the data storage unit 12 and narrows down the products (step S22). Then, the process proceeds to step S32 to determine whether the products have been narrowed down.

[0026] If, in step S21, the captured image is not an image of one or more products (No in step S21), the control unit 11, acting as the filtering information acquisition unit 11A, then determines whether the captured image is a product display label, shelf tag, or product POP (step S23 in Figure 6). If the captured image is a product display label, shelf tag, or product POP (Yes in step S23), the control unit 11, acting as the filtering information acquisition unit 11A, recognizes the content of the product display label, shelf tag, or product POP through pattern recognition processing or character recognition processing. The recognized content is then compared with information in the product database stored in the data storage unit 12 to narrow down the related products (step S24). The process then proceeds to step S32 (see Figure 5) to determine whether the products have been successfully narrowed down.

[0027] If the determination in step S32 above has allowed the products to be narrowed down (step S32), the control unit 11, acting as the narrowing information acquisition unit 11A, determines whether the narrowed-down products are of the same type but differ in content volume and price from product to product, such as fresh food (step S41 in Figure 7). Figure 10 is an explanatory diagram showing an image 100 of packaged fresh fish fillets as an example of multiple products of the same type but with variations in content volume and price from product to product. If the content volume and price differ from product to product (Yes in step S41), in an embodiment that includes a fixed-position image acquisition unit 11F and a fixed-position image selection unit 11S, the image 100 taken by the fixed-position camera in the store may be transmitted to the portable information processing device 16. The device may then request that the ranking of desired purchase products be accepted using the image taken by the fixed-position camera 19 (step S42). In this way, when accepting the selection of any of multiple products of the same type but with variations in content volume and price, such as fish fillets, the product selection can be accepted based on the fixed-position image. By using the images 100 captured by the fixed-position camera, customers can rank their desired purchases while checking the products using clear images of each product stationary in its designated position. Furthermore, in this way, instead of using images of the store environment acquired from the portable information processing device 16, it is possible to generate a product list by narrowing down the products from the fixed-position images captured by the fixed-position camera 19, as described later, and provide it to the customer. The control unit 11 then proceeds to step S43, which will be described later.

[0028] If, as determined in step S41, the narrowed-down products do not differ in content volume or price for each product (No in step S41), the control unit 11 then determines whether the narrowed-down results in the identification of multiple products or a single product. First, it determines whether multiple products have been identified (step S51). If multiple products have been identified (step S51), the control unit 11 proceeds to step S43 and determines whether it has received a request for a product list, which is a list display of those products, from the customer's portable information processing device 16 (step S43). If it receives a request for a product list (Yes in step S43), the control unit 11 determines whether a shopping list for the customer to purchase products has already been prepared for that customer (step S44). The shopping list is a data area prepared for each customer to accept the operation of the customer selecting the products they want to purchase on the screen of the portable information processing device 16 and adding them to the "shopping cart".

[0029] If a shopping list for that customer has not yet been prepared (No in step S44), the control unit 11, acting as the product list generation unit 11L, reserves a data area for the customer's shopping list (step S45). It then generates product list data for the narrowed-down products and transmits the product list data to the customer's portable information processing device 16 (step S46). On the other hand, if a shopping list for that customer has already been prepared (Yes in step S44), there is no need to generate a duplicate shopping list, so the process in step S45 is skipped and the process proceeds to S46.

[0030] If, as determined in step S51, multiple products have not been identified, the control unit 11, acting as the product list generation unit 11L, determines whether a single product has been identified (step S52). If a single product has been identified, the process proceeds to step S44, and if a customer shopping list has not been prepared, it is prepared, and then it is determined whether a request for a product list has been received.

[0031] On the other hand, if no single product is identified in the determination in step S52 (No. in step S52), the control unit 11 determines whether updated location information has been obtained from the customer's portable information processing device 16 (step S53). This is to narrow down the products corresponding to the updated location when the location information is updated due to the customer's movement. Once updated location information is obtained, the control unit 11 returns to step S12 (see Figure 5) to determine the customer's location within the sales floor based on the updated location information.

[0032] If the updated location information was not acquired in the determination in step S53 (No in step S53), the control unit 11 then determines whether a new image has been acquired from the customer's portable information processing device 16 (step S54). If a new image has been acquired from the customer (Yes in step S54), the control unit 11 returns to step S14 (see Figure 5) to determine whether the newly acquired image is a scene in the store, one or more products, a product display label, shelf tag, or product POP, and performs processing according to the image. In this way, if the image of the scene in the store acquired from the portable information processing device 16 changes, a product list narrowed down to products near the customer can be generated and provided to the customer in accordance with that change.

[0033] In step S46 described above, after generating one or more product lists and sending them to the portable information processing device 16, the control unit 11 then determines whether the number of products listed in the product list is less than or equal to a predetermined value (step S61 in Figure 8). If the number of products is less than or equal to the predetermined value (Yes in step S61), attribute data is sent to the portable information processing device 16 that allows the customer to select a product using the product list displayed on the portable information processing device 16 (step S62). Then, the process proceeds to step S64, which will be described later. On the other hand, if the number of listed products exceeds the predetermined value (No in step S61), the control unit 11 determines that there are too many products listed. In that case, attribute data is sent to the portable information processing device 16 that makes it impossible to accept product selections from the customer using the product list displayed on the portable information processing device 16 (step S63). According to this embodiment, when the number of products in the product list falls below a predetermined number, the system can accept selections of products listed in the product list. Then, the process proceeds to step S64, which will be described later.

[0034] An example of a product list displayed on the portable display unit 16D of the portable information processing device 16, based on the product list data transmitted in step S46 described above, is now presented. Figure 11 is an explanatory diagram showing an example in Embodiment 1 and Embodiment 2 in which multiple products within a predetermined range from the customer are displayed as a product list. Figure 12 is an example of a product list generated based on captured images in which multiple products have been photographed, as in Embodiment 3 and Embodiment 4.

[0035] Figure 13 also shows an example of a configuration in which a request for product reservation can be accepted. For example, as in step S42 above, if there are multiple products of the same type but with variations in content and price, the customer can prioritize individual products of the same type, specify the quantity to purchase, and request a reservation from the store staff. The store staff who receive the request go to the sales floor where the target product is located and check, in order of priority, whether the target product is available or has already been purchased by another customer, and if there is a product that can be reserved, they will temporarily reserve it. Then, they will contact the customer who requested the reservation to let them know that the product has been reserved. For example, an image of the reserved product may be taken with the camera of the POS terminal 18 (not shown in Figure 1), and the captured image may be sent to the portable information processing device 16. The customer who receives the notification checks the temporarily reserved product on a screen (not shown), and if they wish to purchase the temporarily reserved product, they add the target product from the product list to their shopping cart, or if they do not wish to purchase it, they perform an operation (not shown) to indicate their intention to cancel the reservation. These operations are sent as instructions to the store staff's POS terminal 18. Upon receiving the instructions, the store staff either finalize the reserved items according to the instructions or cancel the reservation and return them to their original location. In this way, the customer can purchase the desired quantity of the reserved items.

[0036] For example, the product list 111 shown in Figure 11 consists of a product name column 112, a link information column 113, a price column, and a quantity column 114. However, this is just one example and is not limited to this. Each row of the product list 111 displays information for one or more specified products. The product name column 112 displays the product name. If "Details" is displayed in the link information column 113, the customer can touch that section to be provided with detailed information about the product. For example, the information that should be displayed on the product label as stipulated by law or self-regulation may be provided. For food products, the name, ingredients, country of origin of ingredients, allergens, additives, nutritional information, net weight, expiration date, storage method, etc. may be displayed. For clothing products, the fiber composition, care symbols, manufacturer / company name, country of origin, contact information, etc. may be displayed. The price column of the product list 111 displays the price of the product.

[0037] The quantity column 114 of the product list 111 initially displays zero, corresponding to the unselected state. If a customer wishes to purchase, they can change this initial value to any quantity greater than or equal to 1. However, the quantity column 114 is only displayed if the product list is set to accept selection in step S62 described above. If the product selection is set to not accept in step S63 described above, the quantity column 114 may be hidden or its initial value may be grayed out and unchangeable.

[0038] Below the product list 111, a clear button 116 labeled "Clear Product List" and an "Add to Cart" button 117 are displayed. When the clear button 116 is touched, the mobile operation detection unit 16S detects the operation. In response to the detection, the mobile control unit 16P clears the display of the product list. When the "Add to Cart" button 117 is touched, the mobile operation detection unit 16S detects the operation. In response to the detection, the mobile control unit 16P transmits the following data to the product purchase support device 10 for products where the value in the quantity column 114 is not zero: information about each product related to the product where the value in the quantity column 114 is not zero, information about the quantity set for each product, and identification information of the portable information processing device 16. Upon receiving this information, the control unit 11 of the product purchase support device 10 updates the corresponding customer's shopping list based on the received information.

[0039] The product list 121 shown in Figure 12, like the product list 111 in Figure 11, consists of a product name column 122, a link information column 123, a price column, and a quantity column 124. Below the product list 121, there is a clear button 126 labeled "Clear Product List" and an "Add to Cart" button 127, and the response to the operation of these buttons is the same as described for product list 111.

[0040] The product list 131 shown in Figure 13 is basically the same as the product list in Figure 12. The product list 131 consists of a product name column 132, a link information column 133, a price column, and a quantity column 134. Below the product list 131, there is a clear button 136 labeled "Clear Product List" and an "Add to Cart" button 137, and the responses to the operation of these buttons are the same as those described for product list 111. However, in Figure 13, the quantity column 134 does not allow the customer to specify individual quantities for the listed products, but rather to specify the total number of prioritized products to purchase (purchase quantity). In addition, a request button 138 that accepts requests to reserve products is located below the "Add to Cart" button 137.

[0041] From step S64 onward, the control unit 11 executes processing corresponding to the customer's operation on the product list in the portable information processing device 16. It determines whether detailed product information has been requested in the link information column 113 of the product list 111 (step S64). If detailed information has been requested (Yes in step S64), it refers to the product database and sends data related to the detailed information of the corresponding product to the portable information processing device 16 (step S65). Then, it returns to step S53 (see Figure 7) and repeats the processing to handle cases where updated location information or new captured images have been acquired.

[0042] If detailed product information was not requested in step S64 (No in step S64), the control unit 11 then determines whether a new product has been added to the shopping list or whether the quantity of a product has been increased (step S66). If a new product has been added (Yes in step S66), the control unit 11 adds that product to the shopping list. If the quantity of a product has been increased (Yes in step S66), the control unit 11 also changes the quantity of the corresponding product in the shopping list. Then, it transmits the updated shopping list data to the customer's portable information processing device (step S67). After that, the process returns to step S53 (see Figure 7) and the processing to handle updated location information or newly captured images is repeated.

[0043] If no items were added to the shopping list or the quantity of items was increased in step S66 (No in step S66), the control unit 11 then determines whether an item was deleted from the shopping list or the quantity of items was decreased (step S68). If an item was deleted (Yes in step S68), the control unit 11 removes that item from the shopping list. If the quantity of an item was decreased (Yes in step S68), the control unit 11 also changes the quantity of the corresponding item in the shopping list. The updated shopping list data is then sent to the customer's portable information processing device (step S69). After that, the process returns to step S53 (see Figure 7) and the processing to handle updated location information and newly captured images is repeated.

[0044] If, in step S68, an item is removed from the shopping list or the quantity of an item is not reduced (No in step S68), the control unit 11 then determines whether or not it has received a request to reserve the desired purchase item for which priority was received in step S42 (step S71 in Figure 9). It should be noted that step S42 is a process limited to cases where the fixed-position image acquisition unit 11F and fixed-position image selection unit 11S are provided, which are of any configuration. However, the process of prioritizing items and requesting reservations is not limited to cases using fixed-position images from the fixed-position camera 19, and can be considered as a form of product purchase support. If a request to reserve an item is received (Yes in step S71), the control unit 11, as the desired purchase item notification unit 11N, sends the request for the prioritized desired purchase item to the POS terminal 18 for store staff in the back room (step S72). Then, the process returns to step S53 (see Figure 7), and the process to handle updated location information or newly captured images is repeated.

[0045] If the product reservation request was not accepted in step S71 (No in step S71), the control unit 11, acting as the product reservation notification unit 11N, then determines whether it has received notification of the product reservation result from the POS terminal 18 of the store staff who received the reservation request (step S73). If it has received notification of the product reservation result (Yes in step S73), the control unit 11 updates the customer's shopping list based on the product reservation result and transmits the updated shopping list data to the customer's portable information processing device 16 (step S74). Then, the process returns to step S53 (see Figure 7) and repeats the process to handle updated location information and newly captured images.

[0046] If, as described above, no notification of the product reservation result was received in step S73 (No in step S73), the control unit 11 then determines whether or not it has accepted the request to purchase the items on the shopping list, i.e., to make a payment (step S75). If the payment request has been accepted (Yes in step S75), the control unit 11 processes the payment for the items on the shopping list, including the reserved items (step S76). In this way, the product purchase support device not only notifies the store staff of the desired purchase items and has them prepare the desired items, but also performs the payment processing for the prepared desired purchase items. Then, the process related to product purchase support is terminated.

[0047] This invention should be understood to include combinations of any of the above-described embodiments. In addition to the embodiments described above, various modifications of this invention are possible. These modifications should not be considered outside the scope of this invention. This invention should include the meaning of the claims and equivalents and all variations that fall within the scope of this invention. [Explanation of Symbols]

[0048] 10: Product purchase support device, 11: Control unit, 11A: Narrowing information acquisition unit, 11F: Fixed position image acquisition unit, 11L: Product list generation unit, 11N: Purchase desired product notification unit, 11R: Image recognition unit, 11S: Fixed position image selection unit, 12: Data storage unit, 13: Communication unit, 16: Portable information processing device, 16A: Portable shooting unit, 16C: Portable communication unit, 16D: Portable display unit, 16G: Portable location information provision unit, 16M: Portable storage unit, 16P: Portable control unit, 16S: Portable operation detection unit, 18: POS terminal, 18C: Terminal communication unit, 18D: Terminal display unit, 18M: Terminal storage unit, 18P: Terminal control unit, 18S: Terminal operation detection unit, 19: Fixed position camera, 20,30,40,100: Captured images, 111,121,131: Product list, 112,122,132: Product name column, 113,123,133: Link information column, 114,124,134: Quantity column, 116,126,136: Clear button, 117,127,137: "Add to cart" button, 138: Request button

Claims

1. A filtering information acquisition unit that acquires at least one of the following as filtering information from a portable information device carried by a customer in the store: location information within the sales floor, or images of the scenery within the sales floor or products. When a captured image is acquired as the aforementioned narrowing information, an image recognition unit narrows down the range of the sales area or the products based on matching it with a landscape image or product image of the sales area. A product list generation unit that determines one or more products based on at least one of the location information and the narrowed range of the sales area, or based on the product image, generates a product list relating to the determined products, and provides information relating to each product in a viewable format. A product purchase support device comprising: a product selection unit that receives the customer's selection of products from the product list and notifies the store staff of the selected products as products the customer wishes to purchase; and a product purchase request notification unit that notifies the store staff of the selected products as products the customer wishes to purchase.

2. The product purchase support device according to claim 1, wherein the product list generation unit determines products located within a predetermined range from the location based on the location information and provides information relating to those products in a viewable format.

3. The image recognition unit determines the position and direction corresponding to the customer's viewpoint from the acquired image of the scenery inside the sales area. The product purchase support device according to claim 1, wherein the product list generation unit determines products located within a predetermined distance and direction range from a position corresponding to the customer's viewpoint, and provides information relating to those products in a viewable format.

4. The image recognition unit updates the position and direction of the customer's viewpoint in accordance with the change in the landscape image acquired as the customer moves. The product purchase support device according to claim 3, wherein the product list generation unit updates the products in the product list according to the updated position and direction of the viewpoint.

5. The product purchase support device according to claim 4, wherein if the number of products in the product list exceeds a predetermined number, the product list generation unit provides information relating to those products in a viewable format, but does not accept selection by the customer.

6. The product purchase support device according to claim 1, wherein if the image recognition unit determines that the captured image is not a product image but a product label or shelf tag corresponding to a product, the product list generation unit provides information about the product related to that product label or shelf tag in a viewable format.

7. A fixed-position image acquisition unit that acquires fixed-position images taken by one or more fixed-position cameras installed in the sales area of ​​a store, The system further includes a fixed-position image selection unit that selects a fixed-position image corresponding to a range narrowed down from the aforementioned location information or images of the scenery within the sales area, The image recognition unit narrows down the products in the store based on the selected fixed-position image, The product purchase support device according to claim 1, wherein the product list generation unit provides information related to the narrowed-down products in a viewable format and accepts the customer's selection.

8. If the image recognition unit determines a plurality of products based on the captured images of the products, and these products include products of the same type but with variations in content volume and price, the product list generation unit enables the selection of each product based on fixed-position images for those products, and accepts the selection by the customer, as described in claim 7.

9. The product purchase support device according to claim 1, further comprising a payment processing unit that performs payment processing for the aforementioned product to be purchased.

10. Computers The process involves obtaining, as filtering information, at least one of the following from a portable information device carried by a customer in the store: location information within the sales floor, or images of the scenery within the sales floor or products within the sales floor. When a captured image is obtained as the aforementioned filtering information, the steps include narrowing down the range of the sales area or the products based on matching it with images of the scenery or products within the sales area, The steps include determining one or more products based on at least one of the location information and the narrowed-down sales area range, or based on the product image, generating a product list related to the determined products, and providing information related to each product in a viewable format, A step of receiving the customer's selection of products from the product list mentioned above that the customer wishes to purchase, A method for supporting product purchases, comprising the step of notifying the staff of the store of the selected product as a product to be purchased.

Images