Information processing device, commodity ordering system, control method for information processing device, and program

Abstract

To provide an information processing device, commodity ordering system, control method for the information processing device, and program that can smoothly and quickly display commodities in which a customer has shown interest as selection candidates.SOLUTION: An information processing device 30 includes: a menu identification unit 301a configured to identify a menu medium in which a customer has shown interest from captured images of the customer in a movement path; and a display adjustment unit 301b configured to execute processing for causing a display 106 (display unit) to display, at least at an arrival timing when the customer has arrived at a face-to-face position, a display screen such that commodities related to the menu medium identified by the menu identification unit 301a for the customer are prioritized as selection candidates.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device for processing product orders, a product ordering system, a control method for the information processing device, and a program. [Background technology]

[0002] Currently, stores are equipped with product ordering devices used to order products. For example, a ticket vending machine is installed in a store as this type of device. A customer operates the ticket vending machine to select a product he or she wants and then performs a payment process. This causes the ticket vending machine to issue an exchange ticket for that product. It is desirable for this type of device to enable customers to select products smoothly.

[0003] The following Patent Document 1 describes a product selection support method that detects a customer's line of sight and supports product selection. In this method, a screen on which images of multiple products scroll in a random arrangement is displayed on a display unit. When a customer approaches this screen and gazes at the image of a specific product, the image of the product in the gaze area including the gaze point is displayed in the center area of ​​the screen. If the customer then continues to gaze at the image of the product in this area, the image of the product is highlighted. This allows the customer to smoothly select the product. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-022589 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the above-mentioned product selection support method, after the customer faces the display unit, the product that the customer is interested in is determined from the customer's line of sight, and the display screen is switched. Therefore, it takes time from when the customer faces the display unit until the product that the customer is looking at is displayed, and during that time, the customer must wait to select a product.

[0006] In view of such problems, the present invention aims to provide an information processing device, a product ordering system, a control method for an information processing device, and a program that can smoothly and quickly display products in which a customer has shown interest as selection options. [Means for solving the problem]

[0007] A first aspect of the present invention relates to an information processing device for use in a store. In the store, a plurality of types of menu media are arranged so as to be visible to the customer along the customer's path to a meeting position where the customer faces a display unit that displays product selection candidates. The information processing device includes a menu identification unit that identifies the menu medium in which the customer has shown interest from a captured image of the customer along the path, and a display adjustment unit that executes a process to display a display screen on the display unit such that products related to the menu medium identified by the menu identification unit for the customer are prioritized as selection candidates at least at the time when the customer arrives at the meeting position.

[0008] According to the information processing device of this aspect, the products related to the menu medium in which the customer showed interest along the route of movement are preferentially displayed on the display screen as selection candidates at least when the customer arrives at the face-to-face position. Therefore, the products in which the customer showed interest can be displayed smoothly and quickly as selection candidates. This can improve the convenience of the customer in selecting products.

[0009] In the information processing device of this embodiment, the menu identification unit can be configured to obtain at least one of the number of times and the accumulated time that the customer viewed each of the menu media along the travel route, and to identify the menu media in which the customer showed interest based on at least one of the number of times and the accumulated time.

[0010] According to this configuration, it is possible to comprehensively judge the customer's interest in each menu medium, and therefore it is possible to appropriately display products that correspond to the customer's interest as selection candidates.

[0011] In the information processing device of this embodiment, the menu identification unit can be configured to estimate the direction in which the customer is looking from the captured image, determine whether or not the menu medium is present in that direction, and, based on the determination result, identify the menu medium in which the customer has shown interest.

[0012] According to this configuration, by using the process of estimating the customer's viewing direction, it is possible to appropriately determine which menu medium the customer has looked at. Therefore, from the determination result, it is possible to accurately identify the menu medium in which the customer has shown interest.

[0013] In the information processing device of this embodiment, the menu identification unit may configure linking information linking the customer with the menu medium in which the customer has shown an interest, and the display adjustment unit may be configured to cause the display unit to display a display screen based on the linking information so that products related to the menu medium in which the customer has shown an interest are given priority as selection candidates.

[0014] According to this configuration, the linking information is prepared before the customer moves to the meeting position, so that a display screen according to the customer's interests can be smoothly presented to the customer.

[0015] In this configuration, the menu specification unit can be configured to acquire face information of the customer from the captured image, and to associate the acquired face information with the menu medium in which the customer has shown interest.

[0016] According to this configuration, even without using a means for tracking a customer moving from his / her movement path to the meeting position, the menu medium in which the customer is interested can be smoothly identified by comparing the facial information of the customer acquired at the meeting position with the facial information acquired by the menu identification unit.

[0017] In the information processing device of this aspect, the menu medium may be placed within the store, and the menu identification unit may be configured to identify the menu medium in which the customer has shown interest from images captured of the customer on his or her path of movement within the store.

[0018] According to this configuration, the menu medium in which the customer is interested can be identified from the captured image of the customer moving around the store.

[0019] In the information processing device of this aspect, the menu medium may be placed outside the store, and the menu identification unit may be configured to identify the menu medium in which the customer has shown interest from images of the customer on their path of movement outside the store.

[0020] According to this configuration, the menu medium in which a customer is interested can be identified from a captured image of the customer looking at the menu medium outside the store.

[0021] A second aspect of the present invention relates to a product ordering system that accepts product orders using a product ordering device that displays product selection candidates. The product ordering system is used in a store, and multiple types of menu media are arranged in the store so that the customer can see them along the customer's movement path to a meeting position where the customer faces the product ordering device. The product ordering system includes a menu identification unit that identifies the menu media in which the customer has shown interest from a captured image of the customer along the movement path, and a display adjustment unit that executes a process to display a display screen on a display unit of the product ordering device so that products related to the menu media identified by the menu identification unit for the customer are prioritized as selection candidates at least at the time when the customer arrives at the meeting position.

[0022] According to the second aspect of the present invention, the same effects as those of the first aspect can be achieved.

[0023] In the product ordering system of this embodiment, the menu identification unit may be configured to acquire first facial information of the customer from the captured image and link the menu medium in which the customer has shown interest to the acquired first facial information, and the display adjustment unit may be configured to acquire second facial information from a second captured image of the customer near the meeting position and acquire the menu medium linked to the first facial information that matches the acquired second facial information.

[0024] According to this configuration, the menu medium in which the customer is interested can be smoothly identified without using a means for tracking the customer who moves from the movement path to the meeting position.

[0025] In the product ordering system of the present embodiment, the product selection screen displayed on the display unit is composed of multiple hierarchical levels, and the product ordering device can be configured to include product selection items corresponding to the menu medium identified by the menu identification unit on the initial selection screen displayed on the display unit.

[0026] According to this configuration, the initial selection screen includes product selection items corresponding to the menu medium in which the customer is interested, so that the customer can smoothly select the product in which he or she is interested.

[0027] In this configuration, the initial selection screen includes a plurality of type selection items for selecting a product type, and the product ordering device can be configured to display, among the plurality of type selection items displayed on the initial selection screen, the type selection items of a product type including a product corresponding to the menu medium identified by the menu identification unit more effectively than the other type selection items.

[0028] According to this configuration, type selection items of product types including products corresponding to the menu medium in which the customer has shown interest are effectively displayed, so that the customer can smoothly select products of the same type when he or she does not select the product in which he or she has shown interest.

[0029] Alternatively, in the above configuration, the initial selection screen includes a plurality of type selection items for selecting a product type, and the product ordering device may be configured to display, among the plurality of type selection items displayed on the initial selection screen, type selection items of the product corresponding to the menu medium identified by the menu identification unit and other product types that satisfy predetermined classification conditions more effectively than the other type selection items.

[0030] According to this configuration, type selection items of product types including a product corresponding to a menu medium in which a customer has shown an interest are effectively displayed together with the type selection items of other product types that satisfy the product type and a predetermined classification condition, so that when a customer does not select the product in which he or she has shown an interest, the customer can smoothly select a product similar to the product.

[0031] A third aspect of the present invention relates to a control method for an information processing device used in a store. In the store, a plurality of types of menu media are arranged so as to be visible to the customer along a moving path of the customer to a meeting position where the customer faces a display unit that displays product selection candidates. The control method according to this aspect identifies the menu media in which the customer has shown an interest from a captured image of the customer along the moving path, and executes a process for displaying a display screen on the display unit such that products related to the menu media identified for the customer are prioritized as selection candidates at least at the time when the customer arrives at the meeting position.

[0032] A fourth aspect of the present invention relates to a program for causing an information processing device used in a store to execute a predetermined function. In the store, a plurality of types of menu media are arranged in a manner that is visible to the customer along the customer's path to a meeting position where the customer faces a display unit that displays product selection candidates. The program according to this aspect causes the information processing device to execute a function of identifying the menu media in which the customer has shown an interest from a captured image of the customer along the path, and a function of displaying a display screen on the display unit such that products related to the menu media identified for the customer are prioritized as selection candidates at least at the time when the customer arrives at the meeting position.

[0033] According to the third and fourth aspects of the present invention, the same effects as those of the first aspect can be achieved. Effect of the Invention

[0034] As described above, according to the present invention, it is possible to provide an information processing device, a product ordering system, a control method for an information processing device, and a program that are capable of smoothly and quickly displaying products in which a customer has an interest as selection candidates.

[0035] The effects and significance of the present invention will become clearer from the following description of the embodiment. However, the embodiment described below is merely an example of how the present invention can be put into practice, and the present invention is not limited to the embodiment described below. [Brief description of the drawings]

[0036] [Figure 1] FIG. 1 is a diagram showing a configuration of a product ordering system according to the first embodiment. [Diagram 2] FIG. 2 is a plan view illustrating an example of the arrangement of a ticket vending machine, a camera, and a menu medium in a store according to the first embodiment. [Diagram 3] FIG. 3 is a block diagram showing a configuration of a product ordering system according to the first embodiment. [Figure 4]4(a) to 4(f) are diagrams each showing a schematic diagram of a method for generating training data (state data set) used in machine learning according to the first embodiment. [Diagram 5] 5(a) to (f) are diagrams each showing a schematic diagram of a method for generating training data (state data set) used in machine learning according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing a configuration of the gaze state association information according to the first embodiment. [Figure 7] 7(a) and (b) are diagrams showing product linking information and category linking information stored in the storage unit of the ticket vending machine according to the first embodiment, respectively. [Figure 8] FIG. 8 is a flowchart showing a process performed by the control unit of the information processing device in response to detection of a face image from an image captured by a camera directed toward a travel route according to the first embodiment. [Figure 9] FIG. 9 is a flowchart showing a process performed by the control unit of the ticket vending machine in response to a customer reaching the vicinity of the meeting position according to the first embodiment. [Figure 10] FIG. 10 is a diagram showing an initial selection screen when the product selection screen is set as the default screen according to the first embodiment. [Figure 11] FIG. 11 is a diagram showing a selection screen of the second layer according to the first embodiment. [Figure 12] FIG. 12 is a diagram showing an initial selection screen when the product selection screen is adjusted according to the first embodiment. [Figure 13] FIG. 13 is a diagram showing another example of the configuration of the initial selection screen when the product selection screen is adjusted according to the first embodiment. [Figure 14] FIG. 14 is a diagram showing yet another example of the configuration of the initial selection screen when the product selection screen is adjusted according to the first embodiment. [Figure 15] FIG. 15 is a diagram showing a configuration of purchase management information for managing product purchasing status of customers in a store according to the first embodiment. [Figure 16]FIG. 16 is a plan view illustrating an example of the arrangement of a ticket vending machine, a camera, and a menu medium in a store according to the second embodiment. [Figure 17] FIG. 17 is a block diagram showing a configuration of a product ordering system according to the third embodiment. As shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0038] <Embodiment 1> FIG. 1 is a diagram showing a configuration of a product ordering system 1 according to the first embodiment.

[0039] The product ordering system 1 includes a ticket vending machine 10, cameras 21 and 22, and an information processing device 30. The product ordering system 1 is used in stores such as restaurants.

[0040] The ticket vending machine 10 is installed in a store. A customer visiting the store purchases a voucher for a product he or she desires from the ticket vending machine 10. The ticket vending machine 10 is a product ordering device that displays product selection options. In the store, multiple types of menu media 40 are arranged so that the customer can see them along the customer's path to the position where the customer faces the ticket vending machine 10. Here, multiple menu media 40 are arranged in the store.

[0041] Each menu medium 40 displays a different product image and simple information about the product (price, catchphrase, etc.). The menu medium 40 is, for example, a poster. The menu medium 40 may be other media such as digital signage or a large screen.

[0042] The camera 21 captures an image of a customer approaching the ticket vending machine 10 inside the store. The camera 21 is used to determine whether or not the customer has looked at the menu medium 40. The camera 21 is positioned so as to be able to capture an image of the face of a customer looking at the menu medium 40. Preferably, the camera 21 is positioned around the area where the menu medium 40 is placed, such as directly above the area where the menu medium 40 is placed.

[0043] Camera 22 captures an image of a predetermined area in front of ticket vending machine 10 from the rear of ticket vending machine 10. The image captured by camera 22 is used to match a customer who has seen menu medium 40 on his / her path of movement with a customer who has reached the facing position. If this matching can be performed by camera 21 alone in relation to the placement position of menu medium 40 and the imaging field of camera 21, camera 22 may be omitted.

[0044] The information processing device 30 is connected to the ticket vending machine 10 and the camera 21 so as to be able to communicate with them. The information processing device 30 is installed in an office or the like of the store in which the ticket vending machine 10 is installed. The information processing device 30 may also be installed in a facility other than a store. In this case, the information processing device 30 communicates with the ticket vending machine 10 and the camera 21 via a public communication network such as the Internet.

[0045] The information processing device 30 performs processing for adjusting the screen (product selection screen) displayed on the ticket vending machine 10 based on the image captured by the camera 21. The information processing device 30 is configured, for example, as a server computer or a personal computer. When multiple sets of ticket vending machines 10 and cameras 22 are installed in a store, the information processing device 30 may be connected to each set of ticket vending machines 10 so as to be able to communicate with them. In this case, the information processing device 30 performs the above-mentioned processing for each set of ticket vending machine 10 and camera 22.

[0046] The ticket vending machine 10 is connected to the camera 22 so as to be able to communicate with it. The ticket vending machine 10 has a substantially cubic shaped housing that constitutes the outer shell of the device. A touch panel type operation display unit 11 is disposed on the upper front surface of the ticket vending machine 10. As will be described later with reference to FIG. 3, the operation display unit 11 is configured by disposing a transparent touch sensor 107 on the upper surface of a display 106. The display 106 is, for example, a liquid crystal display, and the touch sensor 107 is, for example, a pressure-sensitive (resistive film) touch pad. However, the configuration of the operation display unit 11 is not limited thereto, and the touch sensor 107 may be, for example, a capacitive touch pad.

[0047] A banknote deposit / withdrawal port 12 for depositing and withdrawing banknotes, a coin deposit port 13 for depositing coins, a coin withdrawal port 14 for withdrawing coins, and a ticket issuing port 15 for issuing tickets are arranged in the center of the front of the ticket vending machine 10. In addition, a human presence sensor is arranged in the ticket vending machine 10. The human presence sensor detects when a customer has arrived near the facing position where the customer faces the operation display unit 11 of the ticket vending machine 10.

[0048] Here, the facing position refers to a position where a customer faces the operation display unit 11 (a display unit that displays product selection candidates) in order to order a product. In other words, the facing position is a position where a product can be selected using the display screen.

[0049] In the first embodiment, the position where the customer faces the operation and display unit 11 to issue a product ticket corresponds to the facing position. In other words, the position where the customer operates the operation and display unit 11 (position where operation is possible) is the facing position. The facing position does not necessarily have to be directly in front of the operation and display unit 11, and may be slightly shifted to the left or right from directly in front. In the ticket vending machine 10, the facing position is defined as a position that is a predetermined distance (for example, several tens of centimeters) forward from the operation and display unit 11.

[0050] The farthest position of the detection range of the human presence sensor is set to be at least the distance to the face-to-face position. The farthest position of the detection range may be set to a position that is a predetermined distance (for example, several tens of centimeters) away from the distance to the face-to-face position. The above-mentioned vicinity of the face-to-face position refers to the range between the farthest position and the face-to-face position.

[0051] It should be noted that the arrival of the customer near the meeting position may be determined using the image captured by camera 22. In this case, ticket vending machine 10, for example, extracts a face area from the image captured by camera 22, and determines that the customer has arrived near the meeting position when the size (e.g., area, vertical width, etc.) of the extracted face area becomes equal to or larger than a predetermined threshold. In this configuration, the human sensor may be omitted from ticket vending machine 10.

[0052] When the human presence sensor detects that the customer has reached the meeting position, a product selection screen is displayed on the operation and display unit 11. Here, a selection of food and drink options that the store can provide is displayed. The customer touches the desired option from among the multiple options displayed on the operation and display unit 11 to select the product to be purchased.

[0053] The customer then inserts bills or coins in the amount required to purchase the product into the bill deposit / withdrawal slot 12 or coin deposit slot 13. This causes a voucher corresponding to the selected item to be issued from the ticket issuing slot 15. If there is change, the bills or coins corresponding to the change are dispensed from the bill deposit / withdrawal slot 12 or coin withdrawal slot 14. This completes one step of the transaction.

[0054] FIG. 2 is a plan view showing a schematic example of the arrangement of the ticket vending machine 10, cameras 21 and 22, and menu medium 40 in the store 2. As shown in FIG.

[0055] A ticket vending machine 10 is installed at the back of the store 2 in the direction in which a customer 3 will go straight from the entrance 2a. The ticket vending machine 10 is installed so that its front faces the entrance 2a. The area from the entrance 2a to just before the ticket vending machine 10 forms a movement path 2b1 that the customer 3 will take toward the ticket vending machine 10. To the left of the movement path 2b1 is the eating and drinking space 2c. A number of tables and a number of chairs are arranged in the eating and drinking space 2c. The movement path 2b1 is defined between the eating and drinking space 2c and the inner wall of the store 2. The customer 3 goes straight along the movement path 2b1 to reach the meeting position P1. After issuing the ticket, the customer 3 proceeds from the meeting position P1 to the eating and drinking space 2c.

[0056] A number of menu media 40 are arranged in a line at the back of the eating and drinking space 2c. The multiple menu media 40 are visible to the customer 3 along the movement path 2b1. The camera 21 captures an image of the vicinity of the movement path 2b1 in a diagonal direction from a position directly above the area in which the multiple menu media 40 are arranged. The range between the dashed lines extending from the camera 21 is the imaging field of the camera 21. The boundary of the imaging field of the camera 21 on the ticket vending machine 10 side is a predetermined distance toward the entrance 2a from the facing position P1. The other camera 22 is arranged approximately in the center of the back side of the ticket vending machine 10, and captures an image of the vicinity of the facing position P1.

[0057] In the example of FIG. 2, a showcase 50 is installed on the left side of the entrance 2a outside the store 2. A plurality of menu media 51 are displayed in this showcase 50. The menu media 51 are model product samples. The space in front of this showcase 50 also becomes the movement route 2b2 that the customer 3 follows from the entrance 2a toward the ticket vending machine 10.

[0058] FIG. 3 is a block diagram showing the configuration of the product ordering system 1. As shown in FIG.

[0059] The ticket vending machine 10 includes a control unit 101, a memory unit 102, a banknote processing unit 103, a coin processing unit 104, a ticket issuing processing unit 105, a display 106, a touch sensor 107, a speaker 108, and a communication unit 109.

[0060] The control unit 101 includes a calculation processing circuit such as a CPU (Central Processing Unit), and controls each unit according to a program stored in the storage unit 102 .

[0061] The storage unit 102 includes storage media such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and stores programs executed by the control unit 101 and various data. The storage unit 102 is also used as a work area when the control unit 101 performs control. The storage unit 102 stores a face recognition engine for acquiring face information of the customer 3 from an image captured by the camera 22.

[0062] The banknote processing unit 103 includes a banknote storage unit for storing banknotes of each denomination, a transport unit for transporting the banknotes, and a denomination discrimination unit for discriminating the denomination of the transported banknotes, and transports banknotes between the banknote storage unit and the banknote deposit / withdrawal port 12 (see FIG. 1) under the control of the control unit 101. The coin processing unit 104 includes a coin storage unit for storing coins of each denomination, a transport unit for transporting the coins, and a denomination discrimination unit for discriminating the denomination of the transported coins, and transports coins between the coin storage unit and the coin deposit port 13 and coin withdrawal port 14 (see FIG. 1) under the control of the control unit 101.

[0063] The ticket issuing processing unit 105 includes a strip generating unit that generates a paper strip, and a printing unit that prints on the strip, and sends the voucher, on which the name of the food or drink, etc. has been printed by the printing unit, to the ticket issuing port 15.

[0064] The display 106 and the touch sensor 107 constitute the operation display unit 11 in FIG. 1. The touch sensor 107 is a transparent film-like member that is overlaid on the display surface of the display 106. The display 106 displays predetermined information under the control of the control unit 101. The touch sensor 107 outputs coordinate information of the position touched by the operator to the control unit 101. The speaker 108 outputs predetermined sound under the control of the control unit 101. The output sound is output to the outside from a sound output window (not shown) formed on the housing of the ticket vending machine 10. The communication unit 109 communicates with the camera 22 and the information processing device 30 under the control of the control unit 101.

[0065] The information processing device 30 includes a control unit 301 , a storage unit 302 , and a communication unit 303 .

[0066] The control unit 301 includes a processing circuit such as a CPU, and controls each unit according to a program stored in the storage unit 302. The storage unit 302 includes storage media such as a ROM, a RAM, and a hard disk, and stores the programs executed by the control unit 301 and various data. The storage unit 302 is also used as a work area when the control unit 301 performs control.

[0067] The communication unit 303 communicates with the camera 21 and the communication unit 109 under the control of the control unit 301. The communication unit 303 is also connected to an external communication network 60 such as the Internet, and communicates with devices outside the store 2 under the control of the control unit 301.

[0068] In the first embodiment, the program stored in the storage unit 302 provides the control unit 301 with the functions of a menu specification unit 301a and a display adjustment unit 301b.

[0069] The menu specification unit 301a specifies the menu medium 40 in which the customer 3 is interested from the image captured by the camera 21, i.e., the image captured of the customer 3 on the moving path 2b1. The display adjustment unit 301b executes a process for displaying a display screen (product selection screen) on the operation display unit 11 so that a product related to the menu medium 40 specified by the menu specification unit 301a for the customer is given priority as a selection candidate at least at the arrival timing when the customer 3 arrives at the meeting position P1.

[0070] In the first embodiment, direct control of the adjustment of the display screen of the operation and display unit 11 is performed by the control unit 101 of the ticket vending machine 10, and information serving as a trigger or condition for this control is transmitted from the information processing device 30 to the ticket vending machine 10 by the function of the display adjustment unit 301b. In other words, the display adjustment unit 301b executes indirect control of the adjustment of the display screen of the operation and display unit 11.

[0071] In addition, the storage unit 302 stores a face recognition engine for the control unit 301 to extract a face image from an image captured by the camera 21 and acquire face information. The acquired face information is information on the feature amount of a face. The face information may be the extracted face image itself.

[0072] The storage unit 302 also stores a viewpoint estimation engine that allows the control unit 301 to estimate the direction in which the customer 3 is looking from the image captured by the camera 21 and to estimate whether the customer 3 is looking at any menu medium 40 from the estimated direction. These engines are used by the menu identification unit 301a.

[0073] Furthermore, the storage unit 302 stores an attribute estimation engine for the control unit 301 to estimate customer attributes such as age and sex of each customer from each piece of face information extracted by the face recognition engine.

[0074] The above-mentioned viewpoint estimation engine may be, for example, a machine learning model. As the machine learning, machine learning using a neural network is applied. For example, a neural network by deep learning in which neurons are combined in multiple stages is applied. However, the applied machine learning is not limited to this, and other machine learning such as a support vector machine or a decision tree may be applied to the machine learning model.

[0075] 4(a) to (f) and 5(a) to (f) are diagrams that diagrammatically show a method of generating training data (state data set) used in machine learning.

[0076] The state data set is a collection of a large number of state data. The state data is key point data indicating the positions of a plurality of body parts obtained from a person on a captured image captured by the camera 21. The key point data is obtained by applying a skeleton estimation engine to the image range of the person on the captured image.

[0077] In generating the state data set, a two-dimensional region 70, which is a virtual plane, is set. The two-dimensional region 70 is set so as to have a predetermined positional relationship with respect to the imaging direction (the central axis direction of the imaging field of view) of the camera 21. The two-dimensional region 70 is set so as to be perpendicular to the imaging direction of the camera 21. The center of the two-dimensional region 70 coincides with the central axis of the imaging field of view of the camera 21. However, this is not limiting, and for example, the middle position of the upper side of the two-dimensional region 70 may coincide with the central axis of the imaging field of view of the camera 21.

[0078] The two-dimensional area 70 has a predetermined width in each of the X-axis direction (horizontal direction of the imaging field of view) and the Z-axis direction (vertical direction of the imaging field of view). Here, the two-dimensional area 70 is a rectangle with its long side parallel to the X-axis direction. However, the method of setting the two-dimensional area 70 is not limited to this. For example, the two-dimensional area 70 may be a square.

[0079] The two-dimensional area 70 is divided into a grid pattern, and a plurality of reference areas 71 are set. Here, the two-dimensional area 70 is divided into a grid pattern at the same pitch in the X-axis direction and the Z-axis direction. However, the method of dividing the two-dimensional area 70 into a grid pattern is not limited to this. For example, the division pitch of the two-dimensional area 70 in the X-axis direction may be different from the division pitch of the two-dimensional area 70 in the Z-axis direction.

[0080] The state dataset, which is the teacher data, is a dataset that collects state data regarding the state of a person in a captured image when the person directs their gaze toward each reference area 71, for multiple people's states (postures) for each reference area 71.

[0081] 4(a)-(f) show the state of person 3a when viewed from the positive side of the Y axis when person 3a directs his / her gaze toward reference area 71a located in the third row and third column from the upper left corner (origin) of two-dimensional area 70. In Fig. 4(a)-(f), an arrow is added indicating the direction in which person 3a's nose 3a1 is pointing. In Fig. 4(d)-(f), person 3a is on the positive side of the Y axis than in Fig. 4(a)-(c), so the range of person 3a is larger.

[0082] In this case, six state data are obtained for the reference region 71a. That is, key point data is obtained for each state of the person 3a in Fig. 4(a) to (f), and the obtained key point data is associated with the reference region 71a.

[0083] Here, the key point data is acquired as, for example, coordinate positions (pixel positions) of the following human body parts (key points) on the captured image.

[0084] Skeletal center (middle between left and right shoulders), left ankle, right ankle, left ear, right ear, left elbow, right elbow, left eye, right eye, left thigh, right thigh, left knee, right knee, neck, nose, left shoulder, right shoulder, left wrist, right wrist, left corner of mouth, right corner of mouth

[0085] To acquire key point data, a method may be used in which, in an environment in which the relationship between the camera 21 and the two-dimensional region 70 is set to the above-mentioned relationship (for example, the two-dimensional region 70 is set so as to be perpendicular to the imaging direction of the camera 21, and the center of the two-dimensional region 70 coincides with the central axis of the imaging field of view of the camera 21), captured images are actually acquired of the person 3a looking at the reference region 71a in various postures at various positions within the imaging field of view of the camera 21, and each of the acquired captured images is applied to a skeleton estimation engine to extract the coordinate position (pixel position) of each key point in the captured images.

[0086] Alternatively, key point data when a person looks at the reference area 71a may be obtained using an extraction engine capable of extracting key point data in a virtual space. Various extraction engines of this type have already been developed and are available. The virtual space includes a virtual human body model to which the above-mentioned key points are attached in advance. The operator can arbitrarily change the position and posture of this human body model. In this virtual space, the imaging field of view of the camera 21 and the two-dimensional area 70 are set in the same relationship as above. The operator can direct the line of sight of the human body model to each reference area 71 in the two-dimensional area 70. The operator can also cause the extraction engine to extract the coordinate positions of the key points of the human body model on the image captured by the camera 21 at that time.

[0087] In this case, as shown in Fig. 4(a) to (f), the operator sets the line of sight of the human body model corresponding to the person 3a to the reference area 71a, performs an operation to change the position and posture of the human body model, and then performs an operation to extract key point data. As a result, key point data corresponding to each position and posture is obtained. The obtained key point data is linked to the reference area 71a.

[0088] 5(a) to (f), similarly to the above, using an extraction engine based on actual imaging or virtual space, state data when the person 3a looks at the reference area 71b is acquired and linked to the reference area 71b. For reference areas 71 other than the reference areas 71a and 71b, multiple state data are acquired in the same manner as above and linked to each reference area 71.

[0089] 4(a)-(f) and 5(a)-(f) each show six types of human states when looking at the reference areas 71a and 71b, but the actual state data associated with one reference area 71 in the state data set is set to several levels more than six. From the viewpoint of machine learning, it is preferable that the number of state data associated with each reference area 71 is as large as possible.

[0090] The input of the machine learning model is assigned with a plurality of input items corresponding to a plurality of parts from which keypoint data is obtained, respectively. The output of the machine learning model is assigned with a plurality of output items corresponding to a plurality of reference regions 71 of the two-dimensional region 70, respectively.

[0091] When learning the machine learning model, the state data set generated as described above is input sequentially into multiple input items of the machine learning model. Multiple key point data included in each state data is input into the input item of the corresponding body part. When inputting one state data, 100% is set in the output item corresponding to the reference area 71 from which this state data was obtained (the reference area 71 seen by the person 3a), and 0% is set in the other output items. In this way, learning is performed for all state data in the state data set.

[0092] By such learning, when the machine learning model receives keypoint data, which is state data, it outputs the probability (0 to 100%) from each output item that the gaze was directed at that output item. Therefore, when the skeleton estimation engine acquires keypoint data (state data) of the customer 3 on the captured image from the image captured by the camera 21 and inputs the acquired keypoint data into the corresponding input items, the probability of the output item corresponding to the reference area 71 viewed by the customer 3 among the multiple output items becomes higher than the other output items. Therefore, the reference area 71 corresponding to the output item with the highest probability can be acquired as the reference area 71 at which the customer 3 directed his / her gaze.

[0093] In the information processing device 30, the area of ​​each menu medium 40 arranged in the store 2 is set in advance on a two-dimensional area 70 based on the positional relationship between the camera 21 and the menu medium 40. In the example of Fig. 1 and Fig. 2, the arrangement plane of the multiple menu media 40 is inclined with respect to the two-dimensional area 70 perpendicular to the central axis of the imaging field of view of the camera 21. Therefore, the area of ​​each menu medium 40 is set on the two-dimensional area 70 by projecting each of the multiple menu media 40 onto the two-dimensional area 70 in a direction parallel to the central axis of the imaging field of view.

[0094] This setting is performed by the manager of the store 2 in response to the completion of the arrangement of the menu media 40. At this time, the manager associates the area of ​​each menu medium 40 on the two-dimensional area 70 with the menu medium 40 (identification information of the menu medium 40). This association is stored in the storage unit 302 of the information processing device 30.

[0095] The storage unit 302 in FIG. 3 stores the machine learning model that has been trained. The menu identification unit 301a acquires key point data (state data) of the customer 3 on the captured image from the captured image of the camera 21 using a skeleton estimation engine, and inputs the acquired key point data into the corresponding input items of the machine learning model. The menu identification unit 301a identifies the output item with the highest probability based on this input, and determines whether the reference area 71 corresponding to the identified output item is included in the area of ​​any of the menu media 40 set on the two-dimensional area 70. If the determination is YES, the menu identification unit 301a identifies the menu medium 40 corresponding to the area including the reference area 71 as the menu medium 40 viewed by the customer 3.

[0096] The process for estimating that the customer 3 has viewed the menu medium 40 is not limited to the above example. For example, in the above, the viewpoint estimation engine used by the menu specification unit 301a is a machine learning model using a neural network, but other machine learning models may be used as the viewpoint estimation engine. Also, in the above, key point data based on skeletal estimation is used as information used for viewpoint estimation, but the line of sight may be estimated from the captured image, and it may be determined whether or not this line of sight is directed toward any of the menu media 40. Various methods may be used for the process for determining whether or not the customer 3 has viewed the menu medium 40 from the captured image of the camera 21.

[0097] When the menu identification unit 301a acquires facial information of the customer 3 from the image captured by the camera 21, it generates gaze situation linking information indicating the situation in which the customer 3 looked at each menu medium 40 along the movement route 2b1 and stores the information in the memory unit 302.

[0098] FIG. 6 is a diagram showing the configuration of the gaze state linking information.

[0099] The gaze situation linking information is composed of a serial number, face information, and the number of times and time (accumulated time) each menu medium 40 was viewed. The "serial number" is a consecutive number that is assigned when new face information is registered in the gaze situation linking information. The "face information" is the face information of the customer extracted by the face recognition engine described above. The "number of times" is the number of times that the corresponding customer 3 viewed the menu medium 40 corresponding to each product (in FIG. 6, menu media A'-P' corresponding to products A-P) on the movement route 2b1. The "time" is the total time (accumulated time) that the corresponding customer viewed each menu medium 40 on the movement route 2b1.

[0100] For convenience, in Fig. 6, a schematic diagram of a face is shown in the item of face information, but as described above, face information is a feature amount extracted from a face image. However, as described above, a face image itself may be registered as face information in the gaze situation linking information. The registration process of the gaze situation linking information will be described in detail later with reference to Fig. 8.

[0101] 6, in the gaze status linking information, high values ​​are held in the detection items (number of times, time) of the menu medium 40 in which the customer 3 showed interest. Therefore, it can be said that the gaze status linking information is linking information linking the customer 3 with the menu medium 40 in which the customer 3 showed interest.

[0102] 7(a) and (b) are diagrams showing the product linking information and the category linking information stored in the memory unit 102 of the ticket vending machine 10. FIG.

[0103] 7(a) is information linking menu medium 40 (here, menu media A'-P') with the target products (here, products A-P) of the menu medium 40. The product linking information is registered in ticket vending machine 10 in response to placing menu medium 40 in store 2, and is updated thereafter in response to updates to menu medium 40.

[0104] The category linking information in FIG. 7(b) is information linking products handled by store 2 (here, products A to Q) with the categories of those products (here, first category to third category). The first category to third category are defined by mutually different category conditions. The category linking information in FIG. 7(b) also includes products for which menu media 40 have not been generated.

[0105] If store 2 is a restaurant that handles a variety of products, for example, the first category specifies whether the product is a main dish or a side dish, and the second category specifies whether the product is a food or a drink. The third category specifies whether the product is noodles, rice dishes (rice, rice bowls), set meals, salads, desserts, or drinks. Furthermore, sub-categories may be set for the third category. For example, if the third category is noodles, udon, ramen, pasta, and the like are set as its sub-categories. This linking information is registered in ticket machine 10 when store 2 opens, and is updated thereafter according to changes in the menu.

[0106] The product linking information and category linking information are registered by an administrator using the operation and display unit 11 of the ticket vending machine 10. The administrator sets the ticket vending machine 10 to a registration mode for this linking information to perform this registration. This registration may be performed via the information processing device 30. In this case, the administrator operates the information processing device 30 to input this linking information. Once the input is complete, this linking information is transmitted from the information processing device 30 to the ticket vending machine 10 and stored in the memory unit 302 of the ticket vending machine 10.

[0107] The product linking information and category linking information are used when the ticket vending machine 10 displays the product selection screen. The display process of the product selection screen in the ticket vending machine 10 will be described in detail later with reference to FIG.

[0108] FIG. 8 is a flowchart showing a process performed by the control unit 301 of the information processing device 30 in response to detection of a face image from an image captured by the camera 21.

[0109] 8, the processes of steps S101 to S106 are executed by the control unit 301 using the function of the menu specification unit 301a, and the process of step S107 is executed by the control unit 301 using the function of the display adjustment unit 301b. In the following, a description will be given on the assumption that the control unit 301 executes the processes of FIG. 8 using these functions.

[0110] When the control unit 301 detects a new face of the customer 3 on the image captured by the camera 21 (S101: YES), it acquires face information from the face and newly registers the acquired face information in the gaze state linking information of Fig. 6 (S102). By this registration, a new serial number is assigned, and the new face information is linked to this serial number.

[0111] The control unit 301 determines whether or not the customer with the detected face is within the movement route 2b1 (S103). The determination in step S103 is YES while the target customer 3 is included in the captured image of the camera 21, and is NO when the customer 3 leaves the imaging field of the camera 21 and disappears from the captured image. If the imaging field of the camera 21 is wide, the determination in step S103 may be NO when the customer 3 reaches a position a predetermined distance before the meeting position P1 on the entrance 2a side.

[0112] The control unit 301 tracks the customer 3 on the image captured by the camera 21 and performs a determination in step S103. If the customer 3 is on the movement route 2b1 (S103: YES), the control unit 301 determines whether the customer 3 has viewed any of the menu media 40 (S104).

[0113] When the customer 3 looks at any of the menu media 40 (S104: YES), the control unit 301 accumulates a value in the detection item of the gaze state linking information in FIG. 6 corresponding to the looked menu medium 40 (S105).

[0114] Specifically, when the determination in step S104 turns to YES, the control unit 301 adds 1 to the "number of times" corresponding to the viewed menu medium 40. Also, when the determination in step S104 turns to YES, the control unit 301 starts accumulating the time to the "time" corresponding to the viewed menu medium 40, and then when the determination in step S104 turns to NO, the control unit 301 ends this accumulation.

[0115] In this manner, while the customer 3 is on the movement route 2b1 (S103: YES), the control unit 301 accumulates values ​​for the detection items associated with the menu medium 40 viewed by the customer 3 (S104: YES, S105).

[0116] Thereafter, when the customer 3 leaves the movement route 2b1 (S103: NO), the control unit 301 refers to the gaze state linking information and identifies the menu medium 40 in which the customer 3 has shown an interest (S106).

[0117] For example, the control unit 301 identifies the menu medium 40 in which the customer 3 showed the most interest from the gaze status linking information. In this case, the menu medium 40 with the longest "time" in the gaze status linking information of the customer 3 is identified as the menu medium 40 in which the customer 3 showed the most interest. In this identification, if there are multiple menu media with the longest "time," the menu medium 40 with the most "number of times" is identified as the menu medium 40 in which the customer 3 showed the most interest. If the "number of times" is also the same, multiple menu media 40 are identified as the menu medium 40 in which the customer 3 showed the most interest.

[0118] Here, the menu medium 40 was identified by prioritizing "time" over "number of times," but the menu medium 40 in which the customer 3 showed the most interest may be identified by prioritizing "number of times." In addition, in addition to the menu medium 40 in which the customer 3 showed the most interest, the menu media 40 in which the customer 3 showed interest up to a predetermined number (e.g., second place) may be identified as the menu medium 40 in which the customer 3 showed interest.

[0119] Thus, when the menu medium 40 in which the customer 3 has shown interest is identified, the control unit 301 transmits information about the identified menu medium 40 and the face information of the customer 3 together with the serial number linked thereto to the ticket vending machine 10 (S107). This information is stored in the memory unit 102 of the ticket vending machine 10. Then, the control unit 301 ends the processing of FIG. 8.

[0120] If the customer 3 does not look at any of the menu media 40, all of the detection items for the menu media 40 associated with the customer 3 will be zero. In this case, the control unit 301 determines in step S106 that the customer 3 was not interested in any of the menu media 40, and in step S107 transmits information indicating this to the ticket vending machine 10 together with the face information and the serial number. In this case, the processing of step S107 may be skipped.

[0121] 8 is performed individually for each piece of face information of customer 3 included in the captured image of camera 21. After registering face information first acquired from the captured image of camera 21 in the gaze situation linking information, control unit 301 may register the best face information acquired while the customer 3 is on movement route 2b1 (for example, face information closest to a face in the front) in the gaze situation linking information in place of the first registered face information.

[0122] FIG. 9 is a flow chart showing the process performed by the control unit 101 of the ticket vending machine 10 in response to the customer 3 reaching the vicinity of the meeting position P1.

[0123] The control unit 101 monitors whether the customer 3 has arrived near the meeting position P1 based on the output of the human presence sensor or the like (S201). When the customer 3 has arrived near the meeting position P1 (S201: YES), face information of the customer 3 is acquired from the image captured by the camera 22, and the acquired face information is compared with face information received from the information processing device 30 during the most recent predetermined period by the process of step S107 in FIG. 8 (S202).

[0124] Here, the facial information acquired from the captured image of the camera 22 is the largest facial image near the center among the facial images included in the captured image. That is, the facial information acquired in step S202 is the facial information acquired from the customer 3 (the customer 3 operating the operation display unit 11) at the meeting position P1.

[0125] The control unit 101 receives from the information processing device 30 face information that matches the face information acquired from the image captured by the camera 22, and further determines whether or not information indicating the menu medium 40 in which the customer 3 showed interest has been received from the information processing device 30 together with the face information (S203). If the determination in step S203 is NO, the control unit 101 sets the product selection screen to be displayed on the operation display unit 11 to a default selection screen (S207).

[0126] An example of an event in which the determination in step S203 is NO is when face information of the customer 3 cannot be acquired from the image captured by the camera 21 on the travel route 2b1.

[0127] Another event that will result in a NO judgment in step S203 is when the facial information obtained from the image captured by the camera 22 matches one of the above facial information received from the information processing device 30, but no information indicating the menu medium 40 has been received from the information processing device 30 along with this facial information.

[0128] That is, as described above, when the detection items (number of times, time) of any menu medium 40 linked to this face information in the gaze state linking information are zero, the information processing device 30 transmits information indicating that the menu medium 40 was not specified together with this face information. In this case, since there is no menu medium 40 corresponding to this face information, the determination in step S203 is NO.

[0129] On the other hand, if the acquired facial information matches one of the above facial information received from the information processing device 30 and information identifying the menu medium 40 in which the customer 3 showed interest has been received together with this facial information (S203: YES), the control unit 101 identifies the product corresponding to this menu medium 40 from the linking information in Figure 7 (a) (S204), and adjusts the product selection screen so that the identified product is given priority as a selection candidate (S205).

[0130] The control unit 101 causes the operation display unit 11 to display the product selection screen set in step S205 or step S207 (S206), whereby the control unit 101 ends the process of FIG.

[0131] FIG. 10 is a diagram showing the initial selection screen 80 when the product selection screen is set as the default screen in step S207 of FIG.

[0132] Here, the product selection screen is made up of multiple hierarchical layers of selection screens. Figure 10 shows an initial selection screen 80, which is the first layer of these selection screens.

[0133] The selection screen 80 includes a message 81 prompting the selection of a product type, buttons 82 for selecting the product type, and image images 83 of representative products included in each product type. The image images 83 are arranged directly below the buttons 82 of the corresponding product types. These product types correspond to the third classification in the classification linking information in FIG. 7(b).

[0134] 10, on the initial selection screen 80 set by default, a pair of a button 82 and an image 83 is uniformly arranged for all product types. Therefore, the customer 3 selects one of the product types while treating all product types equally without being inferior or superior. When the customer 3 touches one of the buttons 82 on the selection screen 80, the control unit 101 displays a second hierarchical selection screen including a group of products belonging to the type of the button 82.

[0135] FIG. 11 is a diagram showing a second hierarchical level selection screen 90 that is displayed when the "Noodles" button 82 is operated on the selection screen 80 of FIG.

[0136] The selection screen 90 includes a message 91 prompting the customer to select a product, a display 92 showing the sub-categories of the product type (here, udon, ramen, and pasta), a button 93 for selecting a product, a button 94 for returning the screen to the previous level, and a button 95 for confirming the selection. The customer 3 touches the button 93 corresponding to the product he or she desires from the group of displayed products, and then touches button 95. This transitions the screen to a payment processing screen. If the customer 3 wishes to redo their product selection, he or she can simply touch button 94. Another product may be selected before transitioning to the payment processing screen.

[0137] FIG. 12 is a diagram showing the initial selection screen 80 when the product selection screen has been adjusted in step S205 of FIG.

[0138] The selection screen 80 in this case also includes a message 81 and buttons 82 corresponding to each type. However, in this selection screen, the image images 83 corresponding to each button 82 are omitted, and the layout of the buttons 82 has been changed accordingly. In addition, this selection screen 80 includes a message 84 recommending the selection of a specific product, and further includes a button 85 for selecting this product and an image image 86 of this product.

[0139] Here, the product displayed as recommended is the product corresponding to the menu medium 40 in which the customer 3 showed interest on the movement route 2b1. The control unit 101 acquires, from the product linking information in Fig. 7(a), the product of the menu medium 40 that is paired with the first face information that matches the face information (second face information) acquired from the camera 22, among the pairs of face information (first face information) and menu medium 40 received from the information processing device 30. Then, the control unit 101 includes a button 85 for selecting the acquired product in the selection screen 80 as shown in Fig. 12, and further includes an image 86 of this product in the selection screen 80.

[0140] As described above, when there are multiple top-ranked menu media 40 with the same number of times and frequency in the gaze status linking information, or when a predetermined number of menu media 40 from the top are identified as the menu media 40 in which the customer 3 showed interest, multiple menu media 40 are included in the pair of face information and menu media 40 received from the information processing device 30. In this case, pairs of buttons 85 and image images 86 may be displayed on the selection screen 80 for all the combined menu media 40.

[0141] When the button 85 is touched on this selection screen 80, the display of the selection screen 90 of FIG. 11 is skipped and a transition to a payment processing screen is made. Therefore, the customer 3 can smoothly and quickly select a product corresponding to the menu medium 40 in which he or she has shown interest along the movement route 2b1. Note that in this case as well, the customer 3 can operate the button 82 for selecting the product type to select a product other than the recommended product. In this case, the second hierarchical selection screen 90 similar to that of FIG. 11 is displayed.

[0142] It should be noted that, instead of the screen of FIG. 12, the screen of FIG. 13 or FIG.

[0143] 13, an image 83 of a representative product of a type (noodles in this case) including the recommended product displayed in button 85 is displayed next to a button 82 corresponding to the type. This allows this type to be displayed more effectively in the image 83 compared to other types, so that if customer 3 does not select the product of button 85, he or she can smoothly select a group of products of the same type (noodles).

[0144] In the selection screen 80 of Fig. 14, an image 83 of a representative product of a product type (here, main dish) that satisfies the recommended product displayed in button 85 and a predetermined affiliation condition (here, the condition of main dish / side dish, which is the first classification in Fig. 7(b)) is displayed below the button 82. In this case, since these types are displayed more effectively in the image 83 than other types, if the customer 3 does not select the product of button 85, he or she can smoothly select a group of products of the same type (main dish).

[0145] <Send purchase results> The control unit 101 of the ticket vending machine 10 may transmit information regarding the product actually purchased by the customer 3 facing the facing position P1 to the information processing device 30. In this case, the control unit 101 may transmit to the information processing device 30 relationship information indicating the relationship between the product on the menu medium 40 (hereinafter referred to as a "specific product") received from the information processing device 30 in combination with the face information of the customer 3, and the product actually purchased by the customer 3 (hereinafter referred to as a "purchased product").

[0146] For example, the control unit 101 may transmit, as the relationship information, information indicating whether the specific product and the purchased product match / mismatch, and information indicating the similarity of the two products in the case of mismatch, to the information processing device 30. The information indicating the similarity may include information indicating whether each classification defined in the classification linking information in Fig. 7(b) matches between the specific product and the purchased product. The evaluation criteria for similarity are not limited to this, and may be other evaluation criteria.

[0147] The control unit 101 transmits information for identifying the specific product and the purchased product (for example, product name), the above-mentioned related information, and the serial number received together with the specific product to the information processing device 30. The control unit 301 of the information processing device 30 sequentially stores the received information in association with each other.

[0148] At this time, the control unit 301 may further extract attribute information such as the gender and age of the customer 3 from the face information linked to this serial number in the gaze status linking information, and further link this attribute information to the above-mentioned information. The attribute information of the customer 3 may be acquired from a captured image (image captured by the camera 21) of the customer 3 on the movement route 2b1, and may be registered in advance in association with the gaze status linking information of the customer 3. The attribute information may include attributes other than the customer's gender and age (for example, height, color of clothes, etc.).

[0149] FIG. 15 is a diagram showing the configuration of purchase management information for managing the product purchase status of the customer 3 in the store 2. As shown in FIG.

[0150] The control unit 301 of the information processing device 30 registers the above-mentioned information received from the ticket vending machine 10 at any time in response to the purchase of a product in the purchase management information of Fig. 15. The "serial number" in the purchase management information is the same as the "serial number" in the gaze situation linking information of Fig. 6, and the purchase management information and the gaze situation linking information are linked by this "serial number."

[0151] This purchase management information can be used to evaluate, for example, the effectiveness of the menu medium 40. In other words, the information in the items "match / mismatch" and "similarity" can be used to evaluate whether the menu medium 40 stimulated the purchase of the product or a similar product, and further, this evaluation can be performed in detail for each attribute of the customer 3.

[0152] Such evaluation may be performed by an administrator on the information processing device 30, or may be performed by an external evaluation tool (server, etc.). In the latter case, the purchase management information of FIG. 15 is transmitted from the information processing device 30 to the external evaluation tool at a predetermined timing (for example, every other day, every other week, etc.). Thereafter, the information processing device 30 receives an evaluation result of the purchase management information in the store 2 from the evaluation tool. This evaluation result may include a marketing analysis of the store 2 in addition to the effectiveness of the menu medium 40 as described above.

[0153] The relationship information may be generated by the information processing device 30. In this case, information similar to the classification linking information in FIG.

[0154] <Effects of the First Embodiment> As shown in Figures 1 to 3 and 12, the information processing device 30 includes a menu identification unit 301a that identifies a menu medium 40 in which the customer 3 is interested from an image captured of the customer 3 on the movement path 2b1, and a display adjustment unit 301b that executes processing to display a product selection screen 80 (display screen) on the operation display unit 11 (display unit) so that products related to the menu medium 40 identified by the menu identification unit 301a for the customer 3 are given priority as selection candidates, at least at the arrival timing when the customer 3 arrives at the meeting position P1.

[0155] According to this configuration, the products related to the menu medium 40 in which the customer 3 showed interest along the movement path 2b1 are preferentially displayed as selection candidates on the product selection screen (display screen) at least when the customer 3 arrives at the meeting position P1. Therefore, the products in which the customer 3 showed interest can be displayed smoothly and quickly as selection candidates. This can improve the convenience of the customer 3 in selecting products.

[0156] As described above, the transmission in step S107 is performed when the customer 3 deviates from the moving route 2b1, and the process in Fig. 9 is started when the customer 3 reaches the vicinity of the meeting position P1 in step S201, that is, a position that is a predetermined distance (for example, several tens of centimeters) away from the meeting position P1 from the operation display unit 11 (display unit). Therefore, the selection screen 80 (display screen) displayed by the process of the display adjustment unit 301b is displayed at least when the customer 3 arrives at the meeting position P1.

[0157] As shown in FIG. 6, the menu identification unit 301a acquires at least one of the number of times and the cumulative time that the customer 3 viewed each menu medium 40 along the travel route 2b1, and identifies the menu medium 40 in which the customer 3 showed interest based on at least one of the number of times and the cumulative time.

[0158] This makes it possible to comprehensively determine the interest of the customer 3 in each menu medium 40. Therefore, products that match the interest of the customer 3 can be appropriately displayed as selection candidates.

[0159] As explained with reference to Figures 4(a) to (f) and Figures 5(a) to (f), the menu identification unit 301a estimates the direction in which the customer 3 is looking from the image captured by the camera 21, determines whether or not there is a menu medium 40 in that direction, and identifies the menu medium 40 in which the customer 3 has shown interest based on the determination result.

[0160] This makes it possible to appropriately determine which menu medium 40 the customer 3 looked at by using a process for estimating the viewing direction of the customer 3. Therefore, from this determination result, it is possible to accurately identify the menu medium 40 in which the customer 3 showed interest.

[0161] 6, the menu specification unit 301a composes gaze status linking information (linking information) linking the customer 3 with the menu medium 40 in which the customer 3 has shown an interest. The display adjustment unit 301b transmits information in step S107 in Fig. 8 based on the gaze status linking information (linking information), thereby causing the operation display unit 11 (display unit) to display a selection screen 80 (display screen) so that products related to the menu medium 40 in which the customer 3 has shown an interest are given priority as selection candidates.

[0162] As a result, the gaze status linking information (linking information) is prepared before the customer 3 moves to the meeting position P1, so that the selection screen 80 (display screen) according to the interests of the customer 3 can be smoothly presented to the customer 3.

[0163] As shown in FIG. 6, the menu identification unit 301a acquires face information of the customer 3 from the image captured by the camera 21, and associates the acquired face information with the menu medium 40 in which the customer 3 has shown an interest.

[0164] As a result, even without using any means for tracking the customer 3 moving from the movement path 2b1 to the meeting position P1, the menu medium 40 in which the customer 3 is interested can be smoothly identified by comparing the facial information of the customer 3 acquired from the image captured by the camera 22 near the meeting position P1 with the facial information acquired by the menu identification unit 301a in step S202 of Figure 9.

[0165] As shown in Figures 1 and 2, menu media 40 are placed within store 2, and menu identification unit 301a identifies menu media 40 in which customer 3 shows interest from captured images of customer 3 along path of movement 2b1 within store 2.

[0166] According to this configuration, it is possible to identify the menu medium 40 in which the customer 3 has shown an interest from a captured image of the customer 3 moving around the store 2.

[0167] As shown in Figures 12 to 14, the product selection screens 80, 90 displayed on the operation display unit 11 (display unit) are composed of multiple layers, and the ticket vending machine 10 (product ordering device) includes product buttons 85 (selection items) corresponding to the menu medium 40 identified by the menu identification unit 301a on the initial selection screen 80 displayed on the operation display unit 11 (display unit).

[0168] According to this configuration, the initial selection screen 80 includes product buttons 85 (selection items) corresponding to the menu media 40 in which the customer 3 has an interest, so that the customer 3 can smoothly select the product in which he or she has an interest.

[0169] As shown in FIG. 13, the initial selection screen 80 includes a plurality of buttons 82 (type selection items) for selecting a product type, and the ticket vending machine 10 (product ordering device) associates a representative image 83 with the button 82 (type selection item) for the noodle product type, which includes the product corresponding to the menu medium 40 identified by the menu identification unit 301a, among the plurality of buttons 82 (type selection items) displayed on the initial selection screen 80, thereby displaying this button 82 (type selection item) more effectively than the other buttons 82 (type selection items).

[0170] This effectively displays buttons 82 (type selection items) of product types including products corresponding to the menu media 40 in which the customer 3 has shown interest, so that if the customer 3 does not select the product in which he or she has shown interest, the customer 3 can smoothly select products of the same type.

[0171] As shown in FIG. 14, the initial selection screen 80 includes a plurality of buttons 82 (type selection items) for selecting a product type, and the ticket vending machine 10 (product ordering device) associates a representative image 83 with the buttons 82 (type selection items) of the multiple buttons 82 (type selection items) displayed on the initial selection screen 80 that correspond to the product corresponding to the menu medium 40 identified by the menu identification unit 301a and the product types (here, set meal as main dish, rice / rice bowl, and noodles) that satisfy a predetermined classification condition (here, main dish / side dish of the first classification in FIG. 7(b)), thereby displaying these buttons 82 (type selection items) more effectively than the other buttons 82 (type selection items).

[0172] According to this configuration, buttons 82 (type selection items) for products corresponding to the menu medium 40 in which the customer 3 has shown an interest and other product types that satisfy predetermined classification conditions are effectively displayed, so that when the customer 3 does not select the product in which he or she has shown an interest, the customer 3 can smoothly select products of the same type and similar products.

[0173] As shown in FIG. 8, the control unit 301 of the information processing device 30 identifies the menu medium 40 in which the customer 3 is interested from the captured image of the customer 3 on the movement route 2b1 (S103 to S106), and executes processing (S107) to display a selection screen 80 (display screen) on the operation display unit 11 (display unit) so that products related to the menu medium 40 identified for the customer 3 are given priority as selection candidates, at least at the arrival timing when the customer 3 arrives at the meeting position P1.

[0174] As a result, the information processing device 30 exhibits the above-mentioned effects.

[0175] As shown in FIG. 8, the program stored in the memory unit 302 causes the information processing device 30 to execute a function (S103 to S106) of identifying the menu medium 40 in which the customer 3 is interested from the captured image of the customer 3 on the movement route 2b1, and a function (S107) of displaying a selection screen 80 (display screen) on the operation display unit 11 (display unit) so that products related to the menu medium 40 identified for the customer 3 are given priority as selection candidates, at least at the time when the customer 3 arrives at the meeting position P1.

[0176] According to the third and fourth aspects of the present invention, the same effects as those of the first aspect can be achieved.

[0177] As a result, the information processing device 30 exhibits the above-mentioned effects.

[0178] <Embodiment 2> FIG. 16 is a plan view showing a schematic example of an arrangement of the ticket vending machine 10, the camera 23, and the menu medium 51 in the store 2 according to the second embodiment.

[0179] In the second embodiment, camera 21 and menu medium 40 are omitted from the arrangement example in Fig. 2, and camera 23 is added. Customer 3 looking at menu medium 51 on movement route 2b2 outside store 2 is imaged by camera 23. Camera 23 is capable of communicating with information processing device 30. Control unit 301 of information processing device 30 uses the function of menu identification unit 301a to identify that customer 3 has looked at menu medium 51 from the image captured by camera 23.

[0180] 6 is configured for the menu medium 51. While the customer 3 is on the movement route 2b2, the control unit 101 updates the detection items corresponding to each menu medium 51 in the gaze situation linking information. The update process of the items corresponding to the menu medium 51 is the same as that in FIG.

[0181] <Effects of the second embodiment> In the second embodiment, the same effects as in the first embodiment are achieved.

[0182] In the second embodiment, from a captured image of a customer 3 looking at a menu medium 51 outside a store 2, the menu medium 51 in which the customer 3 has shown an interest can be identified.

[0183] In the second embodiment, as in the first embodiment, the menu medium 40 and the camera 21 may be arranged in the store 2, and the gaze state linking information may be further configured for these menu media 40. In this case, if the menu medium 40 in the store 2 and the menu medium 51 outside the store 2 correspond to the same product, these menu media 40 and 51 may be linked to each other. When identifying the menu medium in which the customer 3 has shown an interest, the "time" and "number of times" corresponding to the linked menu media 40 and 51 may be added together, and the value after the addition may be used to identify the menu medium in which the customer 3 has shown an interest. When transmitting information on the linked menu media 40 and 51 from the information processing device 30 to the ticket vending machine 10, it is sufficient to transmit information on either one of the menu media.

[0184] <Embodiment 3> FIG. 17 is a block diagram showing a configuration of a product ordering system 1 according to the third embodiment.

[0185] In the third embodiment, the information processing device 30 is omitted, and the functions of the menu specification unit and the display adjustment unit are included as functions of the control unit 101 of the ticket vending machine 10. The camera 21 is connected to the communication unit 109 of the ticket vending machine 10.

[0186] The function of the menu identification unit 101a given to the control unit 101 is generally similar to the function of the menu identification unit 301a in the above-mentioned embodiment 1. However, the menu identification unit 101a does not perform the process of step S107 in the flowchart of Fig. 8, but instead stores a pair of the identified menu medium 40 and face information in the storage unit 102. In step S202 in Fig. 9, the control unit 101 compares the pair of face information stored in the storage unit 102 with face information acquired from an image captured by the camera 23. The other processes in Fig. 9 are similar to those described above.

[0187] In the configuration of the third embodiment, the functions of the display adjustment unit 101b given to the control unit 101 correspond to the functions shown in the flowchart of Fig. 9. That is, in the first embodiment, the display adjustment unit 301b executes indirect control (sending information in step S107 of Fig. 8) to cause the control unit 101 on the ticket vending machine 10 side to adjust the display screen (product selection screen). In contrast, in the third embodiment, the display adjustment unit 101b on the ticket vending machine 10 side directly executes the adjustment process of the display screen (product selection screen) of Fig. 8 in response to the identification of the menu medium 40 by the menu identification unit 101a.

[0188] In the third embodiment, the information processing device of the present invention is configured by the control unit 101, the storage unit 102, and the communication unit 109. That is, in the third embodiment, the ticket vending machine 10 has the information processing device of the present invention built therein.

[0189] <Effects of the Third Embodiment> In the third embodiment, the same effects as in the first embodiment are achieved.

[0190] In addition, in the product ordering system 1 of Figure 17, the menu identification unit 101a acquires facial information (first facial information) of the customer 3 from the image captured by the camera 21 and links the acquired first facial information to the menu medium 40 in which the customer 3 has shown interest, and the display adjustment unit 101b acquires facial information (second facial information) from the image captured of the customer 3 near the meeting position P1 and acquires the menu medium 40 linked to the first facial information that is consistent with the acquired second facial information.

[0191] According to this configuration, the menu medium 40 in which the customer 3 has shown an interest can be smoothly identified without using a means for tracking the customer who moves from the movement route 2b1 to the meeting position P1.

[0192] In the third embodiment, the same modifications as those in the second embodiment are possible.

[0193] <Example of change> In the above-mentioned first embodiment, one menu medium 40 includes one product, but one menu medium 40 may include multiple products of the same type. In this case, buttons 85 and image pictures 86 for selecting each of the multiple products corresponding to the menu medium 40 in which the customer 3 has shown an interest may be included in the selection screen 80 in Figures 12 to 14, or the button 82 of this type may be displayed more effectively than the other buttons 82.

[0194] In the above-mentioned first embodiment, the menu medium 40 in which the customer 3 showed interest was identified based on the time and number of times the customer 3 viewed the menu medium 40, but this identification may be performed based on other evaluation factors. For example, the time from the start to the end of viewing the menu medium 40 may be obtained for each time the customer 3 viewed the menu medium 40, and the obtained time may be linked to the menu medium 40 viewed, and the menu medium 40 corresponding to the longest time may be identified as the menu medium 40 in which the customer 3 showed interest. Alternatively, the menu medium 40 viewed by the customer 3 near the end of the movement route 2b1 or the menu medium 40 last viewed by the customer 3 on the movement route 2b1 may be identified as the menu medium 40 in which the customer 3 showed interest.

[0195] In the above-mentioned embodiment 1, the face information of the customer 3 is linked to the time and the number of times the customer 3 looked at each menu medium 40, but the face information of the customer 3 does not necessarily have to be linked. For example, in the case of the layout of FIG. 2, when the customer 3 moves backward out of the imaging field of the camera 21, the customer 3 usually moves to the facing position P1. Therefore, in this layout, when the customer 3 moves backward out of the imaging field of the camera 21, information on the menu medium 40 in which the customer 3 showed interest is transmitted to the ticket vending machine 10, and the ticket vending machine 10 displays the product corresponding to the menu medium 40 on the operation display unit 11 with priority as a selection candidate. As a result, when the customer 3 reaches the facing position P1, the product in which the customer 3 showed interest can be displayed with priority as a selection candidate. In this case, the camera 22 may be omitted.

[0196] Furthermore, in the above-mentioned first embodiment, the matching of the facial information is performed in the ticket vending machine 10, but the matching of the facial information may be performed in the information processing device 30. In this case, the camera 22 is connected to the information processing device 30. Furthermore, the information processing device 30 stores in the storage unit 302 a combination of the menu medium 40 in which the customer 3 has shown an interest and the facial information.

[0197] When the customer 3 reaches the vicinity of the facing position P1, the information processing device 30 receives a notification of this from the ticket vending machine 10. In response, the information processing device 30 acquires facial information of the customer 3 facing the operation display unit 11 from the image captured by the camera 22, and compares the acquired facial information with the above-mentioned combined facial information stored in the storage unit 302. If there is a facial image that matches the acquired facial image, the information processing device 30 transmits information about the menu medium 40 combined with this facial image to the ticket vending machine 10. If not, the information processing device transmits information to the ticket vending machine 10 indicating that there is no facial image that matches the acquired facial image.

[0198] When the ticket vending machine 10 receives this information, it executes the process from step S203 onwards in Fig. 9. If information about the menu medium 40 is received from the information processing device 30, the determination in step S203 is YES, and if information indicating that no matching face image was found is received, the determination in step S203 is NO. The process from step S203 onwards is the same as above.

[0199] In the above-described first embodiment, information on the menu medium 40 in which the customer 3 showed an interest is transmitted in step S107 in Fig. 8, but information on a product corresponding to the menu medium 40 may be transmitted. In this case, the product linking information in Fig. 7(a) is stored in the storage unit 302 of the information processing device 30.

[0200] In addition, in the above-mentioned embodiment 1, the functions of the menu specification unit and the display adjustment unit are included in the information processing device 30, and in the above-mentioned embodiment 3, these functions are included in the ticket vending machine 10, but these functions may be included in any device constituting the product ordering system 1, or may be distributed and arranged in a plurality of devices. For example, when the product ordering system 1 includes another information processing device in addition to the ticket vending machine 10 and the information processing device 30, the functions of the menu specification unit and the display adjustment unit may be included in this information processing device. In this case, the camera 21 is connected to the other information processing device.

[0201] Furthermore, in the above embodiments 1 to 3, the ticket vending machine 10 is the product ordering device, but a device other than the ticket vending machine 10 may be the product ordering device. For example, a device that executes the function of displaying product selection candidates and accepting orders without issuing tickets may be the product ordering device, or a device that is simply equipped with the function of displaying product selection candidates may be the product ordering device. In these cases, as in the above embodiments, it is sufficient that the products corresponding to the menu medium in which the customer 3 has an interest are preferentially displayed as selection candidates.

[0202] Furthermore, the manner in which the products corresponding to the menu medium 40 in which the customer 3 has shown an interest are preferentially displayed as selection candidates is not limited to the manner shown in FIGS. 12 to 14 in the first embodiment.

[0203] For example, in the selection screen 90 of Fig. 11, the button 93 of the product corresponding to the menu medium 40 in which the customer 3 has shown an interest may be displayed in a more emphasized manner, such as by enlarging the size or color, compared to the other buttons 93. A voice announcing recommended products may also be output. Also, the button 85 of the product corresponding to the menu medium 40 in which the customer 3 has shown an interest and its image 86 may not be displayed, and the button 82 (type selection item) of the product type including that product may be displayed more effectively compared to the other buttons 82 (type selection items).

[0204] Similarly, the method of displaying the category selection items of the product category including the product corresponding to the menu medium 40 in which the customer 3 showed an interest more effectively than the other category selection items is not limited to the method of Fig. 13, and for example, the image 83 may be omitted and the button 82 may be highlighted by enlarging its size or by using a different color. Also in the selection screen 80 of Fig. 14, the image 83 may be omitted and the button 82 may be highlighted by enlarging its size or by using a different color.

[0205] 14, the affiliation condition for the product in which the customer 3 is interested is shown to be a main dish / side dish relationship, but this affiliation condition is not limited to this. For example, this affiliation condition may be a food / drink relationship, or this affiliation condition may satisfy another relationship.

[0206] Furthermore, the linking information linking the customer 3 with the menu medium 40 in which the customer 3 showed an interest is not limited to the configuration shown in FIG. 6. As described above, face information may be omitted from the linking information, and the customer 3 may be managed by a serial number. Furthermore, the detection item may be either the number of times or the time, and other evaluation parameters may be used. The linking information may be configured to link only the customer 3 with the menu medium 40 in which the customer 3 showed an interest.

[0207] In addition, in the above-mentioned first and third embodiments, the menu medium 51 outside the store 2 is a product sample, but the menu medium 51 outside the store 2 may be other menu medium such as a poster placed on the outer wall of the store 2.

[0208] Furthermore, the layout within the store 2 and the products and product categories handled by the store 2 are not limited to those described in the above embodiment.

[0209] In addition, the embodiments of the present invention can be modified as appropriate within the scope of the claims. [Explanation of symbols]

[0210] 1. Product ordering system 2 stores 2b1, 2b2 movement route 3 customers 10 Ticket vending machine (product ordering device) 21, 22, 23 Camera 30 Information processing device 40, 51 Menu media 80, 90 selection screen 101a, 301a Menu specific section 101b, 301b Display adjustment section P1 Facing position

Claims

1. An information processing device for use in a store in which a plurality of types of menu media are arranged so as to be visible to a customer along a customer's path to a meeting position where the customer faces a display unit that displays product selection options, a menu identification unit that identifies the menu medium in which the customer has shown interest from a captured image of the customer along the moving path; and a display adjustment unit that executes a process for displaying a display screen on the display unit such that a product related to the menu medium identified by the menu identification unit for the customer is prioritized as a selection candidate at least at a timing when the customer arrives at the face-to-face position.

23. An information processing apparatus comprising:

2. 2. The information processing device according to claim 1, the menu identification unit acquires at least one of the number of times and the cumulative time that the customer has viewed each of the menu media along the movement route, and identifies the menu media in which the customer has shown an interest based on at least one of the number of times and the cumulative time.

23. An information processing apparatus comprising:

3. 2. The information processing device according to claim 1, the menu identification unit estimates a direction in which the customer is looking from the captured image, determines whether the menu medium is in that direction, and identifies the menu medium in which the customer is interested based on a result of the determination.

23. An information processing apparatus comprising:

4. 2. The information processing device according to claim 1, the menu identification unit configures linkage information linking the customer with the menu medium in which the customer has shown an interest; The display adjustment unit executes a process for displaying a display screen on the display unit such that a product related to the menu medium in which the customer has shown an interest is given priority as a selection candidate based on the association information.

23. An information processing apparatus comprising:

5. 5. The information processing device according to claim 4, the menu identification unit acquires face information of the customer from the captured image, and associates the acquired face information with the menu medium in which the customer has shown an interest.

23. An information processing apparatus comprising:

6. 2. The information processing device according to claim 1, The menu medium is placed in the store, the menu identification unit identifies the menu medium in which the customer has shown an interest from a captured image of the customer along the movement path within the store.

23. An information processing apparatus comprising:

7. 2. The information processing device according to claim 1, The menu medium is placed outside the store; the menu identification unit identifies the menu medium in which the customer has shown an interest from a captured image of the customer on the path of movement outside the store.

23. An information processing apparatus comprising:

8. A product ordering system is used in a store in which a plurality of types of menu media are arranged so as to be visible to a customer along a path of a customer to a position where the customer faces a product ordering device that displays product selection options, and the product ordering system receives product orders using the product ordering device, a menu identification unit that identifies the menu medium in which the customer has shown interest from a captured image of the customer along the moving path; and a display adjustment unit that executes a process for displaying a display screen on a display unit of the product ordering device so that a product related to the menu medium identified by the menu identification unit for the customer is given priority as a selection candidate at least at a timing when the customer arrives at the face-to-face position. A product ordering system.

9. 9. The product ordering system according to claim 8, the menu identification unit acquires first face information of the customer from the captured image, and associates the acquired first face information with the menu medium in which the customer has shown an interest; the display adjustment unit acquires second face information from a captured image of the customer in the vicinity of the face-to-face position, and acquires the menu medium associated with the first face information that matches the acquired second face information. A product ordering system.

10. 9. The product ordering system according to claim 8, The product selection screen displayed on the display unit is composed of multiple layers, the product ordering device includes, on an initial selection screen displayed on the display unit, a selection item of a product corresponding to the menu medium identified by the menu identification unit; A product ordering system.

11. The product ordering system according to claim 10, the initial selection screen includes a plurality of type selection items for selecting a product type, the product ordering device displays, among the plurality of product type selection items displayed on the initial selection screen, a product type selection item including a product corresponding to the menu medium identified by the menu identification unit more effectively than other product type selection items; A product ordering system.

12. The product ordering system according to claim 10, the initial selection screen includes a plurality of type selection items for selecting a product type, the product ordering device displays, among the plurality of type selection items displayed on the initial selection screen, a type selection item of a product type that satisfies a predetermined classification condition and a product corresponding to the menu medium identified by the menu identification unit more effectively than other type selection items; A product ordering system.

13. A method for controlling an information processing device used in a store in which a plurality of types of menu media are arranged so as to be visible to a customer along a path of the customer to a facing position where the customer faces a display unit that displays product selection options, comprising: Identifying the menu medium in which the customer showed interest from a captured image of the customer along the movement path; execute a process for displaying a display screen on the display unit so that a product related to the menu medium identified for the customer is given priority as a selection candidate at least at a timing when the customer arrives at the face-to-face position; 23. A method for controlling an information processing apparatus comprising:

14. An information processing device for use in a store in which a plurality of types of menu media are arranged so as to be visible to a customer along a customer's path to a meeting position where the customer faces a display unit that displays product selection options, A function of identifying the menu medium in which the customer showed interest from a captured image of the customer along the moving path; A program for executing a function to display a display screen on the display unit so that products related to the menu medium identified for the customer are given priority as selection candidates at least at the time when the customer arrives at the face-to-face position.

Images