Control device, control method, and recording medium

Abstract

This control device comprises a gaze detection unit, a determination unit, an in-store customer detection unit, and an in-store output control unit. The gaze detection unit detects gazes of a customer at each among product advertisements displayed on a display device installed in front of a store using an image captured by an imaging device installed so as to be able to capture images of customers looking at the display device installed in front of the store. The determination unit determines a product to be recommended to the customer on the basis of the detected gazes of the customer. The in-store customer detection unit detects the customer in the store using an image captured by an imaging device installed in the store. When the customer is detected in the store, the in-store output control unit outputs information regarding the determined product to a device in the store.

Description

Control device, control method, and recording medium

[0001] The present disclosure relates to a control device and the like.

[0002] 2. Description of the Related Art In some cases, a display device such as a digital signage is installed in front of a store to display product advertisements and the like.

[0003] Furthermore, Patent Document 1 discloses a technology that uses a camera to track the movement of customers in a store and displays promotional information about products that customers are interested in on a terminal device in the store.

[0004] International Publication No. 2015 / 033575

[0005] For example, when a product advertisement is displayed on a display device such as a digital signage installed in front of a store, even if a customer becomes interested in the product, they may not end up purchasing the product after entering the store.

[0006] An example of an object of the present disclosure is to provide a control device or the like that improves the appeal of advertisements displayed on a display device.

[0007] A control device in one aspect of the present disclosure includes: a gaze detection means for detecting a customer's gaze at each product advertisement displayed on a display device installed in front of a store using images captured by an imaging device installed so as to capture images of customers looking at the display device installed in front of the store; a determination means for determining a product to recommend to the customer based on the detected gaze of the customer; an in-store customer detection means for detecting the customer within the store using images captured by an imaging device installed in the store; and an in-store output control means for outputting information about the determined product to a device within the store when the customer is detected within the store.

[0008] In one aspect of the present disclosure, a control method includes a computer executing the following processes: using images captured by an imaging device installed in front of a store so as to capture images of customers looking at a display device installed in front of the store, detecting customers' gazes toward each of product advertisements displayed on a display device installed in front of the store; determining products to recommend to the customers based on the detected customer gazes; detecting the customers within the store using images captured by an imaging device installed in the store; and, when the customers are detected within the store, outputting information about the determined products to a device in the store.

[0009] A program in one aspect of the present disclosure causes a computer to perform the following processes: detect a customer's line of sight toward each product advertisement displayed on a display device installed in front of a store using images captured by an imaging device installed in such a way that the customer is able to capture an image of the customer looking at the display device installed in front of the store; determine a product to recommend to the customer based on the detected customer's line of sight; detect the customer within the store using images captured by an imaging device installed in the store; and, when the customer is detected within the store, output information about the determined product to a device in the store.

[0010] Each program may be stored in a non-transitory computer-readable recording medium.

[0011] According to the present disclosure, it is possible to improve the appeal of advertisements displayed on display devices.

[0012] 1 is a first block diagram showing an example configuration of a control device according to the present disclosure; FIG. 2 is a flowchart showing an example operation of the control device; FIG. 3 is an explanatory diagram showing an example in which a device is installed in front of a store; FIG. 4 is an explanatory diagram showing an example of connection between a control device and other devices; FIG. 5 is a second block diagram showing an example configuration of a control device according to the present disclosure; FIG. 6 is an explanatory diagram showing an example in which a product that a customer has focused on among advertisements displayed on digital signage in front of a store is displayed on digital signage within the store so that the product can be ordered; FIG. 7 is an explanatory diagram showing an example in which information about a product that a customer has focused on among advertisements displayed on digital signage in front of the store and an image of the customer are displayed on a terminal device of a store clerk within the store; FIG. 8 is an explanatory diagram showing an example display of an indicator related to the length of a customer's gaze; FIG. 9 is an explanatory diagram showing example 1 in which an image combining a main product and a secondary product is displayed; FIG. 10 is an explanatory diagram showing example 2 in which an image combining a main product and a secondary product is displayed; FIG. 11 is a flowchart showing an example operation of the control device; FIG. 12 is an explanatory diagram showing an example hardware configuration of a computer.

[0013] Hereinafter, embodiments of a control device, a control method, a program, and a non-transitory recording medium for recording a program according to the present disclosure will be described in detail with reference to the accompanying drawings. The disclosed technology is not limited to these embodiments.

[0014] First Embodiment First, a first embodiment will be described. In the first embodiment, an example of basic functions of a control device will be described. FIG. 1 is a first block diagram showing an example configuration of a control device according to the present disclosure. The control device 10 includes a gaze detection unit 101, a determination unit 102, an in-store customer detection unit 103, and an in-store output control unit 104.

[0015] For example, the gaze detection unit 101 detects the gaze of customers toward each product advertisement displayed on the display device installed in front of the store using images captured by an imaging device installed in front of the store so as to capture images of customers looking at the display device installed in front of the store. Here, the store refers to, for example, a retail store where customers purchase products. The type of store is not particularly limited, and may include a restaurant, apparel store, supermarket, drugstore, etc. Here, the "front of the store" refers to a location visible to customers who may enter the store, and may be a location within the store or its surroundings. The display device installed in front of the store is not particularly limited, and may be a digital signage display or the like. The imaging device is installed so as to capture an image that can detect the gaze of customers toward the display device. The imaging device may be installed in front of the store, for example. For example, the imaging device installed in front of the store may be an imaging device included in the display device installed in front of the store, or may be another imaging device such as a surveillance camera. The advertisement may include an image of the product or text describing the product's features, but is not particularly limited.

[0016] The determination unit 102 determines products to recommend to the customer based on the detected customer's gaze. These products are also referred to as recommended products. The number of recommended products is not particularly limited. For example, the determination unit 102 may determine recommended products from products for which advertisements are displayed. For example, the determination unit 102 may determine recommended products from products for which advertisements are displayed based on at least one of the time for which the detected customer's gaze is directed toward each advertisement and the number of times the gaze is directed toward each advertisement.

[0017] More specifically, the determination unit 102 may determine, as the recommended product, a product on which the customer's gaze is directed for a long time. The length of time the customer's gaze is directed may be determined based on a relative comparison or an absolute comparison. As an example of relative comparison, the determination unit 102 may determine, as the recommended product, a product on which the customer's gaze is directed for the longest time. The determination unit 102 is not limited to the product with the longest gaze time, and may be, for example, the product with the second longest gaze time. As another example of relative comparison, the determination unit 102 may determine, as the recommended product, a predetermined number of products in descending order of the length of time the customer's gaze is directed. As an example of absolute comparison, the determination unit 102 may determine, as the recommended product, products on which the customer's gaze is directed for a predetermined period of time or longer. The determination unit 102 may determine, as the recommended product, products on which the customer's gaze is directed for a predetermined period of time or longer if there is no product on which the customer's gaze is directed for a predetermined period of time or longer. Alternatively, the determination unit 102 may determine, as the recommended product, a predetermined product on which the customer's gaze is directed for a predetermined period of time or longer. Relative comparison and absolute comparison may be combined. For example, when there is no product to which the customer's gaze has been directed for a predetermined period of time or longer, the determination unit 102 may determine a recommended product using an example of relative comparison.

[0018] Alternatively, the determination unit 102 may determine, as the recommended product, a product to which customers have directed their gazes a large number of times. The number of times that customers have directed their gazes may be determined based on a relative comparison or an absolute comparison. As an example of relative comparison, the determination unit 102 may determine, as the recommended product, a product to which customers have directed their gazes the most. As another example of relative comparison, the determination unit 102 may determine, as the recommended product, a predetermined number of products in descending order of the number of times that customers have directed their gazes. As an example of absolute comparison, the determination unit 102 may determine, as the recommended product, a product to which customers have directed their gazes a predetermined number or more times. Note that the determination unit 102 may determine that there is no recommended product if there is no product to which customers have directed their gazes a predetermined number or more times. Alternatively, the determination unit 102 may determine a predetermined product to be the recommended product if there is no product to which customers have directed their gazes a predetermined number or more times. Relative comparison and absolute comparison may be combined. For example, when there is no product that has been gazed upon by the customer a predetermined number of times or more, the determination unit 102 may determine a recommended product using an example of relative comparison.

[0019] Furthermore, the determination unit 102 may determine recommended products by combining the time that the customer's gaze is directed and the number of times that the customer's gaze is directed. For example, the determination unit 102 may determine, as recommended products, products for which the detected customer's gaze is directed for a long time and the number of times that the customer's gaze is directed.

[0020] Next, the in-store customer detection unit 103 detects customers within the store using images captured by an imaging device installed within the store. For example, the in-store customer detection unit 103 detects customers who are the same as customers who were in front of the store using images captured by an imaging device installed within the store. A specific example of the detection method will be described in the second embodiment.

[0021] Then, when a customer is detected in the store, the in-store output control unit 104 outputs information about the determined recommended products to a device in the store. Various output methods are available, such as display, audio output, lighting output, and vibration output. The type of device in the store is not particularly limited. For example, the device in the store may be a terminal device operated by a store clerk. For example, a customer may place an order in the store by operating a digital signage or tablet-type terminal device installed in the store. Therefore, for example, the device installed in the store may be a device that can be operated by the customer. The information about the recommended products is not particularly limited, such as an image of the recommended product, the price of the recommended product, the name of the recommended product, and the features of the recommended product.

[0022] (Flowchart) Figure 2 is a flowchart showing an example of the operation of the control device 10. The gaze detection unit 101 detects the gaze of customers toward an advertisement displayed on a display device installed in front of the store using an image captured by an imaging device installed in front of the store so as to capture an image of the customer looking at the display device (step S101). The determination unit 102 then determines recommended products based on the detected gaze of the customer (step S102). The in-store customer detection unit 103 detects customers within the store (step S103). The in-store output control unit 104 outputs information about the recommended products to a device in the store (step S104), and the control device 10 ends the series of processes.

[0023] Here, the order of the processes of step S102 and step S103 is not particularly limited. However, in the example of FIG. 2, the determination unit 102 determines recommended products regardless of whether the customer enters the store or not. This allows the in-store output control unit 104 to immediately output information about recommended products to a device in the store when a customer is detected. On the other hand, by performing step S102 after step S103, the determination unit 102 can narrow down recommended products to customers who enter the store.

[0024] As described above, in the first embodiment, the control device 10 determines recommended products based on the customer's line of sight toward an advertisement displayed on a display device installed in front of a store. Then, when the control device 10 detects the customer in the store, it outputs information about the recommended products to a device inside the store. This allows the product that the customer is looking at in front of the store to be determined as the recommended product. Then, when the customer enters the store, the arc customer can instantly recommend a product that the customer is interested in on the display device in front of the store. This improves the appeal of advertisements displayed on a display device installed in front of the store.

[0025] Second Embodiment Next, a second embodiment will be described in detail with reference to the drawings. In the second embodiment, a specific example of processing by the control device will be described using a specific usage scenario as an example. Below, to the extent that the description of the second embodiment is not unclear, descriptions of content that overlaps with the above description will be omitted.

[0026] Fig. 3 is an explanatory diagram showing an example in which a device is installed in front of a store, and Fig. 4 is an explanatory diagram showing an example of connections between the control device and other devices.

[0027] The store may be a single store as shown in Figure 3, or may be a collection of multiple stores such as a shopping mall. In a shopping mall, the front of a store may not refer to each individual store, but may refer to a shared area.

[0028] For example, the digital signage 21 is an example of a display device installed in front of a store. The digital signage 21 is connected to the control device 20 via a communication network. The digital signage 21 may include, for example, an imaging device. Note that the "front of the store" here refers to a position visible to customers who may enter the store, and the digital signage 21 may be installed in the store or a location in the vicinity of the store. For example, the digital signage 21 is not limited to being installed in front of the store, but may also be installed inside the store, such as near the entrance or near the cash register. For example, the terminal device 23 may be pre-installed with an application program that can output information from the control device 20 or transmit information to the control device 20.

[0029] The digital signage 22 is an example of a device installed in a store. For example, the digital signage 22 is connected to the control device 20 via a communication network. The digital signage 22 may include, for example, an imaging device. The digital signage 22 may be a device that can be operated by a user via an input device. For example, the terminal device 23 may be pre-installed with an application program that can output information from the control device 20 or transmit information to the control device 20.

[0030] The terminal device 23 is another example of a device installed in a store. For example, the terminal device 23 is connected to the control device 20 via a communication network. For example, the terminal device 23 may be pre-installed with an application program that can output information from the control device 20 or transmit information to the control device 20. For example, the terminal device 23 may access the website of the control device 20 via the communication network NT. The type of the terminal device 23 is not particularly limited, and may be a PC (Personal Computer), a smartphone, a tablet device, or the like. The number of terminal devices 23 may be prepared for each user, and is not particularly limited.

[0031] The imaging device 24 is an example of an imaging device installed in a store. The imaging device 24 is connected to the control device 20 via a communication network. The imaging device 24 is, for example, a camera capable of capturing images of customers entering the store. The location in the store where the imaging device 24 is installed is not particularly limited. For example, the imaging device 24 may be installed in a position where it can immediately recommend recommended products to customers when they enter the store. Alternatively, for example, the imaging device 24 may be installed so as to be able to capture images of users operating the digital signage 22.

[0032] The type of communication network is not particularly limited, and may be configured by a plurality of communication networks.

[0033] 5 is a second block diagram showing an example configuration of a control device according to the present disclosure. The control device 20 includes a gaze detection unit 201, a determination unit 202, an in-store customer detection unit 203, an in-store output control unit 204, a front-of-store customer detection unit 205, and a front-of-store output control unit 206.

[0034] The gaze detection unit 201 may have, as a basic function, the function of the gaze detection unit 101 shown in Fig. 1. The determination unit 202 may have, as a basic function, the function of the determination unit 102 shown in Fig. 1. The in-store customer detection unit 203 may have, as a basic function, the function of the in-store customer detection unit 103 shown in Fig. 1. The in-store output control unit 204 may have, as a basic function, the function of the in-store output control unit 104 shown in Fig. 1.

[0035] The control device 20 may also have a product database (DB) 2001. The product DB 2001 may be used to display product advertisements or to display recommended products. For example, the product DB 2001 stores product information for each product. The product DB 2001 stores, for example, product identification information for identifying a product in association with product information. Examples of product information include product name, product type, product price, product features, and product images. The product type may be a general or subcategory of the product and is not particularly limited. For example, in the product DB 2001 for apparel products, the product type may be a type such as tops or bottoms. For example, in the case of a restaurant, the product type may be a type such as a main dish or a side dish. The product feature may be a product feature used in an advertisement. The product image is, for example, an image of the product. The number of product images is not particularly limited. The product image may be an image of the product combined with other products. For example, in the case of apparel products, the product images may be images of the product being worn by a person, a mannequin, etc. The product images may include images for use in advertising the product, images for use in ordering the product, etc. The multiple product images may be taken from different angles.

[0036] Next, each functional unit will be described.

[0037] The in-store output control unit 206 displays product advertisements on the digital signage 21. Existing technology may be used as the method for displaying the advertisements. However, advertisement display information, such as which product advertisement is displayed at which position, is stored in a storage device or the like. The advertisement display information, such as which product advertisement is displayed at which position on the digital signage 21, is used by the determination unit 202 to determine recommended products. Furthermore, when the display of advertisements or the like is switched, the advertisement display information may include a timestamp or the like. This allows the determination unit 202 to identify the product advertisements and their display positions that are displayed at each time.

[0038] The front-of-store customer detection unit 205 detects customers in front of the store, for example, using an image captured by an imaging device installed in front of the store. The front-of-store customer detection unit 205 may detect people from the image. For example, if the imaging device is provided in the digital signage 21, the front-of-store customer detection unit 205 may detect customers using an image captured by the imaging device of the digital signage 21. More specifically, the front-of-store customer detection unit 205 may detect customers in front of the digital signage 21 using an image captured by the imaging device of the digital signage 21. The imaging device installed in front of the store may be separate from the digital signage 21.

[0039] Here, the in-store customer detection unit 205 may detect characteristics of customers in front of the store from the image. Customer characteristic information representing the characteristics of customers detected by the in-store customer detection unit 205 may be stored in a storage device or the like. This customer characteristic information is used by the in-store customer detection unit 203 to detect whether a customer who was looking at the digital signage 21 in front of the store has entered the store. Therefore, the customer characteristic information is not particularly limited as long as it represents characteristics that allow a customer to be identified. For example, the customer characteristic information may be a facial image of the customer's face, or a facial feature amount of the customer that can be identified from the facial image of the customer's face. In this way, the customer characteristic information may be detailed biometric information of the customer. Alternatively, the customer characteristic information may be information related to the customer's appearance. For example, the information related to appearance may be information such as clothing, body type, age, and mood.

[0040] Next, the gaze detection unit 201 uses an image captured by an imaging device installed in front of the store to detect the gaze of the customer detected by the store-front customer detection unit 205. The imaging device installed in front of the store may be included in the digital signage 21.

[0041] The determination unit 202 then determines the recommended product based on the customer's line of sight. The determination unit 202 determines the recommended product based on the customer's line of sight and advertisement display information about the advertisement displayed on the digital signage 21. The determination unit 202 can identify which product the customer is looking at by comparing the customer's line of sight with the advertisement display information. As described in the first embodiment, the determination unit 202 may also determine the recommended product based on at least one of the duration and number of times the customer directs their gaze. The determination unit 202 may then associate the customer characteristic information with recommended product identification information that identifies the recommended product and store the information in a storage device or the like.

[0042] Next, the in-store customer detection unit 203 detects customers within the store using images captured by the imaging device 24 installed within the store. For example, the in-store customer detection unit 203 detects the same customer as the customer detected by the in-store customer detection unit 205 using images captured by the imaging device 24 installed within the store. As a specific process for detecting the same customer, for example, the in-store customer detection unit 203 detects the characteristics of the person from the image captured by the imaging device 24 installed within the store. Then, the in-store customer detection unit 203 may detect the same customer as the customer detected in front of the store by comparing the characteristics of the person detected within the store with the characteristics of the person detected by the in-store customer detection unit 205.

[0043] For example, when a customer is detected in the store by the in-store customer detection unit 203, the in-store output control unit 204 outputs information about the determined recommended products to a device in the store. Specifically, for example, as a method of acquiring information about recommended products, the in-store output control unit 204 may acquire recommended product identification information associated with the customer characteristic information from a storage device that stores customer characteristic information and recommended product identification information that identifies recommended products in association with each other, and then acquire information about the products associated with the recommended product identification information from the product DB 2001.

[0044] For example, if the in-store device is a display device installed in the store, such as digital signage 22, the in-store output control unit 204 may output information about the determined recommended products when a customer is detected, so that the recommended products can be ordered. The in-store output control unit 204 can acquire information about the recommended products from the product DB 2001. An example in which information about recommended products is displayed so that they can be ordered will be described using FIG. 6 .

[0045] 6 is an explanatory diagram showing an example in which products that a customer has noticed among advertisements displayed on a digital signage 21 in front of a store are displayed on a digital signage 22 inside the store so that they can be ordered. In FIG. 6, a screen including advertisements for various foods is displayed on the digital signage 21.

[0046] A series of processes in FIG. 6 will be briefly described. The in-store customer detection unit 205 detects a customer in front of the digital signage 21, and the gaze detection unit 201 detects the customer's gaze. In FIG. 6, the customer is gazing at the beef bowl. Therefore, the determination unit 202 determines the beef bowl as the customer's recommended product. Then, when the customer enters the store, the in-store customer detection unit 203 detects the customer within the store. The in-store output control unit 204 displays information about the recommended product on the digital signage 22 so that the recommended product can be ordered. For example, the digital signage 22 displays a screen including an image of the beef bowl that the customer is looking at on the digital signage 21 in front of the store and the price of the beef bowl. The screen also includes an order button. For example, when the order button is pressed, the order receiving unit 207 receives an order for the recommended product. The order receiving unit 207 may register information about the recommended product in the customer's order list. Note that the screen may also include a button or the like that allows the customer to view a menu of other products.

[0047] For example, if the device in the store is a terminal device 23, particularly a store clerk's terminal device 23, when a customer is detected by the store customer detection unit 203, the in-store output control unit 204 may display an image of the customer along with information about recommended products on the store clerk's terminal device 23. Note that the in-store output control unit 204 is not limited to displaying an image of the customer, and may also display characteristic information of other customers along with information about recommended products. This allows the store clerk to confirm which products to recommend to which customers.

[0048] 7 is an explanatory diagram showing an example in which information about a product that a customer has focused on among advertisements displayed on digital signage 21 in front of a store and an image of the customer are displayed on a terminal device 23 of a store clerk. In FIG. 7, a screen including advertisements for various clothes is displayed on digital signage 21.

[0049] The series of processes in FIG. 7 will be briefly described. The in-store customer detection unit 205 detects a customer in front of the digital signage 21, and the gaze detection unit 201 detects the customer's gaze. In FIG. 7, the customer is gazing at the polka-dot shirt, which is the top item of clothing displayed. Therefore, the determination unit 202 determines the polka-dot shirt as the customer's recommended product. Then, when the customer enters the store, the in-store customer detection unit 203 detects the customer within the store using an image captured by the imaging device 24. The in-store output control unit 204 displays an image of the customer on the terminal device 23 along with information about the polka-dot shirt. For example, the digital signage 22 displays a screen including an image, price, and name of the polka-dot shirt that the customer was looking at on the digital signage 21 in front of the store. Furthermore, an image of the customer is displayed on the screen.

[0050] Taking a shopping mall as an example, advertisements for products from each store are displayed on digital signage 21 installed in a common area. A customer may like the advertisement displayed on digital signage 21 and actually visit the store. Then, by checking the screen displayed on terminal device 23, a store clerk at the store can recommend products to the customer who visited the store after seeing the advertisement displayed on digital signage 21. Therefore, the control device 20 can support customer service.

[0051] Furthermore, in a case where there are multiple stores, such as in a shopping mall, and advertisements for products from different stores are displayed on the digital signage 21, the determination unit 202 may determine recommended products for each store. The determination unit 202 may then associate the customer characteristic information, store identification information, and recommended product identification information and store them in a storage device or the like.

[0052] For example, a customer may want to add a product they like after seeing an advertisement to a favorites list. Therefore, the determination unit 202 may have a function of adding a product that the customer is interested in to the favorites list from among the products displayed on a display device installed in front of the store. The determination unit 202 may register information about a product that the customer looks at for a predetermined period of time or more in the favorites list. The determination unit 202 may also determine at least one of the products registered in the favorites list as a recommended product. To more clearly confirm whether the customer selected a product of their own volition, the in-store output control unit 206 may display an indicator on the digital signage 21 indicating either the length or number of times the customer looked at the product. The method of displaying the indicator is not particularly limited. An example of the indicator is shown in FIG. 8 .

[0053] FIG. 8 is an explanatory diagram showing an example of an indicator display indicating the length of time a customer has gazed. In a function that registers an item in a favorites list after gazing for five seconds, the in-store output control unit 206 displays information such as the current time in the five seconds as an indicator. In FIG. 8 , for example, the indicator can identify the current time in the five seconds. While the indicator is displayed as a circle, the shape is not particularly limited. For example, a customer is gazing at a polka-dot shirt. When the customer gazes at the polka-dot shirt for five seconds, the determination unit 202 registers information about the polka-dot shirt in the customer's favorites list. Note that when a favorites list is used, the control device 20 may perform biometric authentication. For example, the determination unit 202 registers information about products viewed for five seconds or more in the favorites list of a customer authenticated by biometric authentication. The determination unit 202 then determines the products registered in the favorites list as recommended products.

[0054] Furthermore, multiple products may be usable in combination. "Usable in combination" means that, in the case of food products, they can be eaten in combination, or, in the case of apparel products, they can be worn in combination. Therefore, the in-store output control unit 206 displays an advertisement for the main product and advertisements for secondary products that can be used with the main product on a display device installed in front of the store. Specifically, for example, the in-store output control unit 206 displays an image of the main product and images of secondary products that can be used with the main product on a display device installed in front of the store. The main product and secondary products may be determined by the store, or may be determined based on the product type in the product information stored in the product DB 2001. For example, the in-store output control unit 206 may display advertisements for multiple different secondary products.

[0055] The determination unit 202 determines a recommended product from the secondary products based on the customer's line of sight. When a customer is detected by the in-store customer detection unit 203, the in-store output control unit 204 displays an image combining the secondary product, which is a recommended product determined based on the customer's line of sight detected by the in-store customer detection unit 205, with the main product. The combined image may be stored in advance in the product DB 2001 or may be newly generated using an image generation AI (Artificial Intelligence) model or the like. The in-store output control unit 206 may also display an image combining an image of the main product with images of secondary products that can be used with the main product on a display device installed in front of the store. The combined image may simply be an image of the main product and the secondary product side by side, or may be an image of the main product and the secondary product being used together.

[0056] Furthermore, if the in-store device is a digital signage 22 that can be operated by a user such as a customer, the in-store output control unit 204, when a customer is detected in the store, displays the combined image on the in-store device so that the product included in the combined image can be ordered. That is, the in-store output control unit 204 displays the combined image so that both the main product and the sub-product can be ordered. For example, the in-store output control unit 204 may display the total price of the main product and the sub-product. An example of how a combined image is displayed will be described using FIG. 9 .

[0057] 9 is an explanatory diagram showing an example 1 in which an image combining a main product and a secondary product is displayed. In FIG. 9, a screen including an advertisement for a beef bowl as the main product and advertisements for a number of different beef bowl toppings as secondary products is displayed on the digital signage 21.

[0058] A series of processes in FIG. 9 will be briefly described. The in-store customer detection unit 205 detects a customer in front of the digital signage 21, and the gaze detection unit 201 detects the customer's gaze. In FIG. 9, the customer is gazing at an image of cheese. Therefore, the determination unit 202 determines cheese as the customer's recommended item from among multiple secondary items. Then, when the customer enters the store, the in-store customer detection unit 203 detects the customer within the store. The in-store output control unit 204 displays an image on the digital signage 22 combining the main item and secondary item so that the determined secondary item can be ordered together with the main item. In FIG. 9, the combined image is an image of a beef bowl topped with cheese. The in-store output control unit 204 may also display the respective prices of the main item and secondary item on the digital signage 22.

[0059] For example, the digital signage 22 displays a screen including an image of a beef bowl topped with cheese that the customer has been looking at on the digital signage 21 in front of the store, as well as the prices of the beef bowl and the cheese. The screen also displays the total price for the beef bowl and the cheese. The screen also includes an order button. For example, when the order button is pressed, the order receiving unit 207 receives an order for both the beef bowl and the cheese. The order receiving unit 207 simply registers information about the received products in the customer's order list. The screen may also include buttons that allow the customer to view menus for other products, or may include buttons that allow the customer to delete secondary products.

[0060] Furthermore, in the example of the beef bowl, the combined image is an image of the actual toppings, but the combination may be different depending on the type of side item or the type of main item.

[0061] Next, another example of how a combined image is displayed will be described with reference to FIG.

[0062] 10 is an explanatory diagram showing Example 2 in which an image combining a main product and a secondary product is displayed. In Fig. 10, the digital signage 21 displays a screen including an advertisement for a skirt as the main product and advertisements for a number of different tops as secondary products.

[0063] A series of processes in FIG. 10 will be briefly described. The in-store customer detection unit 205 detects a customer in front of the digital signage 21, and the gaze detection unit 201 detects the customer's gaze. In FIG. 10, the customer is gazing at a polka-dot shirt, which is the top item displayed among multiple secondary products. Therefore, the determination unit 202 determines the polka-dot shirt as the customer's recommended product. When the customer enters the store, the in-store customer detection unit 203 detects the customer within the store using an image captured by the imaging device 24. The in-store output control unit 204 displays an image on the terminal device 23 that combines the polka-dot shirt, which is the secondary product, with a skirt, which is the main product. Furthermore, an image of the customer is displayed on the screen. Note that other product information, such as the name and price of the secondary product and the name and price of the main product, may also be displayed. The total amount, etc. may also be displayed.

[0064] 11 is a flowchart showing an example of an operation performed by the control device 20. Here, an example will be described in which recommended products are displayed in an orderable manner on the digital signage 22 in the store.

[0065] The front-of-store output control unit 206 displays product advertisements on the digital signage 21 installed in front of the store (step S201). The front-of-store customer detection unit 205 detects customers in front of the digital signage using images captured by an imaging device provided in the digital signage 21 (step S202).

[0066] The gaze detection unit 201 detects the customer's gaze toward the advertisement using an image captured by an imaging device provided in the digital signage 21 (step S203). Note that the image here is an image acquired in real time. The determination unit 202 determines a recommended product based on the customer's gaze (step S204).

[0067] The in-store customer detection unit 203 determines whether the customer detected in front of the store has been detected inside the store using an image captured by an imaging device provided in the digital signage 22 installed in the store (step S205). For example, in step S205, the in-store customer detection unit 203 may match the person detected inside the store with the customer detected in front of the store every time a person is detected using an image captured by the imaging device provided in the digital signage 22 installed in the store.

[0068] If the customer detected in front of the store is not detected inside the store (step S205: No), the in-store customer detection unit 203 returns to step S205. Note that, for example, in step S205, if the customer detected in front of the store is not detected inside the store even after a certain period of time has elapsed, the control device 20 may end the processing. Alternatively, for example, in step S205, if the customer detected in front of the store is not detected inside the store at a predetermined timing, such as when the store is closing, the control device 20 may end the processing.

[0069] If the customer detected in front of the store is detected inside the store (step S205: Yes), the in-store output control unit 204 displays information about the recommended products on the digital signage 22 installed inside the store so that the recommended products can be ordered (step S206).Then, the control device 20 ends the series of processes.

[0070] The order of steps S201 and S202 may be reversed. For example, when a customer is detected in front of digital signage 21, the in-store output control unit 206 may display a product advertisement. The order of steps S204 and S205 may be reversed. For example, when a customer detected in front of digital signage 21 in front of the store is detected in front of digital signage 22 inside the store, the determination unit 202 may determine a recommended product.

[0071] As described above, in the second embodiment, the control device 20 determines, as a recommended product, an advertisement for a product that a customer who is detected in front of the digital signage 21 in front of a store is looking at, and when that customer is detected in front of the digital signage 22 inside the store, displays information about the recommended product on the digital signage 22. This allows a customer to easily enter the store and find a product that catches their eye after seeing the digital signage 21 in front of the store. Also, for example, there are cases where a customer can order a product by operating the digital signage 22. Therefore, when displaying information about recommended products on the digital signage 22, the control device 20 displays the recommended product as orderable. This allows a customer to easily order a product that catches their eye while in front of the store.

[0072] Furthermore, the control device 20 determines as a recommended product an advertisement for a product that a customer detected in front of the digital signage 21 in front of the store is looking at, and when that customer is detected in the store, displays information about the recommended product on the store clerk's terminal device 23. This allows the store clerk to easily check products that the customer is interested in. Furthermore, when displaying information about recommended products on the store clerk's terminal device 23, the control device 20 may also display an image of the customer. This allows the store clerk to easily check which customers are interested in which products. Therefore, the control device 20 can assist the store clerk in serving customers.

[0073] The control device 20 also displays advertisements for the main product and for secondary products that can be used together with the main product on a digital signage 21 installed in front of the store. The control device 20 then determines a recommended product from the secondary products based on the customer's line of sight, and when a customer is detected in the store, displays an image combining the secondary product determined as the recommended product with the main product. This makes it possible to recommend a combination of multiple products. Furthermore, when displaying the combined image on, for example, a user-operable digital signage 22, the control device 20 displays the secondary product determined as the recommended product and the main product in an orderable format. This allows customers to easily order a combination of products.

[0074] Therefore, the control device 20 determines the product to recommend to the customer based on either the length or the number of times the customer's gaze is directed at the product, thereby easily identifying which product the customer is paying attention to.

[0075] The control device 20 may also display an indicator on the digital signage 21 indicating the length or frequency of gaze. This allows the customer to check which product he or she is paying attention to. If the product that the customer is paying attention to can be added to the customer's favorites list, the customer can check which product can be added to the favorites list by checking the indicator.

[0076] This concludes the description of each embodiment. Furthermore, each embodiment may be combined. In each embodiment, the control device 20 may be configured to include some of the functional units and information.

[0077] Furthermore, the embodiments are not limited to the above examples and may be modified in various ways. For example, the example in which images of recommended products are displayed on the digital signage 22 in a store so that they can be ordered has been given. For example, images of recommended products may be displayed on a tablet-type terminal device 23 at a table seat in a restaurant or the like. For example, when the in-store customer detection unit 203 detects that a customer has sat at a table seat using an image captured by the imaging device 24, the in-store output control unit 204 may display images of recommended products on the tablet-type terminal device 23 so that the recommended products can be ordered. The tablet-type terminal device 23 may be equipped with an imaging device, and the in-store customer detection unit 203 may detect that a customer detected in front of the store has sat at a table seat using an image captured by the imaging device of the tablet-type terminal device 23.

[0078] Although a distinction has been made between a device in front of a store and a device inside the store, this may also be achieved by a device for displaying advertisements and a device for ordering. For example, the device for displaying advertisements may be installed in front of the store or inside the store.

[0079] Furthermore, the configuration of the control devices 10 and 20 in the embodiments is not particularly limited. For example, the functional units of the control devices 10 and 20 may be implemented by a single device. Alternatively, for example, each functional unit or DB of the control devices 10 and 20 in the embodiments may be implemented by a different device and configured as a system. For example, each functional unit of the control devices 10 and 20 may be implemented by multiple devices, such as multiple servers, and configured as a system. For example, each functional unit may be implemented by a database server including each DB and a server having each functional unit. As an example of a system implementation, if existing technology is used for the in-store output control unit 206 of the control device 20, a system may be implemented that includes an existing server including the in-store output control unit 206 and a server including other functional units of the control device 20. Each functional unit of the control devices 10 and 20 may be implemented by a cloud server or the like. For example, some of the functional units of the control devices 10 and 20 may be implemented by digital signage 21 and 22.

[0080] In the embodiment, each DB may include part of the above-mentioned information. Also, each piece of information may include information other than the above-mentioned information.

[0081] The process of generating information to be displayed on the digital signage 21 may be performed by the in-store output control unit 206 of the control device 20. This process may also be performed by the digital signage 21. That is, the digital signage 21 generates information for a screen to be displayed on the digital signage 21 based on data received from the control device 20, and displays the screen.

[0082] The process of generating information to be displayed on the digital signage 22 may be performed by the in-store output control unit 204 of the control device 20. This process may also be performed by the digital signage 21. That is, the digital signage 22 generates information for a screen to be displayed on the digital signage 22 based on data received from the control device 20, and displays the screen.

[0083] The process of generating information to be displayed on the terminal device 23 may be performed by the in-store output control unit 204 of the control device 20. This process may also be performed by the terminal device 23. That is, the terminal device 23 generates information for a screen to be displayed on the terminal device 23 based on data received from the control device 20, and displays the screen.

[0084] (Example of Computer Hardware Configuration) Next, an example of a hardware configuration in which each device, such as the control devices 10 and 20, the digital signage 21, the digital signage 22, and the terminal device 23 described in the embodiment, is realized by a computer will be described. Fig. 12 is an explanatory diagram showing an example of a computer hardware configuration. For example, some or all of each device can be realized using any combination of a computer 80 and a program as shown in Fig. 12.

[0085] The computer 80 includes, for example, a processor 801, a ROM (Read Only Memory) 802, a RAM (Random Access Memory) 803, and a storage device 804. The computer 80 also includes a communication interface 805 and an input / output interface 806. The components are connected to each other, for example, via a bus 807. The number of each component is not particularly limited, and there may be one or more of each component.

[0086] The processor 801 controls the entire computer 80. The processor 801 may be, for example, a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, or a combination thereof, and is not particularly limited.

[0087] The computer 80 also includes a ROM 802, a RAM 803, and a storage device 804. Examples of the storage device 804 include a semiconductor memory such as a flash memory, a hard disk drive (HDD), and a solid state drive (SSD). For example, the storage device 804 stores an operating system (OS) program, application programs, and programs related to the embodiments. Alternatively, the ROM 802 stores application programs and programs related to the embodiments. The RAM 803 is used as a work area for the processor 801.

[0088] The processor 801 also loads programs stored in the storage device 804, ROM 802, etc. The processor 801 then executes each process coded in the program. The processor 801 may also download various programs via the communication network NT. The processor 801 also functions as a part or all of the computer 80. The processor 801 may then execute the processes or instructions in the illustrated flowchart based on the program.

[0089] The communication interface 805 is connected to a communication network NT such as a LAN (Local Area Network) or a WAN (Wide Area Network) via a wireless or wired communication line. The communication network NT may be composed of multiple communication networks NT. As a result, the computer 80 is connected to external devices and external computers 80 via the communication networks NT. The communication interface 805 serves as an interface between the communication network NT and the inside of the computer 80. The communication interface 805 also controls the input and output of data from external devices and external computers 80.

[0090] Furthermore, the input / output interface 806 is connected to at least one of an input device, an output device, and an input / output device. The connection method may be wireless or wired. Examples of the input device include a keyboard, a mouse, and a microphone. Examples of the output device include a display device, a lighting device, and an audio output device that outputs audio. Examples of the input / output device include a touch panel display. Note that the input device, output device, and input / output device may be built into the computer 80 or may be external.

[0091] The hardware configuration of the computer 80 is an example. The computer 80 may have some of the components shown in FIG. 12 . The computer 80 may have components other than those shown in FIG. 12 . For example, the computer 80 may have a drive device or the like. The processor 801 may then read programs and data stored on a recording medium attached to the drive device or the like into the RAM 803. Examples of non-transitory tangible recording media include optical disks, flexible disks, magneto-optical disks, and USB (Universal Serial Bus) memories. As described above, the computer 80 may have input devices such as a keyboard and a mouse. The computer 80 may have an output device such as a display. The computer 80 may also have an input device, an output device, and an input / output device.

[0092] The computer 80 may also include various sensors (not shown). The types of sensors are not particularly limited. The computer 80 may also include an imaging device capable of capturing images or videos.

[0093] This concludes the description of the hardware configuration of each device. There are various variations in the method of realizing each device. For example, each device may be realized by any combination of a different computer and program for each component. Furthermore, multiple components of each device may be realized by any combination of a single computer and program.

[0094] Furthermore, some or all of the components of each device may be realized by circuits for specific applications. Furthermore, some or all of the components of each device may be realized by general-purpose circuits such as FPGAs (Field Programmable Gate Arrays). Furthermore, some or all of the components of each device may be realized by a combination of circuits for specific applications and general-purpose circuits. These circuits may be a single integrated circuit. Alternatively, these circuits may be divided into multiple integrated circuits. The multiple integrated circuits may be connected via a bus or the like.

[0095] Furthermore, when some or all of the components of each device are realized by a plurality of computers, circuits, etc., the plurality of computers, circuits, etc. may be centrally located or distributed.

[0096] The control methods described in the embodiments may be implemented by a computer such as the control devices 10 and 20 .

[0097] Each program, such as the formulation program and optimization program described in the embodiments, is recorded on a computer-readable recording medium such as an HDD, SSD, flexible disk, optical disk, magneto-optical disk, or USB memory. Each program is executed by being read from the recording medium by a computer. Each program may also be distributed via a communication network NT.

[0098] The functions of each of the components of the control devices 10 and 20 described above may be realized by dedicated hardware, such as a computer. Alternatively, each component may be realized by software. Alternatively, each component may be realized by a combination of hardware and software.

[0099] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. The configuration and details of the present disclosure may include embodiments to which various modifications that would be apparent to those skilled in the art are applied within the scope of the present disclosure. The present disclosure may include embodiments in which the details described herein are appropriately combined or substituted as necessary. For example, details described using a specific embodiment may also be applied to other embodiments to the extent that no contradiction occurs. For example, although multiple operations are described in sequence in the form of a flowchart, the order of description does not limit the order in which the multiple operations are performed. Therefore, when implementing the embodiments, the order of the multiple operations may be changed as long as it does not interfere with the content.

[0100] Some or all of the above-described embodiments can be described as follows: However, some or all of the above-described embodiments are not limited to the following.

[0101] (Supplementary Note 1) A control device comprising: a gaze detection means for detecting a customer's gaze at each product advertisement displayed on a display device installed in front of a store, using an image captured by an imaging device installed so as to capture an image of a customer looking at the display device installed in front of the store; a determination means for determining a product to recommend to the customer based on the detected gaze of the customer; an in-store customer detection means for detecting the customer within the store, using an image captured by an imaging device installed in the store; and an in-store output control means for outputting information about the determined product to a device within the store when the customer is detected within the store.

[0102] (Supplementary Note 2) The control device according to Supplementary Note 1, wherein the in-store device is a device that can be operated by the customer, and the in-store output control means outputs information about the determined product when the customer is detected so that the determined product can be ordered.

[0103] (Supplementary Note 3) The control device according to Supplementary Note 1 or Supplementary Note 2, wherein when the in-store device is a terminal device of a store clerk in the store, the in-store output control means displays an image of the customer together with information about the determined product on the terminal device of the store clerk.

[0104] (Supplementary Note 4) A control device as described in any of Supplementary Note 1 to Supplementary Note 3, further comprising a store front output control means for displaying an advertisement for the main product and advertisements for secondary products that can be used together with the main product on the display device installed in front of the store, wherein the determination means determines a product to recommend to the customer from the secondary products based on the customer's line of sight, and the in-store output control means, when the customer is detected in the store, displays an image combining the secondary product determined as the recommended product based on the customer's line of sight with the main product.

[0105] (Supplementary Note 5) The control device described in Supplementary Note 4, wherein the in-store device is a device that can be operated by the customer, and the in-store output control means displays the secondary product and the main product together with the combined image so that they can be ordered.

[0106] (Supplementary Note 6) The control device according to Supplementary Note 4 or Supplementary Note 5, wherein the in-store output control means displays a total price of the secondary product and the main product together with the combined image.

[0107] (Supplementary Note 7) The control device according to any one of Supplementary Note 1 to Supplementary Note 6, wherein the determining means determines the product to be recommended to the detected customer based on either a length or a number of times that the customer's gaze is directed at the product.

[0108] (Supplementary Note 8) The control device according to Supplementary Note 7, further comprising: a store front output control means for displaying an indicator relating to either the length or the number of times on the display device installed in front of the store.

[0109] (Supplementary Note 9) A control method for executing processing in which a computer: uses images captured by an imaging device installed in front of a store so as to be able to capture images of customers looking at a display device installed in front of the store to detect customers' gazes at each of product advertisements displayed on a display device installed in front of the store; determines products to recommend to the customers based on the detected gazes of the customers; detects the customers within the store using images captured by an imaging device installed in the store; and, when the customers are detected within the store, outputs information about the determined products to a device in the store.

[0110] (Supplementary Note 10) The control method described in Supplementary Note 9, wherein the device in the store is a device that can be operated by the customer, and the process of outputting to the device in the store outputs information about the determined product when the customer is detected so that the determined product can be ordered.

[0111] (Supplementary Note 11) When the device within the store is a terminal device of a store clerk within the store, the process of outputting to the device within the store includes displaying an image of the customer together with information about the determined product on the terminal device of the store clerk. This is the control method described in Supplementary Note 9 or Supplementary Note 10.

[0112] (Appendix 12) A control method described in any of Appendices 9 to 11, wherein the computer executes a process of displaying an advertisement for a main product and advertisements for secondary products that can be used together with the main product on the display device installed in front of the store, wherein in the determination process, a product to be recommended to the customer from the secondary products is determined based on the customer's line of sight, and in the output process to a device in the store, when the customer is detected in the store, an image combining the secondary product determined as the recommended product based on the customer's line of sight with the main product is displayed.

[0113] (Supplementary Note 13) The control method described in Supplementary Note 12, wherein the device in the store is a device that can be operated by the customer, and the process of outputting to the device in the store displays the secondary product and the main product together with the combined image so that they can be ordered.

[0114] (Supplementary Note 14) A control method according to any one of Supplementary Note 11 to Supplementary Note 13, wherein in the process of outputting to the device in the store, the total price of the secondary product and the price of the main product is displayed together with the combined image.

[0115] (Supplementary Note 15) The control method according to any one of Supplementary Note 9 to Supplementary Note 13, wherein the determining process determines the product to be recommended to the detected customer based on either the length or the number of times the customer's gaze is directed.

[0116] (Supplementary Note 16) The control method according to Supplementary Note 14, wherein the computer executes a process of displaying an indicator relating to either the length or the number of times on the display device installed in front of the store.

[0117] (Supplementary Note 17) A program that causes a computer to execute the following processes: detect the line of sight of customers toward each of the product advertisements displayed on a display device installed in front of a store, using images captured by an imaging device installed so as to capture images of customers looking at the display device installed in front of the store; determine products to recommend to the customers based on the detected line of sight of the customers; detect the customers within the store, using images captured by an imaging device installed within the store; and, when the customers are detected within the store, output information about the determined products to a device within the store.

[0118] (Appendix 18) The program described in Appendix 17, wherein the device in the store is a device that can be operated by the customer, and in the process of outputting to the device in the store, when the customer is detected, information about the determined product is output so that the determined product can be ordered.

[0119] (Supplementary Note 19) The program described in Supplementary Note 17 or Supplementary Note 18, wherein, when the in-store device is a terminal device of a store clerk in the store, the process of outputting to the in-store device displays an image of the customer on the terminal device of the store clerk along with information about the determined product.

[0120] (Appendix 20) A program described in any of Appendices 17 to 19, which causes the computer to execute a process of displaying an advertisement for a main product and advertisements for secondary products that can be used together with the main product on the display device installed in front of the store, wherein the determination process determines a product to be recommended to the customer from the secondary products based on the customer's line of sight, and the output process to a device in the store displays an image combining the main product with the secondary product determined as the recommended product based on the customer's line of sight when the customer is detected in the store.

[0121] (Appendix 21) The program described in Appendix 20, wherein the device in the store is a device that can be operated by the customer, and the process of outputting to the device in the store displays the secondary product and the main product together with the combined image so that they can be ordered.

[0122] (Supplementary Note 22) The program according to Supplementary Note 20 or Supplementary Note 21, wherein in the process of outputting to the device in the store, a total price of the secondary product and the main product is displayed together with the combined image.

[0123] (Supplementary Note 23) The program according to any one of Supplementary Note 17 to Supplementary Note 22, wherein the determining process determines a product to recommend to the detected customer based on either the length or the number of times the customer's gaze is directed.

[0124] (Supplementary Note 24) The program according to Supplementary Note 23, which causes the computer to execute a process of displaying an indicator relating to either the length or the number of times on the display device installed in front of the store.

[0125] (Supplementary Note 25) A non-transitory computer-readable recording medium having recorded thereon a program for causing a computer to execute the following processes: using images captured by an imaging device installed so as to be able to capture images of customers looking at a display device installed in front of a store, detect customers' gazes at each of product advertisements displayed on a display device installed in front of the store; determining products to recommend to the customers based on the detected customer gazes; detecting the customers within the store using images captured by an imaging device installed within the store; and, when the customers are detected within the store, outputting information on the determined products to a device within the store.

[0126] This application claims priority based on Japanese Patent Application No. 2023-126662, filed on August 3, 2023, the disclosure of which is incorporated herein in its entirety by reference.

[0127] 10, 20 Control device 21 Digital signage 22 Digital signage 23 Terminal device 24 Imaging device 80 Computer 101, 201 Gaze detection unit 102, 202 Determination unit 103, 203 In-store customer detection unit 104, 204 In-store output control unit 205 Front-of-store customer detection unit 206 Front-of-store output control unit 207 Order reception unit 801 Processor 802 ROM 803 RAM 804 Storage device 805 Communication interface 806 Input / output interface 807 Bus 2001 Product DB NT Communication network

Claims

1. A control device comprising: a gaze detection means for detecting the gaze of a customer at each of product advertisements displayed on a display device installed in front of a store, using images captured by an imaging device installed so as to capture an image of a customer looking at the display device installed in front of the store; a determination means for determining a product to recommend to the customer based on the detected gaze of the customer; an in-store customer detection means for detecting the customer within the store, using images captured by an imaging device installed in the store; and an in-store output control means for outputting information of the determined product to a device in the store when a customer is detected within the store.

2. The control device according to claim 1, wherein the in-store device is a device that can be operated by the customer, and the in-store output control means outputs information about the determined product when the customer is detected so that the determined product can be ordered.

3. A control device as described in claim 1 or 2, wherein when the in-store device is a terminal device of a sales clerk in the store, the in-store output control means displays an image of the customer together with information about the determined product on the sales clerk's terminal device.

4. A control device as described in any of claims 1 to 3, further comprising: a pre-store output control means for displaying an advertisement for a main product on the display device installed in front of the store and an advertisement for a secondary product that can be used together with the main product, wherein the determination means determines a product to be recommended to the customer from the secondary products based on the customer's line of sight, and the in-store output control means, when the customer is detected within the store, displays an image combining the secondary product determined as the recommended product based on the customer's line of sight with the main product.

5. The control device according to claim 4, wherein the in-store device is a device that can be operated by the customer, and the in-store output control means displays the secondary product and the main product together with the combined image so that they can be ordered.

6. The control device according to claim 4 or 5, wherein the in-store output control means displays a total amount of the price of the secondary product and the price of the main product together with the combined image.

7. The control device according to any one of claims 1 to 6, wherein the determining means determines the product to be recommended to the detected customer based on either the length or the number of times the customer's gaze is directed thereto.

8. The control device according to claim 7, further comprising: a front-of-store output control means for displaying an indicator relating to either said length or said number of times on said display device installed in front of said store.

9. A control method in which a computer executes the following processes: detects the customer's line of sight toward each product advertisement displayed on a display device installed in front of a store, using images captured by an imaging device installed in such a way that it can capture images of customers looking at the display device installed in front of the store; determines a product to recommend to the customer based on the detected line of sight of the customer; detects the customer within the store, using images captured by an imaging device installed in the store; and, when the customer is detected within the store, outputs information about the determined product to a device in the store.

10. The control method according to claim 9, wherein the in-store device is a device that can be operated by the customer, and the process of outputting to the in-store device outputs information about the determined product when the customer is detected so that the determined product can be ordered.

11. A control method as described in claim 9 or claim 10, wherein, when the in-store device is a terminal device of a sales clerk in the store, the process of outputting to the in-store device includes displaying an image of the customer on the sales clerk's terminal device together with information about the determined product.

12. A control method as described in any one of claims 9 to 11, wherein the computer executes a process of displaying an advertisement for a main product and an advertisement for a secondary product that can be used together with the main product on the display device installed in front of the store, and in the determination process, a product to be recommended to the customer from the secondary products is determined based on the customer's line of sight, and in the process of outputting to a device in the store, when the customer is detected in the store, an image combining the secondary product determined as the recommended product based on the customer's line of sight with the main product is displayed.

13. The control method according to claim 12, wherein the in-store device is a device that can be operated by the customer, and the process of outputting to the in-store device displays the secondary product and the main product together with the combined image so that they can be ordered.

14. The control method according to claim 12 or 13, wherein in the process of outputting to a device in the store, a total amount of the price of the secondary product and the price of the main product is displayed together with the combined image.

15. A control method according to any one of claims 9 to 13, wherein the determining process determines the product to be recommended to the detected customer based on either the length or the number of times the customer's gaze is directed.

16. The control method according to claim 15, wherein the computer executes a process of displaying an indicator relating to either the length or the number of times on the display device installed in front of the store.

17. A non-transitory computer-readable recording medium having recorded thereon a program for causing a computer to execute the following processes: using images captured by an imaging device installed in front of a store so as to capture images of customers looking at a display device installed in front of the store, detect a customer's line of sight toward each of product advertisements displayed on a display device installed in front of the store; determining a product to recommend to the customer based on the detected line of sight of the customer; detecting the customer within the store using images captured by an imaging device installed in the store; and when the customer is detected within the store, outputting information of the determined product to a device in the store.

18. The recording medium according to claim 17, wherein the in-store device is a device that can be operated by the customer, and the process of outputting to the in-store device outputs information about the determined product when the customer is detected so that the determined product can be ordered.

19. A recording medium as described in claim 17 or 18, wherein, when the in-store device is a terminal device of a sales clerk in the store, the process of outputting to the in-store device displays an image of the customer on the sales clerk's terminal device together with information about the determined product.

20. A recording medium as described in any of claims 17 to 19, which causes the computer to execute a process of displaying an advertisement for a main product displayed on the display device installed in front of the store and an advertisement for a secondary product that can be used together with the main product, wherein the determination process determines a product to be recommended to the customer from the secondary products based on the customer's line of sight, and the process of outputting to a device in the store displays an image combining the secondary product determined as the recommended product based on the customer's line of sight with the main product when the customer is detected in the store.

Images