Information providing system, information providing method, and program

Abstract

To provide an information providing system, an information providing method and a program which provide commodity information according to a visitor's intention to the visitor, based on declaration of the intention by finger pointing of the visitor.SOLUTION: An information providing system 1 comprises: a detection unit 101 which detects a person's finger pointing from an image acquired by an imaging device 20; an identifying unit 103 which identifies a commodity from a direction of the finger pointing; and a display unit 301 which displays commodity information of the identified commodity, when an identified elapsed time exceeds a predetermined time.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an information providing system, an information providing method, and a program.

Background Art

[0002] In a system for managing products in a store or products on display, a product presentation method and an information display method are disclosed for causing a customer approaching a product shelf to perform a product presentation targeted at the product picked up by the customer. (For example, see Patent Document 1)

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The product presentation method and the information display method described in Patent Document 1 disclose a technique for appealing the charm of a product by performing a product presentation for a customer who does not move from in front of the product shelf after picking up the product. In this case, no intention indication for requesting the provision of product information of the customer is acquired, and related information is displayed according to the action of picking up the product. The object of the present invention is to acquire an intention indication for requesting the provision of product information from a customer, and to provide product information conforming to the intention of the customer to the customer without touching the product.

Means for Solving the Problems

[0005] According to one aspect of the present invention, the system according to the present invention includes a detection means for detecting a finger pointing of a person from a captured image, a specifying means for specifying a product from the direction of the finger pointing, and a display means for displaying product information of the product specified by the specifying means on a display device.

[0006] According to one aspect of the present invention, the method according to the present invention detects a person's pointing gesture from a captured image, identifies a product from the direction of the pointing gesture, and displays product information of the product on a display device.

[0007] According to one aspect of the present invention, the program according to the present invention causes the imaging device to execute a detection process for detecting a person's pointing gesture, an identification process for identifying a product from the direction of the pointing gesture, and a display process for displaying product information of the product identified by the identification means on a display device.

Effects of the Invention

[0008] According to the present invention, it is possible to provide an information providing system, an information providing method, and a program that acquire an intention indication for requesting the provision of product information of a customer without the customer touching the product and provide the product information to the customer.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Mode for Carrying Out the Invention

[0010] First, to facilitate the understanding of the embodiments of the present invention, the background of the present invention will be described. There may be a case where it is desired to provide product information while considering the prevention of the spread of infectious diseases. For example, when a product is touched with a hand to which a virus serving as an infection source of an infectious disease adheres, it is assumed that the virus adheres to the product. If a customer who has touched a product returns the product to the product shelf without purchasing it, the virus adhering to the product may adhere to the hand of another customer who then touches the product, and enter the mucous membranes such as the mouth, nose, and eyes through that hand, potentially spreading the infection.

[0011] On the one hand, there may be a case where a person wants to check product information before purchasing a product. The product information includes, for example, information about specific raw materials contained in the product, calorie information, or expiration date information. The above-mentioned information is often described in small letters on the side of the product package or the like, and it is necessary to pick up the product to check the product information.

[0012] According to the embodiment of the present invention described below, it is possible to provide the store visitor with the product information without the store visitor approaching or touching the product extremely closely.

[0013] Hereinafter, exemplary embodiments of the present invention will be described with reference to the drawings. In the drawings, the same or corresponding elements are denoted by the same reference numerals, and the description thereof may be omitted or simplified.

[0014] [Overview of Functions] The overview of the functions realized by the present invention will be described with reference to FIG. 1. FIG. 1 is a diagram showing an example of a situation where a store visitor acquires information from an information providing system. As shown in FIG. 1, the information providing system according to the present embodiment includes a computer 10, an imaging device 20, and a display device 30.

[0015] The computer 10 detects the pointing of the store visitor acquired by the imaging device 20, and identifies the product for which the store visitor requests information by calculating the direction of the pointing. The computer 10 displays the information related to the identified product on the display device 30. The pointing finger does not have to be a specific finger, as long as the direction for identifying the product is clear. Also, gestures other than pointing may be used.

[0016] The imaging device 20 is installed at a predetermined position or the like, and is a camera or the like that photographs customers in the store or product shelves. The imaging device 20 may be a camera with a fixed orientation and installation location, or a camera with a changeable orientation such as a PTZ (Pan Tilt ZOOM) camera, or a movable camera mounted on a moving body. Further, for example, it may be a camera mounted on a wearable terminal such as a smartphone or a tablet. A plurality of dedicated cameras for photographing the pointing of a person in the store and dedicated cameras for photographing product shelves may be installed separately according to their uses. The imaging device 20 and the computer 10 are communicably connected via an arbitrary network. Note that the computer 10 and the imaging device 20 may be integrated into one device.

[0017] The display device 30 outputs product information specified from the detected direction of the pointing. For example, the display device may be a tablet installed on a product shelf where a customer can easily view it, or a sign installed near the product shelf. The display device 30 and the computer 10 are communicably connected via an arbitrary network. Note that the computer 10, the imaging device 20, and the display device 30 may be integrated into one device.

[0018] As described above, in this embodiment, by detecting the pointing, which is the intention display of the customer, and acquiring the direction of the pointing, the product can be specified, and product information can be provided in accordance with the intention of the customer without the customer touching the product. [First Embodiment]

[0019] A usage example and an information providing method of the information providing system 1 in this embodiment will be described with reference to FIG. 1. The information providing system 1 includes a computer 10, an imaging device 20, and a display device 30. The computer 10 and the imaging device 20 can communicate with each other via a wired or wireless communication unit provided in each of them. Similarly, the computer 10 and the display device 30 can communicate with each other via a wired or wireless communication unit provided in each of them. The number of the computer 10, the imaging device 20, and the display device 30 is at least one or more for each, and it is also possible to connect and install a plurality of them simultaneously.

[0020] Next, the functional configurations of the computer 10, the imaging device 20, and the display device 30 will be described with reference to FIG. 2. The computer 10 includes a detection unit 101, a direction calculation unit 102, an identification unit 103, an information management unit 104, an elapsed time measurement unit 105, a display processing unit 106, and a learning unit 107. The imaging device 20 includes a video acquisition unit 201, and the display device 30 includes a display unit 301 and a video acquisition unit 302. The detection unit 101 is responsible for detection means for detecting a person's pointing from the image acquired from the imaging device 20. The direction calculation unit 102, the identification unit 103, the information management unit 104, the elapsed time measurement unit 105, and the learning unit 107 are responsible for identification means for identifying a product from the direction of the pointing. The display processing unit 106 is responsible for display means for displaying product information of the identified product on the display device 30.

[0021] The detection unit 101 automatically identifies and detects a pointing or other hand gesture from the video acquired from the video acquisition unit 201. This identification of the pointing is specifically performed by any one of various known methods or a combination thereof. For example, machine learning-based image analysis can be mentioned. In the video analysis of machine learning, the pointing in the video image is automatically and efficiently identified by an image recognition technique using deep learning. The location where the detection unit 101 detects the pointing may be within the image acquired by the video acquisition unit 201, and is not limited to in front of the product shelf. Even if a person points behind another customer, it can be detected. Also, since it is only necessary to detect the pointing of the customer, it is not necessary for a part of the body (such as the head or torso) other than the pointing to be photographed in the image.

[0022] The direction calculation unit 102 serves as a direction calculation means for calculating the direction of the finger indicated by the pointing detected by the detection unit 101. For example, the joint points of the finger of the pointing detected by the detection unit 101 are extracted using a joint estimation technique (skeleton estimation technique) such as OpenPose that utilizes machine learning. The direction calculation unit 102 generates a straight line connecting the first joint point and the second joint point of the extracted finger, and calculates the direction of the pointing towards the product by extending the straight line in the direction away from the body. The length of the straight line generated by the direction calculation unit 102 may be a variable value according to the positions of the customer and the product shelf, or may be set to a fixed value. Also, even when there is another customer in front of the product shelf and the customer who made the pointing is not at the position in front of the product shelf closest to the product shelf, the straight line is still generated. The generated straight line is extended in the direction away from the body of the customer who requested the information provision regardless of the presence or absence of the other customer, and the direction calculation unit 102 calculates the direction of the pointing towards the product.

[0023] The specifying unit 103 extracts the object area of the product from the acquired video by image processing, and when the straight line generated by the direction calculation unit 102 intersects the object area, identifies the product within the object area where the straight line intersects using image recognition. The object area of the product is a processing area obtained by cutting out the area of the object from the image to facilitate image recognition. The product is identified by performing image recognition on the image of the product appearance within the extracted object area using the product appearance data managed by the information management unit 104. When the intersection spans the object areas of multiple products and the product cannot be identified as one, two products may be identified as the products for which information is provided. Also, the object area may be a rectangle surrounding the product, a circle surrounding the product, or the outer shape of the product.

[0024] The information management unit 104 is a database for storing the product information of the purchased products. Information such as the external appearance photo of the product, product name, product type, specific raw materials, expiration date, price, calorie, etc. is stored as the product information. Also, the product information is not limited to the above.

[0025] The elapsed time measurement unit 105 counts the elapsed time during which the straight line generated by the direction calculation unit 102 intersects the object area of the product. When the intersection between the straight line and the object area of the product is lost, the elapsed time measurement unit 105 stops the counting.

[0026] When a predetermined time has elapsed since the elapsed time measurement unit 105 started counting, the display processing unit 106 displays the product information specified by the specifying unit 103 on the display unit 301. When a plurality of display devices 30 are installed on the product shelf, the display device 30 corresponding to the position and height of the customer in the store is selected, and the product information is displayed. When there is a large amount of information to be displayed, such as an exterior photo of the product, product name, product type, specified raw materials, expiration date, price, calories, etc., the display processing unit 106 may switch and display the display content at predetermined time intervals.

[0027] The learning unit 107 learns finger pointing or other hand gestures. In addition to learning finger pointing, which is the intention display of a customer who requests acquisition of product information, the learning unit 107 may also learn the action of waving a hand, which is an intention display to cancel the display, when product information different from the intended product is displayed. Further, when the specifying unit 103 specifies two products as candidates, the exterior and name of those two products may be displayed on the display unit 301, and gestures for the customer to select the product for which information is desired may be learned.

[0028] The video acquisition unit 302 is provided in the display device 30 and acquires gestures, which are the intention displays of customers.

[0029] FIG. 3 is a diagram for explaining a usage example of the information providing system in the present embodiment. The imaging device 20 is installed at a position where it can photograph the product shelf. In FIG. 3, it is installed near the ceiling where the product shelf can be photographed, but the installation position may be directly installing a plurality of units on the upper part of the product shelf. Further, the display device 30 is installed at a position where it is easy for customers to see and where it is easy to detect the gestures of customers. Assuming that a customer of low height will use it, it may be installed at a low position on the product shelf.

[0030] FIG. 4 is a flowchart of an information providing system showing the processing flow of the present embodiment from when the video acquisition unit 201 acquires a video until a product is specified according to a pointing gesture which is an intention display of a customer and is displayed on the display unit 301. First, the imaging device 20 installed in the store acquires an image near the product shelf (S101). The detection unit 101 detects the pointing gesture of the customer from the acquired image (S102). The direction calculation unit 102 extracts the first joint point and the second joint point of the finger of the pointing gesture detected by the detection unit 101 (S103). Further, the direction calculation unit 102 generates a straight line connecting the extracted first joint point and second joint point, and calculates the direction toward the product by extending the straight line in a direction away from the body (S104).

[0031] The specifying unit 103 extracts the object area of the product by image processing from the acquired image, and when the straight line generated by the direction calculation unit 102 intersects the object area, specifies the product using image recognition (S105). The elapsed time measurement unit 105 counts the elapsed time during which the straight line generated by the direction calculation unit 102 intersects the object area of the product (S106). The display processing unit 106 displays the product information specified by the specifying unit 103 on the display unit 301 when a predetermined time has elapsed since the elapsed time measurement unit 105 started counting (S107). In the above flowchart, after specifying the product where the straight line intersects the object area of the product, the elapsed time during which the straight line intersects the object area of the product is counted. Conversely, the product may be specified when the elapsed intersection time exceeds a predetermined time.

[0032] FIG. 5 is an example of a pointing finger image of a person learning in the learning unit 107. The pointing fingers photographed from various directions are learned. For example, in FIG. 5, they are images (P1 to P3) of the pointing finger photographed from the Y-axis direction. P1 is an image photographed from an angle where the palm faces the surface, and P3 is an image photographed from an angle where the back of the hand faces the front. P2 is photographed at an angle between P1 and P3 with the thumb side facing the front. When the angle of the pointing finger of P1 is defined as 0° on the XZ plane, P3 is 180°, and P2 is 90°. The learning unit learns the pointing finger for all angles from 0° to 360°. Also, not only the photographing from the Y-axis direction, but also the photographing from the X-axis direction and the Z-axis direction similarly learn the pointing finger for all angles from 0° to 360°. The detection unit 101 detects the pointing finger from the image acquired by the video acquisition unit 201 using the pointing finger learned by the learning unit 107.

[0033] FIG. 6 is a diagram showing an example in which the detection unit 101 detects the pointing finger and the direction calculation unit 102 calculates the direction of the pointing finger. The detection unit 101 detects the pointing finger and extracts the object area P4 of the pointing finger. The direction calculation unit 102 extracts the joint points of the pointing finger in the detected object area P4 using a joint estimation technique such as Open Pose, generates a straight line connecting the first joint and the second joint point of the finger, and further extends the straight line to the product side. The specifying unit 103 specifies the product at the intersection of the straight line and the object area of the product.

[0034] The specifying unit 103 identifies the product appearance P5 from the image acquired from the video acquisition unit 201 and specifies the product by comparing it with the product appearance stored in the database of the information management unit 104. Also, the specifying unit 103 sets the area surrounding the specified product appearance as the object area P6 of the product. The size of the area may be incorporated into the program so as to be defined by the size of the product, or may be designed to be directly inputtable to the system.

[0035] FIG. 7 is a flowchart showing an example of processing when the elapsed time measurement unit 105 starts counting. First, the detection unit 101 detects a pointing (S201). Next, the identification unit 103 identifies the product (S202). The elapsed time measurement unit 105 counts the time during which the straight line generated by the direction calculation unit 102 intersects the object area of the product after the product is identified (S203). When the elapsed time exceeds a predetermined time, product information is displayed on the display unit 301 (S204; YES, S205). On the other hand, if the straight line moves out of the object area of the product before the elapsed time reaches the predetermined time, the state returns to the state before the pointing was detected (S204; NO).

[0036] FIG. 8 is an example of display when the product information of the identified product is displayed on the display unit 301 of the display device 30. The display unit 301 displays an exterior photo of the product when viewed from above or horizontally, the product name, the product type, the specific raw materials, the expiration date, the price, the calories, and the like. Furthermore, discount information, campaign information, etc. when purchased together with other products may be displayed. When the amount of information to be displayed is large, the display content may be switched and displayed at predetermined time intervals. Also, not only display but also product information may be provided by voice.

[0037] As described above, according to the information providing system of the present embodiment, it is possible to obtain the intention indication of the customer and provide the product information requested by the customer without the customer touching the product. Also, for example, even if there is another customer in front of the product shelf and the customer requesting the product information is not at the position in front of the product shelf closest to the product shelf, the product information can be displayed on the display unit 301 by pointing.

[0038] [Second Embodiment] Next, another example applicable to the first embodiment will be described with reference to FIGS. 9 and 10. In the first embodiment, a method for identifying a product whose straight line generated by the direction calculation unit 102 intersects with the object region of one product was described. In the second embodiment, a method for identifying a product when the straight line intersects with a plurality of object regions having different product names will be described in detail. As shown in FIG. 9, when the straight line generated by the direction calculation unit 102 intersects with the object regions of two products, product M1 and product M2, the identification unit 103 identifies the two products and displays them on the display unit 301.

[0039] FIG. 10 is a display example when two products with different product names are displayed on the display unit 301. In the display, information such as the identified product image and product name, and on top of them, product identification numbers indicating each product are displayed. Further, the identification unit 103 displays a statement instructing to select one of the two products identified based on the pointing of the customer. Also, this statement and the identification number may be displayed on the display unit 301 as described above, or may be notified by voice.

[0040] The display device 30 is provided with a video acquisition unit 302. A customer who requests information by pointing shows a gesture indicating the product identification number toward the video acquisition unit 302 to select the product for which information is desired. The gesture indicating the product identification number may be expressed, for example, by raising one finger if it is 1, and raising two fingers if it is 2. The meaning of the expression is defined by the learning unit 107. The detection unit 101 detects a pointing from the image acquired by the video acquisition unit 201 of the imaging device 20, and detects a gesture indicating the product identification number from the image acquired by the video acquisition unit 302 of the display device 30. The display unit 301 displays product information corresponding to the product identification number indicated by the customer's gesture. In FIG. 10, the case where the straight line intersects with two different products, product M1 and product M2, has been described. However, when the object regions of the two products intersecting the straight line are products with the same product name, a display screen instructing the selection as shown in FIG. 10 is not displayed, and product information is provided.

[0041] As described above, even when a straight line intersects the object regions of a plurality of products depending on the shooting position of the imaging device 20, a customer can select one product for which information is desired to be provided, and information can be provided appropriately according to the customer's intention.

[0042] [Third Embodiment] Next, another example applicable to the first embodiment described above will be described with reference to FIGS. 11 and 12. In the first embodiment, the imaging device 20 is installed near the ceiling where the product shelves can be photographed to detect a customer's pointing gesture, identify a product based on the direction of the pointing gesture, and display it on the display unit 301. In the third embodiment, a method of selecting a display device 30 that is easy for a customer to view from among a plurality of display devices 30 when displaying on the display unit 301 will be described.

[0043] In the first embodiment, an image was acquired from the imaging device 20 installed near the ceiling where it is easy to detect a pointing gesture in order to detect the customer's pointing gesture. In this embodiment, an image is also acquired from an imaging device 21 (not shown) installed near the front of the product shelves. FIG. 11 is an example of an image captured by the imaging device 21. A customer and the display device 30 are detected from the image captured by the imaging device 21.

[0044] FIG. 12 is a flowchart showing the processing flow of this embodiment from when the imaging device 20 and the imaging device 21 acquire an image until a product is identified according to the pointing gesture, which is the customer's intention display, and is displayed on the display unit 301. First, the imaging device 20 installed near the ceiling acquires an image of the area near the product shelves, and the imaging device 21 acquires an image of the product shelves from the front (S301).

[0045] The detection unit 101 detects a pointing gesture and a full-body image of the customer from the image acquired by the imaging device 20 (S302). Further, the detection unit 101 detects the display device 30 and the full-body image of the customer from the image acquired by the imaging device 21 (S303). The display device 30 and the full-body image have been pre-learned by the learning unit 107, and the full body and the display device 30 are detected using image recognition. The specifying unit 103 specifies the product appearance in the object area where the straight line intersects by generating a straight line from the direction of the pointing gesture and performing image recognition on the product appearance in the object area where the straight line intersects, in the same manner as in the first embodiment (S304).

[0046] The display processing unit 106 identifies the customer who made the pointing gesture from the image acquired by the imaging device 21 by comparing the full-body image of the customer who made the pointing gesture detected from the image acquired by the imaging device 20 with the full-body image of the customer detected from the image acquired by the imaging device 21 (S305). Based on the position of the identified customer and the installation position of the detected display device 30, the display processing unit 106 selects the display device 30 that displays the identified product information from among the display devices 30 acquired based on the image acquired by the imaging device 21 (S306). The display processing unit 106 displays the product information of the identified product on the display unit 301 of the display device 30 selected from among the plurality of display devices 30 (S307).

[0047] For example, as a means for selecting the display device 30 that displays the product information from among the plurality of display devices 30, the display processing unit 106 selects the one with the shortest straight-line distance from the position of the customer to the installation position of the display device 30. Also, in the case of a customer with a low height, the display device 30 installed at a low position may be selected. When the display device 30 is providing product information in response to the pointing gesture of another customer, the product information may be displayed on the display device 30 at the second-closest straight-line distance from the position of the customer.

[0048] In this embodiment, the imaging device 20 and the imaging device 21 are used. However, if an image is captured over a wide area and the positions of the plurality of display devices 30 and the customer can be recognized by a single imaging device, it may be implemented using only the imaging device 20. In this case, it is not always necessary to detect the entire body of the customer. By detecting the pointing gesture and the display device 30, an appropriate display device is selected from the plurality of display devices 30 based on the positions of the display device 30 and the customer's pointing gesture.

[0049] As described above, the customer can check the product information from the display device 30 close to the position where the customer is located. In addition, by selecting the display device 30 according to the height of the customer, it becomes easier for the customer to check the display device 30.

[0050] [Fourth Embodiment] Next, another embodiment applicable to the first embodiment described above will be described with reference to FIGS. 13 and 14. In the first embodiment, it is assumed that a customer requests product information in the store, and the product information specified based on the customer's pointing gesture is displayed on the display unit 301. In the fourth embodiment, a method of displaying the storage information of the article in the cardboard box specified based on the pointing gesture of a person on the display unit 301 when the article is stored in the warehouse in a state where the inside is not visible, such as in a cardboard box, will be described.

[0051] FIG. 13 is a diagram for showing an example for detecting a pointing gesture from an image acquired by the imaging device 20 that captures the vicinity of a warehouse shelf installed in the warehouse, specifying the cardboard box whose inside is to be confirmed from the direction of the pointing gesture, and displaying the article information on the display unit 301. A cardboard box ID for managing the cardboard box is marked on the side surface of the cardboard box that can be confirmed by the imaging device 20.

[0052] The detection unit 101 detects a pointing gesture from the image acquired by the imaging device 20 in the same manner as in the first embodiment. Also, the direction calculation unit 102 calculates the direction of the pointing gesture. In the information management unit 104, the article information stored in the cardboard corresponding to the cardboard ID in advance is registered using an input device such as a mouse. The article information includes a photo inside the cardboard box, the article name, the model name, the quantity, the expiration date, and the like. The identification unit 103 extracts the object region P7 of the cardboard by image processing from the image acquired by the video acquisition unit 201, and when the straight line generated by the direction calculation unit 102 intersects the object region P7, recognizes the cardboard ID using image recognition. The identification unit 103 acquires the article information in the cardboard corresponding to the recognized cardboard ID from the information management unit 104. The display processing unit 106 displays the article information in the cardboard ID identified by the identification unit 103 on the display unit 301.

[0053] Figure 14 is a flowchart showing the processing flow of this embodiment from when the imaging device 20 acquires a video until the cardboard ID is recognized according to the pointing gesture which is the administrator's intention indication, and the storage information of the article corresponding to the cardboard ID is displayed on the display unit 301. First, the imaging device 20 installed near the ceiling acquires the video near the storage shelf (S401). From the acquired video, the detection unit 101 detects a person's pointing gesture (S402). The direction calculation unit 102 extracts the first joint point and the second joint point of the finger of the pointing gesture detected by the detection unit 101 (S403). Also, the direction calculation unit 102 generates a straight line connecting the extracted first joint point and second joint point, and calculates the direction toward the cardboard by extending the straight line in the direction away from the body (S404).

[0054] The specific unit 103 extracts the cardboard object area from the video acquired by the video acquisition unit 201 through image processing, and when the straight line generated by the direction calculation unit 102 intersects with the object area P7 of the cardboard, it recognizes the cardboard ID using image recognition (S405). The specific unit 103 acquires the article information inside the cardboard corresponding to the recognized cardboard ID from the information management unit 104. The elapsed time measurement unit 105 counts the elapsed time during which the straight line generated by the direction calculation unit 102 intersects with the object area P7 of the cardboard (S406). When a predetermined time has elapsed since the elapsed time measurement unit 105 started counting, the display processing unit 106 displays the article storage information of the cardboard ID specified by the specific unit 103 on the display unit 301 (S407).

[0055] As described above, according to this embodiment, even when articles are stored in an invisible cardboard or the like, it is possible to efficiently check the article information without having to lower the cardboard from the storage shelf and open it.

[0056] [Modification Example] In the above-described first to fourth embodiments, when identifying a product, the product is identified by performing image recognition on the image of the product appearance acquired by the imaging device 20 and the product appearance managed by the information management unit 104. In this modification example, a method for identifying a product without using image recognition will be described with reference to FIGS. 15 to 17. The processing from detecting a pointing gesture in the image acquired by the imaging device 20 to generating a straight line from the joint points is the same as that in the first embodiment (FIG. 4; S101 to S104).

[0057] FIG. 15 is an example of a processing screen in which grid lines indicating the X-axis and Y-axis are superimposed on the image acquired by the imaging device 20. FIG. 16 is an example of a database screen of the information management unit 104 for storing product information of purchased products. For example, it is assumed that on the C shelf in FIG. 15, pasta C is displayed on the top shelf, and soup C is displayed on the second shelf from the top. When the store manager wants to specify the position on the screen of soup C, pressing the unspecified button P8 in FIG. 16 using an input device such as a mouse enables the area P9 to be specified on the processing screen in FIG. 15. The information management unit 104 recognizes the coordinates specified by the manager and registers the coordinates in the database. Thereby, the system recognizes the position of soup C on the screen coordinates, and sets the area P9 as an object area in the same manner as in other embodiments.

[0058] The specifying unit 103 acquires, from the database of the information management unit 104, product information at the intersection of the straight line generated by the direction calculation unit 102 and the object area, and displays it on the display unit 301.

[0059] FIG. 17 is a diagram for explaining the positioning of the captured image of the imaging device 20. Alignment marks P10 are installed on the product shelf at locations where the imaging device 20 can easily recognize them. The imaging device 20 automatically controls the pan and tilt of the camera so that the alignment marks appear at predetermined coordinates on the screen coordinates. Thereby, even when the imaging device 20 is not fixed, the position of the product can be grasped, and the product information can be managed in the database.

[0060] As described above, it becomes possible to identify products without using image recognition.

[0061] [Fifth Embodiment] The minimum configuration of the information providing system 4 in the present invention will be described with reference to FIG. 18. FIG. 15 is a diagram showing an overall configuration example of the information providing system 4 in the present embodiment. The information providing system 4 includes a detection unit 401, a direction calculation unit 402, a specifying unit 403, and a display unit 404.

[0062] The detection unit 401 detects the pointing of a person near the product shelves in the store using the imaging device 20. The direction calculation unit 402 calculates the direction of the pointing of the customer detected by the detection unit 401. The identification unit 403 identifies the product from the direction of the pointing calculated by the direction calculation unit 402. The display unit 404 displays the information of the product identified by the identification unit 403.

[0063] Next, the processing flow according to the minimum configuration of the information providing system 4 will be described with reference to FIG. 19. The detection unit 401 detects the pointing of a customer near the product shelves in the store using the imaging device 20 (S501). The direction calculation unit 402 calculates the direction of the pointing of the person detected by the detection unit 401 (S502). The identification unit 403 identifies the product from the direction of the pointing calculated by the direction calculation unit 402 (S503). The display unit 404 displays the information of the product identified by the identification unit 403 (S504).

[0064] As described above, the customer can obtain product information without touching the product by making a gesture indicating that they want the product information to be provided by the information providing system using the computer 10.

[0065] [Hardware Configuration Example] Next, an example of the hardware configuration for realizing the computer 10, the imaging device 20, the display device 30, and the information providing system (1, 4) in each of the above-described embodiments will be described. Each functional unit (detection unit 401, direction calculation unit 402, identification unit 403, display unit 404, etc.) included in the computer 10, the imaging device 20, the display device 30, and the information providing system (1, 4) is realized by an arbitrary combination of hardware and software centered around at least one CPU (Central Processing Unit) of an arbitrary computer, at least one memory, a program loaded into the memory, a storage unit such as at least one hard disk for storing the program, a network connection interface, and the like. It is understood by those skilled in the art that there are various modifications to this realization method and apparatus. Note that the storage unit can store, in addition to the programs stored before the shipment of the device, programs downloaded from storage media such as optical disks, magneto-optical disks, semiconductor flash memories, and servers on the Internet.

[0066] The processor (1A) is an arithmetic processing device such as a CPU, a GPU (Graphics Processing Unit), or a microprocessor, and executes various programs and controls each part. That is, the processor (1A) reads a program from the ROM (2A) and executes the program using the RAM (3A) as a work area. In the above embodiment, the execution program is stored in the ROM (2A).

[0067] The ROM (2A) stores an execution program for causing the processor (1A) to execute a detection process for detecting a finger pointing from a captured image, an identification process for identifying a product from the direction of the finger pointing, and a display process for displaying product information of the product identified by the identification means on a display device, as well as data related to the product information and learning information of the finger pointing. The RAM (3A) temporarily stores a program or data as a work area.

[0068] The communication module (4A) enables the computer 10 to communicate with the imaging device 20 and the display device 30. When multiple computers 10 are installed, it also enables communication between the computers.

[0069] The display (5A) functions as a display unit, accepts input of requests from the user via a touch panel, mouse, etc., displays responses from the information providing system (1, 4), and also has the function of displaying product information.

[0070] The I / O (6A) includes interfaces for acquiring information from input devices, external devices, external storage units, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external storage units, etc. Input devices include, for example, touch panels, keyboards, mice, microphones, cameras, etc. Output devices include, for example, displays, speakers, printers, lamps, etc.

[0071] Note that the configurations of the foregoing embodiments may be combined or some of the component parts may be interchanged. Also, the configuration of the present invention is not limited to only the foregoing embodiments, and various changes may be made without departing from the gist of the present invention.

Explanation of Reference Numerals

[0072] 1 Information providing system 10 Computer 20 Imaging device 30 Display device 101 Detection unit 102 Direction calculation unit 103 Identification unit 104 Information management unit 105 Elapsed time measurement unit 106 Display processing unit 107 Learning unit 201 Video acquisition unit 301 Display unit 302 Video acquisition unit 4 Information providing system 401 Detection unit 402 Direction calculation unit 403 Identification unit 404 Display unit M1, M2 Commodities P1~P3 Pointing P4, P6, P7 Object areas P5 Commodity appearance P8 Unspecified button P9 Area 1A Processor 2A ROM 3A RAM 4A Communication module 5A Display 6A I / O 7A Bus

Claims

From a first image captured by an imaging device, a pointing gesture of a person and a first full-body image of the person making the pointing gesture, which is a full-body image of the pointing person, are detected. From a second image captured by another imaging device, a display device for displaying product information of a product and a second full-body image of the person imaged in the second image, which is a full-body image of the person, are detected. Specific means for specifying a product existing in the direction of the pointing gesture based on the first image. Person identification means for collating the first full-body image and the second full-body image and identifying the second full-body image of the pointing person from the second image. Selection means for selecting a display device for displaying product information of the product specified by the specific means from the display device using the position of the pointing person obtained from the second image and the installation position of the display device. Display means for displaying product information of the product specified by the specific means on the selected display device. Comprising An information providing system.

2. The detection means detects the pointing gesture by learning the pointing gesture. The information providing system according to claim 1.

3. Including direction calculation means for calculating the direction of the pointing gesture by generating a straight line from the pointing gesture. The information providing system according to claim 1 or 2.

4. The specific means extracts an object area from the image and specifies a product where the straight line intersects the extracted object area. The information providing system according to claim 3.

5. The display means selects a display device for displaying product information of the product specified from among a plurality of the display devices based on the position of the customer who made the pointing gesture. The information providing system according to any one of claims 1 to 4. From a first image captured by an imaging device, a pointing gesture of a person and a first full-body image of the person making the pointing gesture, which is a full-body image of the pointing person, are detected. From a second image captured by another imaging device, a display device for displaying product information of a product and a second full-body image of the person imaged in the second image, which is a full-body image of the person, are detected. Based on the first image, a product existing in the direction of the pointing gesture is specified. The first full-body image and the second full-body image are collated, and the second full-body image of the pointing person is identified from the second image. Select a display device that displays product information of a product specified from the display device, using the position of the pointing person obtained from the second image and the installation position of the display device. Display the product information of the specified product on the selected display device. Information providing method.

7. In a computer processor Detect, from a first image captured by one imaging device, a person's pointing and a first whole-body image of the pointing person who is the person making the pointing, and from a second image captured by another imaging device, a display device that displays product information of a product and a second whole-body image of the person captured in the second image; a detection process; A specifying process of specifying a product existing in the direction of the pointing based on the first image; A person specifying process of collating the first whole-body image and the second whole-body image and specifying the second whole-body image of the pointing person from the second image; A selection process of selecting a display device that displays product information of the product specified by the specifying process from the display device, using the position of the pointing person obtained from the second image and the installation position of the display device; A display process of displaying the product information of the product specified by the specifying process on the selected display device; A program for causing the above to be executed.

Images