Information output control device and information output control method

Abstract

An arrangement unit according to the present invention arranges, on the basis of a captured image of a facility in which a plurality of objects are displayed, a behavior image relating to the behavior of at least one person positioned in the facility on a three-dimensional facility model in which the facility is reproduced. An identification unit identifies, on the basis of the three-dimensional facility model and the behavior image, an object of interest in which the at least one person is interested. An output control unit outputs information relating to the object of interest identified by the identification unit from an information output device installed in the facility.

Description

Information output control device and information output control method

[0001] The present invention relates to an information output control device and an information output control method.

[0002] Conventionally, technologies have been known in which monitors are installed in stores to display advertisements and other advertising materials promoting sales of products in the store. For example, a sales support system described in Patent Document 1 below comprehensively analyzes consumers' purchasing intentions by grasping users' in-store psychology and in-store behavior, and provides purchasing support via product display shelves. The purchasing support system includes a product display shelf system and a management device. The product display shelf system includes a sensing unit including a first sensor device that captures facial images of consumers positioned in front of the product display shelf and a second sensor device that captures the consumers' movements relative to the products on the product display shelf, and an output unit that displays predetermined content. The management device receives each sensor detection information acquired by the sensing unit, determines the consumers' purchasing intentions, and performs a distribution process to display content corresponding to the consumers positioned in front of the shelf on the output unit. The management device identifies in-store psychology patterns based on the consumers' facial expressions and gazes detected from the facial images, identifies in-store behavior patterns based on the consumers' movements relative to the products, and determines the consumers' purchasing intention patterns corresponding to each of these patterns. For example, if a consumer is looking at product B, information about product B is highlighted. Furthermore, when a consumer picks up product A, an image relating to product A is displayed.

[0003] Patent No. 6810561

[0004] In the above-mentioned Patent Document 1, a camera, a motion sensor, and a capacitance sensor are installed on each shelf (each monitor to be controlled) in order to track user behavior, which results in a costly capital investment and a heavy burden on the store.

[0005] An object of the present invention is to output information about an object in which a user is interested in a simple format.

[0006] An information output control device according to one aspect of the present invention includes an arrangement unit that arranges, based on an image of a facility where a plurality of objects are displayed, a behavior image relating to the behavior of at least one person located within the facility on a three-dimensional facility model that reproduces the facility; an identification unit that identifies an object of interest that is of interest to the at least one person based on the three-dimensional facility model and the behavior image; and an output control unit that outputs information relating to the object of interest identified by the identification unit from an information output device installed in the facility.

[0007] An information output control method according to one aspect of the present invention places, based on an image of a facility where a plurality of objects are displayed, a behavior image relating to the behavior of at least one person located within the facility on a three-dimensional facility model that reproduces the facility, identifies an object of interest that is of interest to the at least one person based on the three-dimensional facility model and the behavior image, and outputs information relating to the object of interest from an information output device installed in the facility.

[0008] According to one aspect of the present invention, information about an object in which a user is interested can be output with a simple configuration.

[0009] FIG. 1 is a block diagram showing the configuration of a system 1 according to an embodiment. FIG. 2 is a schematic diagram showing the appearance of an information output device 30. FIG. 3 is a block diagram showing the configuration of an information output control device 10. FIG. 4 is a schematic diagram showing an example of a captured image PS. FIG. 5 is a schematic diagram showing an example of a three-dimensional store model MT in which a skeletal model BM is arranged. FIG. 6 is a diagram showing the position of a person J in real space. FIG. 7 is a schematic diagram showing an example of an output mode of information by the information output device 30. FIG. 8 is a schematic diagram showing another example of an output mode of information by the information output device 30. FIG. 9 is a flowchart showing the operation of the information output control device 10.

[0010] A. Embodiment Fig. 1 is a block diagram showing the configuration of a system 1 according to an embodiment. The system 1 includes an information output control device 10, an imaging device 20, and an information output device 30. The information output control device 10 controls the output of information in the information output device 30. The information output device 30 includes a monitor 30A and a speaker 30B, and outputs at least one of an image or sound. The information output device 30 may include only either the monitor 30A or the speaker 30B, and output only either an image or sound.

[0011] FIG. 2 is a schematic diagram showing the appearance of the information output device 30. The information output device 30 is installed in a facility where multiple objects are displayed. In this embodiment, the facility where the information output device 30 is installed is a store RT (e.g., a convenience store, see FIG. 4 ), and the multiple objects are products SH displayed in the store RT. In the example of FIG. 2 , each product SH is identified by, for example, an EAN (European Article Number) code, and the information output device 30 is attached to the backboard of a shelf TA on which the products SH are displayed.

[0012] A plurality of shelves TA-1 and TA-2 are installed in the store RT. An information output device 30 is attached to each of the shelves TA-1 and TA-2. Specifically, for example, an information output device 30-1 is attached to the shelf TA-1, and an information output device 30-2 is attached to the shelf TA-2. In other words, the store RT is provided with at least one shelf TA that displays a plurality of products SH, including at least one product SH. An information output device 30 is provided on each of the at least one shelf TA. Note that the information output device 30 is not limited to being attached to the shelf TA, and may be attached to, for example, the ceiling of the store RT. Furthermore, the number of information output devices 30 installed in the store RT is not limited to being multiple, and only one information output device 30 may be installed in the store RT.

[0013] The imaging device 20 is installed in the store RT and captures images of the inside of the store RT. Multiple imaging devices 20 may be installed in the store RT, or only one may be installed. In this embodiment, multiple imaging devices 20 are installed in the store RT, and the entire sales floor of the store RT is covered by multiple captured images PS (see FIG. 4 ) captured by the multiple imaging devices 20.

[0014] The imaging device 20 and the information output control device 10 may be connected via a network (not shown). Alternatively, the information output control device 10 may acquire the captured image PS from the imaging device 20 via a connection interface (not shown) without using a network. The connection interface is an interface for connecting the information output control device 10 to another device. The connection interface may be, for example, a connector to which a USB (Universal Serial Bus) cable or the like is connected, or a communication module that uses short-range wireless communication such as Bluetooth (registered trademark).

[0015] 3 is a block diagram showing the configuration of the information output control device 10. Specifically, the information output control device 10 is an information processing terminal such as a personal computer, a tablet, or a smartphone. The information output control device 10 includes, for example, an input device 100, a display 110, a communication device 120, a storage device 130, a processing device 140, and a bus 150 connecting these devices.

[0016] The input device 100 is an input device (e.g., a keyboard, a mouse, a switch, a button, or a sensor) that accepts input from the outside. The display 110 is a display device (e.g., various display panels such as a liquid crystal display panel or an organic EL display panel) that displays information to the outside. The input device 100 and the display 110 may be integrated, for example, like a touch panel.

[0017] The communication device 120 has a communication interface connectable to a network (not shown) and communicates with other devices via the network. The other devices are, for example, the imaging device 20 and the information output device 30. The communication interface of the communication device 120 may be a wireless communication interface or a wired communication interface.

[0018] The information output control device 10, the imaging device 20, and the information output device 30 may be connected via the above-mentioned network or via a connection interface. The connection interface may be, for example, a connector to which a USB (Universal Serial Bus) cable or the like is connected, or a communication module that uses short-range wireless communication such as Bluetooth (registered trademark).

[0019] The storage device 130 is a recording medium readable by the processing device 140. The storage device 130 may be configured with at least one of, for example, a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The storage device 130 stores a program PG, model data DM, and output data SD.

[0020] The program PG is a program for operating the information output control device 10. The model data DM is data of a three-dimensional store model MT that reproduces the store RT. The model data DM is not limited to being stored in the storage device 130, but may also be stored in, for example, another device connected to the network. In this case, the model data DM may be transmitted from the other device to the information output control device 10 via the network.

[0021] The output data SD is data recording at least one of an image and an audio to be output from the information output device 30. The output data SD is prepared, for example, for each product SH displayed in the store RT. In this case, the output data SD is associated with the EAN code of the product SH indicated by the output data SD. Alternatively, the output data SD may be prepared, for example, for each category of product SH displayed in the store RT (boxed lunches, sweets, beverages, bread, etc.). In this case, the output data SD is associated with the EAN codes of multiple products SH belonging to the category indicated by the output data SD.

[0022] The processing device 140 includes one or more central processing units (CPUs). When an instruction to execute a program PG is received through an input operation on the input device 100, the processing device 140 reads the program PG from the storage device 130 and executes the program PG. By executing the program PG, the processing device 140 functions as an acquisition unit 141, a person model generation unit 142, an arrangement unit 143, an identification unit 144, and an output control unit 145. At least some of the functions of the acquisition unit 141, the person model generation unit 142, the placement unit 143, the identification unit 144, and the output control unit 145 may be configured by circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array).

[0023] The acquisition unit 141 acquires the captured image PS captured by the imaging device 20. When a plurality of imaging devices 20 are installed as in this embodiment, the acquisition unit 141 acquires the captured image PS from each of the plurality of imaging devices 20.

[0024] In this embodiment, the captured images PS are assumed to be still images that constitute a moving image. The imaging device 20 captures a moving image by generating still images (captured images PS) at a predetermined frame rate. The imaging device 20 transmits each still image (captured image PS) to the information output control device 10 each time it generates it. Each captured image PS is assigned imaging time information indicating the time of capture. By capturing a moving image with the imaging device 20, the surrounding conditions can be continuously grasped.

[0025] FIG. 4 is a schematic diagram showing an example of a captured image PS. The captured image PS shown in FIG. 4 depicts a store RT in real space. Real space is the space in which the actual store RT exists. A three-dimensional store model MT (see FIG. 5), which will be described later, is a virtual space model that mimics the store RT in real space. More specifically, the captured image PS depicts people J, such as a customer J1 visiting the store RT and a store clerk J2 at the store RT, in addition to store facilities such as shelves TA, products SH, a floor F, and an information output device 30.

[0026] The person model generation unit 142 generates a three-dimensional person model that imitates the posture of person J based on the captured image PS. The person model generation unit 142 generates the three-dimensional person model using, for example, a machine learning model that uses a neural network. In this embodiment, the three-dimensional person model is a skeleton model BM (BM1 and BM2, see FIG. 5 ) that indicates the skeleton of person J.

[0027] The person model generation unit 142 first identifies the portion of the captured image PS in which person J appears. Next, the person model generation unit 142 detects feature points, such as the joints and facial features of person J, and generates a skeletal model BM by connecting the feature points. Therefore, the skeletal model BM generated by the person model generation unit 142 is a model that reproduces the posture of person J appearing in the captured image PS. In this embodiment, the skeletal model BM includes gaze parts that indicate the direction in which person J is looking, and hand parts that indicate the position of person J's hands. The gaze parts are an example of a first image, and the hand parts are an example of a second image.

[0028] The person model generation unit 142 changes the posture of the skeletal model BM each time a captured image PS is acquired by the acquisition unit 141. Therefore, it can be said that the skeletal model BM generated based on a plurality of captured images PS captured successively in time represents the behavior of the person J appearing in the captured image PS.

[0029] By using the skeletal model BM as the skeletal model BM, it is possible to prevent individual person J from being identified and protect privacy. Furthermore, by using the skeletal model BM as the skeletal model BM, information that is not directly related to the movement of person J, such as the clothing of person J, is removed, thereby reducing the processing load on the processing device 140. Note that the skeletal model BM is not limited to the skeletal model BM, and may be, for example, an avatar that resembles person J.

[0030] The placement unit 143 places a skeletal model BM of at least one person J located in the store RT on a three-dimensional store model MT that reproduces the store RT, based on a captured image PS of the store RT in which a plurality of products SH are displayed. The three-dimensional store model MT is an example of a three-dimensional facility model. In this embodiment, the placement unit 143 places the skeletal model BM generated by the person model generation unit 142 on the three-dimensional store model MT. The skeletal model BM is an example of a behavior image related to the behavior of the person J.

[0031] As described above, the skeleton model BM includes gaze parts and hand parts. Therefore, arranging the skeleton model BM in the three-dimensional store model MT corresponds to the arranging unit 143 arranging, as the skeleton model BM, at least one of the gaze parts indicating the direction in which at least one person J is looking and the hand parts indicating the position of the hand of at least one person J.

[0032] Furthermore, as described above, the posture of the skeletal model BM changes each time a captured image PS is acquired. Therefore, by placing the skeletal model BM on the three-dimensional store model MT and changing the posture and position of the skeletal model BM based on multiple captured images PS captured consecutively over time (in other words, by rearranging the skeletal model BM), the behavior of at least one person J located within the store RT is projected onto the three-dimensional store model MT. In this embodiment, the at least one person J located within the store RT is a customer J1 and a store clerk J2. The at least one person J located within the store RT may also be referred to as the person J appearing in the captured image PS.

[0033] FIG. 5 is a schematic diagram showing an example of a three-dimensional store model MT in which a skeletal model BM is arranged. Any point on the three-dimensional store model MT can be identified by three-dimensional coordinate information (Xv, Yv, Zv) using the orthogonal Xv, Yv, and Zv axes. FIG. 5 shows a two-dimensional image of the three-dimensional store model MT in which skeletal models BM1 and BM2 are arranged, captured by a virtual camera (not shown) at the same angle of view as the captured image PS shown in FIG. 4. Hereinafter, the two-dimensional image of the three-dimensional store model MT captured by the virtual camera will be referred to as the "model image PM." In the model image PM shown in FIG. 5, the customer J1 and the store clerk J2 are represented as skeletal models BM1 and BM2, respectively. The model image PM also includes a shelf model TM corresponding to the shelf TA, a product model SM corresponding to the product SH, a floor model FM corresponding to the floor F, and an information output device model 30M corresponding to the information output device 30. The product model SM is an example of an object model. In this embodiment, the shelf model TM and the product model SM are arranged in accordance with the actual shelf allocation in the store RT.

[0034] In this embodiment, the placement unit 143 estimates three-dimensional coordinates (hereinafter referred to as "real coordinates") indicating the position of the person J in real space based on, for example, the position of the person J on the captured image PS, the depth of the person J in the captured image PS, and the position of the imaging device 20 in real space. Then, the placement unit 143 places the skeletal model BM at a position on the three-dimensional store model MT that corresponds to the real coordinates.

[0035] 6 is a diagram schematically illustrating the position of person J in real space. Any point in real space can be identified by three-dimensional coordinate information (Xr, Yr, Zr) using orthogonal Xr, Yr, and Zr axes. In FIG. 6, the image capturing direction of the image capturing device 20 is defined as Zr, with the center point O of the lens of the image capturing device 20 as the origin, and the upward direction on a plane perpendicular to the Zr axis is defined as Yr, and the rightward direction is defined as Xr.

[0036] Generally, since person J has a three-dimensional extent, the area that person J occupies in real space is indicated by a plurality of pieces of coordinate information. The position of person J may be a plurality of pieces of coordinate information indicating the area that person J occupies in real space, or may be coordinate information indicating the position of a representative point of person J in real space. In FIG. 6 , the midpoint D between the two contact points where both feet of person J touch the floor surface F is defined as the representative point of person J. That is, the position (real coordinates) of person J is defined as three-dimensional coordinates indicating the midpoint D. The three-dimensional coordinates indicating the midpoint D are defined as (Xrh, Yrh, Zrh). Note that the representative point of person J is not limited to the midpoint D, and may be any other part of the human body as long as it is a feature point (e.g., the head or waist) that can be detected from the captured image PS.

[0037] The imaging device 20 is attached to the ceiling of the store RT, and the installation position of the imaging device 20 in real space is known. Furthermore, if the imaging direction and imaging magnification of the imaging device 20 are changeable (i.e., pan, tilt, and zoom are possible), the acquisition unit 141 also acquires the imaging direction and imaging magnification of the imaging device 20 when acquiring the captured image PS. Furthermore, it is assumed that the camera parameters of the imaging device 20 are also known. Therefore, the placement unit 143 can identify the range of real space depicted in the captured image PS, in other words, the imaging range R of the imaging device 20.

[0038] The placement unit 143 also estimates the depth of person J in the captured image PS. The depth of person J in the captured image PS can be rephrased as the depth of person J relative to the imaging device 20 that captures the captured image PS. The depth of person J relative to the imaging device 20 is, for example, the distance between a representative point of the imaging device 20 (for example, the center point O of the lens of the imaging device 20 shown in FIG. 6 ) and each point (or representative point) on the surface of person J. The placement unit 143 estimates the depth of each pixel of the captured image PS using, for example, a machine learning model that uses a neural network. The depth of the pixels in the area of ​​the captured image PS where person J appears is the depth of person J.

[0039] The placement unit 143 converts the position of person J in the captured image PS into a position on the store RT in real space, and specifies the position in the depth direction relative to the imaging device 20 based on the depth estimated as described above. This makes it possible to estimate three-dimensional coordinates indicating the position of person J in real space. In the example of FIG. 6 , three-dimensional coordinates (Xrh, Yrh, Zrh) indicating the position (midpoint D) of person J are estimated. Then, the placement unit 143 places the midpoint D of the skeletal model BM at a position on the three-dimensional store model MT that corresponds to the three-dimensional coordinates (Xrh, Yrh, Zrh).

[0040] More specifically, any point on the three-dimensional store model MT can be identified by three-dimensional coordinate information (Xv, Yv, Zv) using the orthogonal Xv, Yv, and Zv axes. The correspondence between the coordinate information (Xv, Yv, Zv) of the three-dimensional store model MT and the coordinate information (Xr, Yr, Zr) of real space is known and can be converted into one another. The models of various objects in the three-dimensional store model MT (shelf model TM, product model SM, floor model FM, information output device model 30M, etc.) correspond to the positions of various objects in real space (shelves TA, products SH, floor F, information output device 30, etc.). The three-dimensional coordinates indicating the position of the skeleton model BM1 in the three-dimensional store model MT (position N1, described later) are defined as (Xvh, Yvh, Zvh). The placement unit 143 performs coordinate transformation processing on the three-dimensional coordinates (Xrh, Yrh, Zrh) of person J in the store RT in real space to calculate three-dimensional coordinates (Xvh, Yvh, Zvh) and place the skeletal model BM.

[0041] The identification unit 144 identifies an interesting product that is the subject of interest of at least one person J based on the three-dimensional store model MT and the skeletal model BM. An interesting product is an example of an interesting object. Of the product models SM arranged in the three-dimensional store model MT, the identification unit 144 identifies a product that corresponds to at least one of the product models SM that are in the direction of the gaze of at least one person J and the product models SM that are picked up by at least one person J as an interesting product. In other words, an interesting product is at least one product displayed in the store RT. In the following description, the identification unit 144 identifies an interesting product of the store visitor J1.

[0042] For example, the direction of the eyes or the direction of the face of the skeletal model BM1 corresponds to the direction of the gaze of the store visitor J1. If a product model SM (e.g., product model SM1) is located on an extension line of the direction of the eyes or the direction of the face of the skeletal model BM1, the identification unit 144 determines that the store visitor J1 corresponding to the skeletal model BM1 is looking at the product SH1 corresponding to the product model SM1. The product model SM1 is an example of a product model SM arranged in the three-dimensional store model MT that is located in the direction in which the store visitor J1 is looking. Therefore, the identification unit 144 identifies the product SH1 corresponding to the product model SM1 as the product of interest to the store visitor J1.

[0043] Note that, for example, if the customer J1 looks around and his / her eyes are fixed on the product SH1 for a very short time, it cannot be said that the customer J1 is interested in the product SH1. Therefore, the identification unit 144 may identify the product SH1 as an interesting product if the customer J1 looks at the product SH1 for a predetermined time or more (for example, three seconds or more).

[0044] Furthermore, if the hand of the skeletal model BM1 is in contact with one of the product models SM (for example, product model SM2), the identification unit 144 determines that the store visitor J1 corresponding to the skeletal model BM1 is picking up the product SH2 corresponding to the product model SM2. The product model SM2 is an example of a product model SM arranged in the three-dimensional store model MT that the store visitor J1 has picked up. Therefore, the identification unit 144 identifies the product SH2 corresponding to the product model SM2 as the product of interest to the store visitor J1.

[0045] The output control unit 145 outputs information about the product of interest identified by the identification unit 144 from the information output device 30 installed in the store RT. The output control unit 145 transmits output data SD with an EAN code corresponding to the product of interest to the information output device 30. The information output device 30 outputs the output data SD transmitted by the output control unit 145 from at least one of the monitor 30A and the speaker 30B.

[0046] As described above, the object of interest is at least one product SH (e.g., product SH1 or SH2) displayed in the store RT. The output control unit 145 outputs, as information about the product of interest, at least one of, for example, characteristic information about the characteristics of the at least one product SH, benefit information about benefits for purchasing the at least one product SH, and usage information about how to use the at least one product SH.

[0047] For example, if the product SH is a food product, the characteristic information may be information about nutrients obtained by consuming the product SH. Furthermore, if the product SH is a cosmetic product, the characteristic information may be information about how well the product SH stays on the skin, or information about the texture of the product SH when applied to the skin.

[0048] The bonus information may be, for example, information announcing a campaign in which, when purchasing the product SH, a discount on the purchase price, an extra number of points, or a prize is awarded, is being held.

[0049] For example, if the product SH is a medicine, the usage information may be information about the usage, dosage, and precautions for use of the product SH. Also, if the product SH is a food ingredient such as a vegetable or meat, the usage information may be an explanation of how to cook the product SH and a recipe for a dish using the product.

[0050] Furthermore, the information about the product of interest may be, for example, an image used as a commercial for the product of interest in a television broadcast or an internet advertisement.

[0051] 7A is a schematic diagram showing an example of information output by the information output device 30. In FIG. 7A, customer J1 is standing in front of shelf TA-1. Furthermore, customer J1's gaze is directed toward position N1 on product SH3 displayed on shelf TA-1. In this case, the identification unit 144 identifies product SH3 as the product of interest. The output control unit 145 reads output data SD corresponding to product SH3 from the storage device 130 and causes the information output device 30 to output information G1 indicated by the output data SD.

[0052] 7A illustrates information output devices 30, including information output device 30-1 installed on shelf TA-1 and information output device 30-2 installed on shelf TA-2. In the example of FIG. 7A, the output control unit 145 causes information output device 30-1, which displays product SH3, a product of interest, and is located closest to customer J1, to output information G1 about product SH3 (information indicated by the output data SD described above). This allows customer J1 to obtain information G1 about product SH3 while remaining in a position where product SH3 is easy to pick up.

[0053] FIG. 7B is a schematic diagram showing another example of the manner in which information is output by the information output device 30. In FIG. 7B, customer J1 is standing in front of shelf TA-1, as in FIG. 7A. Furthermore, customer J1's gaze is directed toward position N2 on product SH4 displayed on shelf TA-2. In this case, the identification unit 144 identifies product SH4 as the product of interest. The output control unit 145 reads out output data SD corresponding to product SH4 from the storage device 130 and causes the information output device 30 to output information G2 indicated by the output data SD.

[0054] In the example of Figure 7B, the output control unit 145 causes the information output device 30-2 installed on the shelf TA-2 where the product SH4 of interest is displayed to output information G2 (information indicated by the output data SD described above) about the product SH4. By outputting information from the information output device 30-2 instead of the information output device 30-1 closest to the store customer J1, the likelihood that the store customer J1 will move toward the shelf TA-2 and pick up the product SH4 can be increased. This also makes it easier to understand the correspondence between the product SH4 and the information G2.

[0055] 7B, the information output device 30-1 closest to the customer J1 may output the information G2 about the product SH4, which makes it easier for the customer J1 to recognize that the output of the information G2 is directed at him or her.

[0056] In this way, the output control unit 145 outputs information about the object of interest from at least one of the information output devices 30 provided on the shelf TA on which at least one product SH identified as the product of interest is displayed and the information output device 30 provided on the shelf TA closest to the customer J1. When there are multiple information output devices 30 in the store RT, the operator of the store RT may decide in advance which information output device 30 will output information about the product of interest.

[0057] 8 is a flowchart showing the operation of the information output control device 10. The processing device 140 of the information output control device 10 functions as an acquisition unit 141 and acquires a captured image PS of a store RT in which multiple products SH are displayed from the imaging device 20 (step S10). The processing device 140 functions as a person model generation unit 142 and generates a skeletal model BM that imitates the shape of a customer J1 (person J) within the store RT in real space, based on the captured image PS (step S11).

[0058] The processing device 140 functions as the placement unit 143 and estimates three-dimensional coordinates indicating the position of the person J in real space based on the depth estimated from the captured image PS, the position of the person J on the captured image PS, and the position of the imaging device 20 in real space (step S12). Note that the order of the processing of step S11 and the processing of step S12 may be reversed or may be performed simultaneously. The processing device 140 functions as the placement unit 143 and places the skeletal model BM on a three-dimensional store model MT that simulates the store RT in real space at a position corresponding to the three-dimensional coordinates estimated in step S12 (step S13).

[0059] The processing device 140 functions as the identification unit 144 and determines whether the customer J1 has looked at (directed their gaze at) any product SH in the store RT for a predetermined period of time or more, or whether the customer J1 has picked up any product SH, based on the product models SM arranged in the three-dimensional store model MT and the posture of the skeletal model BM (step S14). If the customer J1 has not looked at any product SH for a predetermined period of time or more, or has not picked up any product SH (step S14: NO), the processing device 140 determines that there is no product of interest, and returns the process to step S10.

[0060] On the other hand, if the customer J1 looks at any product SH in the store RT for a predetermined period of time or picks up the product SH (step S14: YES), the processing device 140 functions as the identification unit 144 and identifies the product SH as a product of interest (step S15). The processing device 140 identifies the EAN code of the product of interest based on, for example, the shelf allocation of the product model SM in the three-dimensional store model MT.

[0061] The processing device 140 functions as the output control unit 145, reads the output data SD corresponding to the product of interest from the storage device 130, and outputs information about the product of interest indicated by the output data SD from the information output device 30 (step S16). After that, the processing device 140 returns the process to step S10.

[0062] As described above, the information output control device 10 according to the embodiment projects the behavior of a customer J1 in the store RT onto the three-dimensional store model MT, and when the customer J1 shows interest in a particular product SH, outputs information about the product from the information output device 30. Therefore, information about the product in which the customer J1 is interested can be output with a simple configuration. Specifically, the simple configuration requires only one imaging device 20 in the store RT, which can reduce the capital investment costs for the store RT compared to, for example, installing an imaging device and a sensor on each shelf TA.

[0063] Furthermore, the information output control device 10 identifies, as the product of interest, the product SH corresponding to at least one of the product models SM arranged in the three-dimensional store model MT that is in the direction of the customer J1's line of sight and the product model SM that the customer J1 has picked up. Thus, the product of interest to the customer J1 can be identified in a reliable and simple manner.

[0064] Furthermore, for example, the information output control device 10 outputs, as information about the object of interest, at least one of information about the characteristics of the product of interest, information about benefits for purchasing the product of interest, and information about how to use the product of interest. Therefore, the store customer J1 can use the information output from the information output device 30 as information about whether or not to purchase the product of interest.

[0065] Furthermore, for example, the information output control device 10 outputs information about the product of interest from at least one of the information output device 30 provided on the shelf TA where the product of interest is displayed and the information output device 30 provided on the shelf TA closest to the customer J1. Therefore, the output location of the information about the product of interest can be set according to various purposes, improving convenience for the operator of the store RT.

[0066] B: Modifications The following are modifications of the above-described embodiment. Two or more modifications selected from the following modifications may be combined as appropriate within the scope of not contradicting each other.

[0067] B1: First Modification In the above-described embodiment, the facility where a plurality of objects are displayed is a store RT, and the object of interest to person J is a commodity. However, this is not limited to this, and for example, the facility where a plurality of objects are displayed may be a cultural facility, and person J's object of interest may be at least one exhibit exhibited at the cultural facility. The cultural facility may be, for example, a museum or an art gallery.

[0068] In this case, the output control unit 145 outputs, as information about the interesting exhibits, at least one of information about the historical background of at least one of the exhibits identified as the interesting exhibits and information about people related to at least one of the exhibits identified as the interesting exhibits. An interesting exhibit is an example of an object of interest.

[0069] For example, if the facility is an art museum and the exhibit is a painting, the information about the historical background of the exhibit may be information about the historical background of the time when the painting was created. Also, for example, if the facility is a museum and the exhibit is a fossil, the information about the historical background of the exhibit may be information about the global environment at the time when the organism that became the fossil lived.

[0070] For example, if the facility is an art museum and the exhibit is a painting, the information about the person related to the exhibit may be information about the life of the artist who created the painting. Also, if the facility is a museum and the exhibit is a fossil, the information about the person related to the exhibit may be information about the person who excavated the fossil or the researcher who conducted research based on the fossil.

[0071] More specifically, in a cultural facility, for example, a monitor may be installed next to an exhibit, and a documentary video related to the exhibit may be displayed on the monitor. Also, for example, in a corner of the exhibition space of a cultural facility, a monitor and buttons for selecting a documentary video may be installed, and when a visitor presses one of the buttons, the documentary video corresponding to that button may be displayed on the monitor. The first modification aims to automatically display such documentary video in accordance with the visitor's interests.

[0072] In many cases, cultural facilities have at least one exhibition area displaying multiple exhibits, including an interesting exhibit. Specifically, for example, multiple fossils estimated to date from the same era may be displayed in a single exhibition area. When an information output device 30 is provided in each of the at least one exhibition area, the output control unit 145 outputs information about the interesting exhibit from at least one of the information output devices 30 provided in the exhibition area where the interesting exhibit is displayed and the information output device 30 provided in the exhibition area closest to the visitor. By outputting information about the interesting exhibit from the information output device 30 provided in the exhibition area where the interesting exhibit is displayed, the correspondence between the output information and the exhibit becomes clear. Furthermore, by outputting information about the interesting exhibit from the information output device 30 provided in the exhibition area closest to the visitor, even if the visitor has difficulty moving around, the information about the interesting exhibit can be obtained on the spot.

[0073] According to the first modification, it is easy to provide visitors at a cultural facility with information about exhibits of interest, allowing the cultural facility to be used effectively. Furthermore, the operator of the cultural facility can analyze which exhibits are likely to be considered interesting exhibits, i.e., which exhibits are popular, and can use the results of this analysis in the operation of the cultural facility.

[0074] C: Others (1) In the above-described embodiment, ROM, RAM, etc. are exemplified as storage device 130, but storage device 130 may also be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or other suitable storage medium.

[0075] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0076] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0077] (4) In the above-described embodiment, the determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a comparison of numerical values ​​(e.g., a comparison with a predetermined value).

[0078] (5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0079] (6) Each function illustrated in FIG. 3 is realized by any combination of hardware and / or software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are connected directly or indirectly (e.g., via wire, wirelessly, etc.) and these multiple devices. A functional block may be realized by combining software with the single device or the multiple devices.

[0080] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0081] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0082] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0083] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0084] (10) In the above-described embodiments, the portable device may be a mobile station (MS). Those skilled in the art may also refer to a mobile station as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other appropriate terminology. In this disclosure, the terms "mobile station," "user terminal," "user equipment (UE)," "terminal," etc. may be used interchangeably.

[0085] (11) In the above-described embodiments, the terms "connected," "coupled," or any variation thereof refers to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0086] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0087] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining something as "determining" or "determining," and the like. Furthermore, "judgment" and "decision" may include regarding receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and accessing (e.g., accessing data in memory) as having been "judgment" or "decision." Furthermore, "judgment" and "decision" may include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judgment" or "decision." In other words, "judgment" and "decision" may include regarding some action as having been "judgment" or "decision." Furthermore, "judgment" may be interpreted as "assuming," "expecting," "considering," etc.

[0088] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or," as used in this disclosure, is not intended to be an exclusive or.

[0089] (15) In this disclosure, when articles are added by translation, such as a, an, and the in English, the disclosure may include the noun following these articles being plural.

[0090] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0091] (17) The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to being explicit, but may be implicit (e.g., not notifying the predetermined information).

[0092] 1...system, 10...information output control device, 20...imaging device, 30 (30-1, 30-2)...information output device, 100...input device, 110...display, 120...communication device, 130...storage device, 140...processing device, 141...acquisition unit, 142...person model generation unit, 143...placement unit, 144...identification unit, 145...output control unit, BM (BM1, BM2)...skeletal model, DM...model data, J...person, J1...store customer, J2...store clerk, MT...3D store model, PM...model image, PS...captured image, RT...store, SD...output data, SH (SH1 to SH4)...product, SM (SM1, SM2)...product model, TA (TA-1, TA-2)...shelf, TM...shelf model.

Claims

1. An information output control device comprising: a placement unit that places, based on a captured image of a facility where multiple objects are displayed, a behavior image relating to the behavior of at least one person located within the facility on a three-dimensional facility model that reproduces the facility; an identification unit that identifies an object of interest that is the subject of interest of the at least one person based on the three-dimensional facility model and the behavior image; and an output control unit that outputs information relating to the object of interest identified by the identification unit from an information output device installed in the facility.

2. The information output control device described in claim 1, wherein the placement unit places at least one of a first image indicating the direction in which the at least one person is looking and a second image indicating the position of the at least one person's hand as the behavior image, and the identification unit identifies, among the object models placed in the three-dimensional facility model, an object corresponding to at least one of an object model in the direction in which the at least one person is looking and an object model held in the hand of the at least one person as the object of interest.

3. An information output control device as described in claim 1, wherein the facility is a store, the object of interest is at least one product displayed in the store, and the output control unit outputs, as information about the object of interest, at least one of information about the characteristics of the at least one product, information about benefits for purchasing the at least one product, and information about how to use the at least one product.

4. An information output control device as described in claim 3, wherein the store is provided with at least one shelf displaying a plurality of products including the at least one product, the information output device is provided on each of the at least one shelf, and the output control unit outputs information about the object of interest from at least one of the information output device provided on the shelf displaying the at least one product and the information output device provided on the shelf closest to the at least one person.

5. An information output control device as described in claim 1, wherein the facility is a cultural facility, the object of interest is at least one exhibit displayed at the cultural facility, and the output control unit outputs, as information about the object of interest, at least one of information about the historical background of the at least one exhibit and information about a person associated with the at least one exhibit.

6. The information output control device of claim 5, wherein the cultural facility has at least one exhibition area displaying a plurality of exhibits including the at least one exhibit, the information output device is provided in each of the at least one exhibition area, and the output control unit outputs information about the object of interest from at least one of the information output device provided in the exhibition area where the at least one exhibit is displayed and the information output device provided in the exhibition area closest to the at least one person.

7. An information output control method comprising: based on a captured image of a facility where a plurality of objects are displayed, arranging a behavior image relating to the behavior of at least one person located within the facility on a three-dimensional facility model that reproduces the facility; identifying an object of interest that is the subject of interest of the at least one person based on the three-dimensional facility model and the behavior image; and outputting information relating to the object of interest from an information output device installed in the facility.

Images