Display control device and display control method

Abstract

In the present invention, on the basis of an image obtained by imaging the surroundings of an object, an arrangement unit arranges a person image on a three-dimensional model in which the surroundings of the object are reproduced, the person image concerning an action of at least one person positioned around the object. When the at least one person has performed a related action related to the object, an identification unit identifies a related position concerning the related action. A display control unit displays a two-dimensional image obtained by imaging the three-dimensional model after applying image processing to a part corresponding to the related position.

Description

Display control device and display control method

[0001] The present invention relates to a display control device and a display control method.

[0002] Conventionally, there is known a technique for analyzing images captured by a surveillance camera or the like and estimating the behavior of an object to be analyzed. For example, an information processing system described in Patent Document 1 below evaluates the visibility of an object (e.g., an outdoor advertisement) by analyzing the movement of individuals located near the object. In a receiving step, captured images including people are received in chronological order. In an estimation step, a movement direction of the people is estimated for each predetermined time based on the time-series changes in the captured images. In a first calculation step, a predetermined first process is applied to a plurality of movement directions for each predetermined time to calculate the degree of attention the people are paying to the object.

[0003] Japanese Patent Application Laid-Open No. 2022-142623

[0004] In the above-mentioned Patent Document 1, the attention level to an object is calculated as the sum of the attention levels of multiple people in the vicinity. However, if it is desired to analyze the behavior of people around the object (the part of the object that is being looked at, the position of the person looking at the object, or the movement before and after looking at the object, etc.) instead of the sum of the attention levels, the technology disclosed in Patent Document 1 cannot be used.

[0005] An object of the present invention is to understand the behavior of a person who is interested in an object.

[0006] A display control device according to one aspect of the present invention includes an arrangement unit that arranges, based on an image of the area around an object, a person image relating to the behavior of at least one person located around the object on a three-dimensional model that reproduces the area around the object; an identification unit that, when the at least one person performs a related behavior related to the object, identifies a related position relating to the related behavior; and a display control unit that displays a two-dimensional image of the three-dimensional model after performing image processing on the portion corresponding to the related position.

[0007] A display control method according to one aspect of the present invention places, based on an image of the area around an object, a person image relating to the behavior of at least one person located around the object on a three-dimensional model that reproduces the area around the object, and when the at least one person performs a related behavior related to the object, identifies a related position relating to the related behavior, and displays a two-dimensional image of the three-dimensional model after performing image processing on the portion corresponding to the related position.

[0008] According to one aspect of the present invention, it is possible to grasp the behavior of a person who is interested in an object.

[0009] 1 is a block diagram showing a configuration of a system 1 according to an embodiment. FIG. 2 is a block diagram showing a configuration of an information management device 10. FIG. 3 is a schematic diagram showing an example of a captured image PS. FIG. 4 is a schematic diagram showing an example of a three-dimensional space model MS in which a skeletal model BM is arranged. FIG. 5 is a schematic diagram showing the position of a person J1 in a real space RS. FIG. 6 is a schematic diagram showing an example of a floor image PM2. FIG. 7 is a schematic diagram showing another example of the floor image PM2. FIG. 8 is a schematic diagram showing another example of the floor image PM2. FIG. 9 is a schematic diagram showing an example of a front image PM3. FIG. 10 is a flowchart showing the operation of the information management device 10.

[0010] A. Embodiment FIG. 1 is a block diagram showing the configuration of a system 1 according to an embodiment. The system 1 includes an information management device 10 and an imaging device 20. The information management device 10 is an example of a display control device. The information management device 10 displays information (hereinafter referred to as "behavior information") regarding the behavior of a person J (see FIG. 3) who is paying attention to an object. In this embodiment, the object is a signage display SD installed in a train station. As will be described in detail later, the behavior information is information indicating, for example, the position of the person J looking at the signage display SD, the part of the signage display SD that the person J is looking at, and the movement trajectory of the person J before and after looking at the signage display SD.

[0011] In this embodiment, the user of the system 1 is a person who evaluates the impact that a display using the signage display SD has on a nearby person J. Specifically, the user is, for example, an installer of the signage display SD, a station operator, or an advertiser of advertisements or the like to be displayed on the signage display SD.

[0012] The imaging device 20 is installed in a position that includes the target signage display SD within its imaging range R, and captures a captured image PS (see FIG. 3 ) that includes the signage display SD and surrounding people J. The imaging device 20 may be a surveillance camera installed within a station. In this embodiment, one imaging device 20 is used; however, multiple imaging devices 20 may be installed around the signage display SD. When multiple imaging devices 20 are installed, captured images PS of multiple signage displays SD with different angles of view can be obtained, improving the accuracy of behavior information. Furthermore, when multiple imaging devices 20 are installed, the total imaging range R is wider, allowing behavior information over a wider range to be obtained. Note that, as long as the positional relationship between the imaging range R of the imaging device 20 and the signage display SD is known, the imaging range R does not necessarily have to include the signage display SD.

[0013] When the imaging device 20 is located within a range where it can be connected to the information management device 10 via a wired connection, the captured image PS may be acquired from the imaging device 20 via a connection interface (not shown) provided in the information management device 10, without going through the network N. The connection interface is an interface for connecting the information management device 10 to other devices. The connection interface may be, for example, a connector to which a USB (Universal Serial Bus) cable or the like is connected, or a communication module that uses short-range wireless communication such as Bluetooth (registered trademark).

[0014] 2 is a block diagram showing the configuration of the information management device 10. Specifically, the information management device 10 is an information processing terminal such as a personal computer, a tablet, or a smartphone. The information management device 10 includes, for example, an input device 100, a display 110, a communication device 120, a storage device 130, a processing device 140, and a bus 150 connecting these devices.

[0015] In this embodiment, the information management device 10 includes a display 110 and displays warning information on the display 110. However, the information management device 10 may not include the display 110, and may output (transmit, etc.) display control information for displaying behavior information on a display of another device, for example.

[0016] The input device 100 is an input device (e.g., a keyboard, a mouse, a switch, a button, or a sensor) that accepts input from the outside. The display 110 is a display device (e.g., various display panels such as a liquid crystal display panel or an organic EL display panel) that displays information to the outside. The input device 100 and the display 110 may be integrated, for example, like a touch panel.

[0017] The communication device 120 has a communication interface connectable to a network N and communicates with other devices via the network N. The other devices are, for example, the imaging device 20. The communication interface of the communication device 120 may be a wireless communication interface or a wired communication interface.

[0018] The storage device 130 is a recording medium readable by the processing device 140. The storage device 130 may be configured with at least one of, for example, a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), and a random access memory (RAM). The storage device 130 stores a program PG and model data MD. The program PG is a program for operating the information management device 10. The model data MD is data of a three-dimensional space model MS that reproduces a real space RS (e.g., the entire interior of a train station) around the signage display SD. The model data MD is not limited to being stored in the storage device 130, but may also be stored in another device connected to the network N, for example. In this case, the model data MD may be transmitted to the information management device 10 from another device via the network N.

[0019] The processing device 140 includes one or more central processing units (CPUs). When an instruction to execute a program PG is received through an input operation on the input device 100, the processing device 140 reads the program PG from the storage device 130 and executes the program PG. By executing the program PG, the processing device 140 functions as an acquisition unit 141, a person model generation unit 142, an arrangement unit 143, an identification unit 144, and a display control unit 145. At least some of the functions of the acquisition unit 141, the person model generation unit 142, the placement unit 143, the identification unit 144, and the display control unit 145 may be configured by circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array).

[0020] The acquisition unit 141 acquires captured images PS of the surroundings of the signage display SD from the imaging device 20. If multiple imaging devices 20 are installed, the acquisition unit 141 acquires captured images PS from each of the multiple imaging devices 20. In this embodiment, the captured images PS are still images that constitute a moving image. The imaging device 20 captures a moving image by generating still images (captured images PS) at a predetermined frame rate. Each time the imaging device 20 generates a still image (captured image PS), it transmits the still image to the information management device 10. Each captured image PS is assigned imaging time information that indicates the time of capture. By capturing a moving image with the imaging device 20, the surrounding conditions can be continuously grasped.

[0021] Fig. 3 is a schematic diagram showing an example of a captured image PS. The captured image PS shown in Fig. 3 shows the inside of a train station, which is a real space RS. The real space RS is a space in which people J (J1 to J4) and the like are actually located. A three-dimensional space model MS (see Fig. 4), which will be described later, is a virtual space model that imitates the real space RS.

[0022] More specifically, the captured image PS includes images of the signage display SD, people J1-J4, the floor F, and the walls W1-W2. In other words, the captured image PS includes images of the signage display SD, people J1-J4, the floor F, and the walls W1-W2 that exist in the real space RS. The signage display SD includes a display surface DP1. Various images (signage images) are displayed on the display surface DP1. The display of images on the signage display SD is managed, for example, by a display management device (not shown). The display management device may be the information management device 10. The display management device manages, for example, display history information that indicates what images were displayed on the display surface DP1 at each time. Therefore, for example, by comparing the capture time of a captured image PS with the display history information, a user can understand what images were displayed on the signage display SD at that capture time.

[0023] The person model generation unit 142 generates a three-dimensional person model that imitates the posture of person J based on the captured image PS. The person model generation unit 142 generates the three-dimensional person model using, for example, a machine learning model that uses a neural network. In this embodiment, the three-dimensional person model is a skeleton model BM (BM1 to BM4, see FIG. 4 ) that indicates the skeleton of person J.

[0024] The person model generation unit 142 first identifies the portion of the captured image PS in which person J appears. Next, the person model generation unit 142 detects feature points, such as the joints and facial features of person J, and generates a skeletal model BM by connecting the feature points. Therefore, the skeletal model BM generated by the person model generation unit 142 is a model that reproduces the posture of person J appearing in the captured image PS.

[0025] The person model generation unit 142 changes the posture of the skeletal model BM each time a captured image PS is acquired by the acquisition unit 141. Therefore, it can be said that the skeletal model BM generated based on a plurality of captured images PS captured successively in time represents the behavior of the person J appearing in the captured image PS.

[0026] By using the skeletal model BM as the skeletal model BM, it is possible to prevent individual person J from being identified and protect privacy. Furthermore, by using the skeletal model BM as the skeletal model BM, information that is not directly related to the movement of person J, such as the clothing of person J, is removed, thereby reducing the processing load on the processing device 140. Note that the skeletal model BM is not limited to the skeletal model BM, and may be, for example, an avatar that resembles person J.

[0027] The placement unit 143 places an image relating to the behavior of at least one person J located around the signage display SD on a three-dimensional space model MS that reproduces the structures around the signage display SD, based on a captured image PS of the area around the signage display SD. In this embodiment, the placement unit 143 places a skeletal model BM generated by the person model generation unit 142 on the three-dimensional space model MS. The skeletal model BM is an example of an image relating to the behavior of person J.

[0028] As described above, the posture of the skeletal model BM changes each time a captured image PS is acquired. Therefore, by placing the skeletal model BM on the three-dimensional space model MS and changing the posture and position of the skeletal model BM based on multiple captured images PS captured consecutively over time (in other words, by rearranging the skeletal model BM), the behavior of at least one person J positioned around the signage display SD is projected onto the three-dimensional space model MS. In this embodiment, the at least one person J positioned around the signage display SD refers to people J1 to J4. The at least one person J positioned around the signage display SD may also be referred to as the person J appearing in the captured image PS.

[0029] FIG. 4 is a schematic diagram showing an example of a three-dimensional space model MS in which a skeletal model BM is arranged. Any point on the three-dimensional space model MS can be identified by three-dimensional coordinate information (Xv, Yv, Zv) using the orthogonal Xv-axis, Yv-axis, and Zv-axis. FIG. 4 is a two-dimensional image of the three-dimensional space model MS in which skeletal models BM1 to BM4 are arranged, captured by a virtual camera (not shown) at the same angle of view as the captured image PS shown in FIG. 3. Hereinafter, the two-dimensional image of the three-dimensional space model MS captured by the virtual camera will be referred to as a "model image PM." The model image shown in FIG. 4 will be referred to as model image PM1.

[0030] In the model image PM1, the people J1 to J4 are shown as skeletal models BM1 to BM4, respectively. The model image PM1 also includes a display model DM corresponding to the signage display SD, a floor model FM corresponding to the floor F, and wall models WM1 to WM2 corresponding to the walls W1 to W2. The surface of the display model DM corresponding to the display surface DP1 of the signage display SD is referred to as the display surface DP2.

[0031] In this embodiment, the placement unit 143 estimates three-dimensional coordinates (hereinafter referred to as "real coordinates") indicating the position of the person J in the real space RS based on, for example, the position of the person J on the captured image PS, the depth of the person J in the captured image PS, and the position of the imaging device 20 in the real space RS. Then, the placement unit 143 places the skeletal model BM at a position on the three-dimensional space model MS that corresponds to the real coordinates.

[0032] 5 is a schematic diagram showing the position of person J1 in real space RS. Any point in real space RS can be identified by three-dimensional coordinate information (Xr, Yr, Zr) using orthogonal Xr, Yr, and Zr axes. In FIG. 5, the image capturing direction of image capturing device 20, with the center point O of the lens of image capturing device 20 as the origin, is Zr, and the upward direction on a plane perpendicular to the Zr axis is Yr, and the rightward direction is Xr.

[0033] Generally, since person J1 has a three-dimensional extent, the area occupied by person J1 in real space RS is represented by a plurality of pieces of coordinate information. The position of person J1 may be a plurality of pieces of coordinate information indicating the area occupied by person J1 in real space RS, or may be coordinate information indicating the position of a representative point of person J1 in real space RS. In FIG. 5 , the midpoint D between the two contact points where both feet of person J1 touch the floor surface F is defined as the representative point of person J1. That is, the position (real coordinates) of person J1 is represented by three-dimensional coordinates indicating the midpoint D (representative point). The three-dimensional coordinates indicating the midpoint D are defined as (Xrh, Yrh, Zrh). Note that the representative point of person J1 is not limited to the midpoint D, and may be any other part of the human body as long as it is a feature point (e.g., the head or waist) that can be detected from the captured image PS.

[0034] The imaging device 20 is attached to the ceiling, and the installation position of the imaging device 20 in the real space RS is known. Furthermore, if the imaging direction and imaging magnification of the imaging device 20 are changeable (i.e., pan, tilt, and zoom are possible), the acquisition unit 141 also acquires the imaging direction and imaging magnification of the imaging device 20 when acquiring the captured image PS. Furthermore, it is assumed that the camera parameters of the imaging device 20 are also known. Therefore, the placement unit 143 can identify the range of the real space RS that appears in the captured image PS, in other words, the imaging range R of the imaging device 20.

[0035] The placement unit 143 also estimates the depth of person J1 in the captured image PS. The depth of person J1 in the captured image PS can be rephrased as the depth of person J1 relative to the imaging device 20. The depth of person J1 relative to the imaging device 20 is, for example, the distance between a representative point of the imaging device 20 (for example, the center point O of the lens of the imaging device 20 shown in FIG. 5 ) and a representative point of person J1. The placement unit 143 estimates the depth of each pixel of the captured image PS using, for example, a machine learning model using a neural network. The depth of the pixel in the area of ​​the captured image PS where the midpoint D is captured is the depth of person J.

[0036] The placement unit 143 converts the position of the person J1 in the captured image PS into a position in the real space RS, and by specifying the position in the depth direction relative to the imaging device 20 based on the depth estimated as described above, it is possible to estimate three-dimensional coordinates indicating the position of the person J1 in the real space RS. In the example of Fig. 5, three-dimensional coordinates (Xrh, Yrh, Zrh) indicating the position (midpoint D) of the person J1 are estimated. Then, the placement unit 143 places the midpoint D of the skeletal model BM at a position on the three-dimensional space model MS that corresponds to the three-dimensional coordinates (Xrh, Yrh, Zrh).

[0037] More specifically, any point in the three-dimensional space model MS can be identified by three-dimensional coordinate information (Xv, Yv, Zv) using the orthogonal Xv, Yv, and Zv axes. Because the three-dimensional space model MS is a virtual space model that simulates the real space RS, the correspondence between the coordinate information (Xv, Yv, Zv) of the three-dimensional space model MS and the coordinate information (Xr, Yr, Zr) of the real space RS is known and mutually convertible. For example, the models of various objects in the three-dimensional space model MS (floor model FM, wall models WM1, WM2, display model DM, etc.) correspond to the positions of various objects in the real space RS (floor F, wall surfaces W1, W2, signage display SD, etc.). For example, the three-dimensional coordinates indicating the position of the skeletal model BM1 in the three-dimensional space model MS (related position N1, described later) are (Xvh, Yvh, Zvh). The placement unit 143 performs coordinate transformation processing on the three-dimensional coordinates (Xrh, Yrh, Zrh) of the person J1 in the real space RS to calculate three-dimensional coordinates (Xvh, Yvh, Zvh) and place the skeletal model BM1.

[0038] When the persons J1 to J4 perform a related behavior related to the signage display SD, the identification unit 144 identifies a related position related to the related behavior. In this embodiment, the related behavior is "looking at the signage display SD." For example, the direction of the eyes or the face of the skeletal models BM1 to BM4 corresponds to the direction of the gaze of each of the persons J1 to J4. When a display model DM is located on an extension line of the direction of the eyes or the face of any of the skeletal models BM1 to BM4, the identification unit 144 determines that the person J corresponding to that skeletal model BM is looking at the signage display SD. In the following example, the description will be given assuming that the person J1 is looking at the signage display SD. The related position N1 in FIG. 4 is the position of the person J1 looking at the signage display SD (more specifically, the midpoint D between the two contact points where both feet of the person J1 touch the floor surface F), and the position N2 is the position on the signage display SD that the person J1 is looking at.

[0039] The display control unit 145 displays the model image PM obtained by capturing the three-dimensional space model MS after performing image processing on the portion corresponding to the relevant position. As the image processing, the display control unit 145 performs, for example, adding an image indicating the relevant position.

[0040] The specification unit 144 and the display control unit 145 will be described in detail below with reference to examples 1 to 3.

[0041] [Example 1] The identification unit 144 identifies, for example, the position of person J1 at the time when person J1 looks at the signage display SD as the relevant position. The display control unit 145 displays a model image PM to which an image indicating the relevant position is added. The position of person J1 identified as the relevant position can be, for example, midpoint D (see FIG. 5 ) between two contact points where both feet of person J1 touch the floor surface F.

[0042] Furthermore, the position of person J1 identified as the relevant position may be the center of gravity of person J1 or the center point of person J1's head. These positions can be estimated, for example, from the positions of feature points of skeletal model BM1 in the three-dimensional space model MS. The position of person J1 identified as the relevant position may be determined, for example, based on the imaging direction of the model image PM. The virtual camera can be set at any position, and the model image PM can be captured from any direction. For example, a model image captured from a direction perpendicular to the floor surface (floor surface model FM) of the three-dimensional space model MS (see FIG. 6A, etc.; hereinafter referred to as the "floor surface image PM2") or a model image captured from a direction perpendicular to the display surface DP2 of the display model DM (see FIG. 7A, etc.; hereinafter referred to as the "front image PM3") may be used. The display surface DP2 of the display model DM is an example of the reference surface of the signage display SD. Note that, if the target object is an object without a surface, such as an objet d'art or a product, the surface that most easily captures the characteristics of the object may be set as the reference surface.

[0043] Use of the floor image PM2 improves the visibility of the distribution or changes of related positions along the floor F compared to an image captured from an oblique direction such as the model image PM1. Use of the front image PM3 also improves the visibility of the distribution or changes of related positions along the display surface DP2 compared to an image captured from an oblique direction such as the model image PM1.

[0044] More specifically, for example, if the relevant position is the position of person J1 at the time when he / she looks at the signage display SD, the visibility of the distribution of the relevant positions is improved by using the floor image PM2. Therefore, the identification unit 144 determines that the relevant position N1 is the midpoint D (a point on the floor F) between the ground contact points of both feet of person J1.

[0045] FIG. 6A is a schematic diagram showing an example of a floor image PM2. In FIG. 6A, a marker PL1 is displayed at the same position as the related position N1 in the model image PM1 of FIG. 4. That is, FIG. 6A is an example in which the display control unit 145 adds a marker PL1 to the related position N1 relative to the signage display SD as image processing. In FIG. 6A, only the marker PL1 corresponding to person J1 is displayed, but if there is a person J other than person J1 looking at the signage display SD, a marker PL indicating the position of that person J may also be displayed in the floor image PM2. Furthermore, an arrow indicating the direction of the person J1's line of sight or the like may be displayed in the floor image PM2 simultaneously with the marker PL1.

[0046] 6B is a schematic diagram showing another example of the floor image PM2. For example, if a person J1 moves while looking at the signage display SD, the display control unit 145 may display multiple markers PL, for example, each time a captured image PS is acquired. The display control unit 145 may also draw line segments LI connecting the multiple markers PL. In this case, the display control unit 145 draws line segments LI indicating changes in position related to behavior with respect to the signage display SD. Specifically, for example, as in the image PD1 of FIG. 6B, multiple markers PL and line segments LI connecting these markers PL (or only line segments LI) may be drawn.

[0047] Furthermore, when person J1 stays in one position while looking at the signage display SD, the display control unit 145 may change the color of the marker PL, for example, based on the duration of time the person J1 stays at that position. Specifically, as with markers PL2 and PL3 in FIG. 6B , the longer the duration the person J1 stays, the brighter or more hued the marker PL may become. In the example of FIG. 5 , marker PL2 is darker in color than marker PL3, indicating that person J1 stayed at the position of marker PL2 for a longer period of time.

[0048] [Example 2] The identification unit 144 further identifies the trajectory of the position of person J1 during a period including the time when the person J1 viewed the signage display SD. In other words, the identification unit 144 identifies the trajectory of the position (movement trajectory) of person J1 before and after the time when the person J1 viewed the signage display SD. The display control unit 145 adds to the model image PM an image indicating the position of person J1 at the time when the person J1 viewed the signage display SD as well as the trajectory of the position of person J1 before and after the time when the person J1 viewed the signage display SD. In Example 1 shown in FIG. 6A , only the position of person J1 while viewing the signage display SD was displayed, but in Example 2, the position of person J1 while viewing the signage display SD and the trajectory of movement before and after the time when the person J1 viewed the signage display SD are displayed.

[0049] FIG. 6C is a schematic diagram showing another example of the floor image PM2. For example, as in Example 1, assume that person J1 views the signage display SD at related position N1. In Example 2, the identification unit 144 identifies, in addition to the related position N1, the movement trajectory of person J1 during a predetermined period before and after reaching the related position N1. As in image PD2, the display control unit 145 displays a marker PL4 at the related position N1, as well as line segments LI2 and LI3 indicating the movement trajectory before and after reaching the related position N1. Line segment L2 indicates the movement trajectory before reaching position N1, and line segment LI3 indicates the movement trajectory after reaching position N1. Line segments L2 and LI3 may be displayed in different colors or line types. The predetermined period may be set to, for example, several tens of seconds or several minutes.

[0050] In this way, by displaying the movement trajectory before and after viewing the signage display SD, it is possible to grasp, for example, changes in the behavior of person J1 before and after viewing the signage display SD. Specifically, for example, assume that an advertisement for shop A located inside a train station is displayed on the signage display SD. If person J1 views the advertisement and moves in the direction of shop A, it can be assumed that person J1 has become interested in shop A due to the effect of the advertisement.

[0051] Fig. 6D is a schematic diagram showing another example of the floor image PM2. For example, by tallying the positions of person J who viewed the signage display SD over a predetermined period of time, the distribution of the positions of person J who viewed the signage display SD can be obtained. Fig. 6D is a heat map in which the positions of person J who viewed the signage display SD are tallied over a predetermined period of time. In other words, Fig. 6D is an example in which the display control unit 145 adds an image showing the distribution of multiple related positions as image processing.

[0052] 6D makes it possible to ascertain, for example, from which positions in the station the signage display SD is most frequently viewed, etc. Therefore, for example, the user can increase the visibility of the signage display SD from positions where the signage display SD is most frequently viewed (by removing obstacles, etc.), or consider whether the flow of people J looking at the signage display SD is disrupting the flow of people in the station.

[0053] The floor image PM2 shown in Figures 6A to 6D is obtained, for example, by the following procedure. First, the floor equation of the three-dimensional space model MS and the barycentric coordinates of the three-dimensional space model MS are obtained. Here, the barycentric coordinates may be the barycentric coordinates of the three-dimensional space model MS itself if the three-dimensional space model MS is a model of a relatively small structure such as a convenience store. On the other hand, if the three-dimensional space model MS is a model of a relatively large structure such as a terminal station, the barycentric coordinates may be, for example, the barycentric coordinates of a partial area of ​​the three-dimensional space model MS (e.g., the concourse area of ​​the station). Next, a virtual camera is placed at a position overlapping the barycentric coordinates of the three-dimensional space model MS. Furthermore, the orientation (imaging direction) of the virtual camera is adjusted so that it is perpendicular to the floor equation. Then, the floor image PM2 is obtained by capturing an image with the virtual camera in ortho mode.

[0054] [Example 3] The identification unit 144 identifies the positions on the signage display SD viewed by person J1 as related positions. The display control unit 145 displays a front image PM3 with markers PL attached to the related positions. When the related positions are positions on the signage display SD, using the front image PM3 improves the visibility of the distribution of the related positions.

[0055] Fig. 7A is a schematic diagram showing an example of a front image PM3. In Fig. 7A, a marker PL5 is displayed at the same position as the related position N2 in the model image PM1 in Fig. 4. That is, Fig. 7A is an example in which the display control unit 145 adds the marker PL5 to the related position N2 on the signage display SD as image processing.

[0056] 7A, only the marker PL5 corresponding to the person J1 is displayed, but if there is another person J other than the person J1 looking at the signage display SD, the position at which the person J is looking may also be displayed in the front image PM3 at the same time. Furthermore, the color or shape of the marker PL may be changed depending on, for example, the time that the person J1 is looking at the signage display SD (viewing duration).

[0057] Furthermore, the image displayed on the signage display SD generally changes over time. Therefore, the image displayed on the display surface DP2 when person J1 was looking at the signage display SD may be displayed in the portion of the display surface DP2 in the front image PM3 shown in Fig. 7A. This makes it possible to know what image person J1 was interested in.

[0058] 7B is a schematic diagram showing another example of the front image PM3. For example, by tallying up the positions on the display surface DP1 of the signage display SD viewed by person J over a predetermined period, the distribution of positions viewed on the display surface DP1 can be obtained. FIG. 7B is a heat map in which the positions viewed by person J on the display surface DP1 are tallied over a predetermined period. In other words, FIG. 7B is an example in which the display control unit 145 adds an image showing the distribution of related positions as image processing.

[0059] 7B, it is possible to grasp not only whether the display on the signage display SD is being viewed, but also which part of the image displayed on the display surface DP1 is attracting attention. Therefore, it is possible to consider the content of the image that is most appealing to person J, for example.

[0060] Fig. 8 is a flowchart showing the operation of the information management device 10. It is assumed that, prior to the processing shown in Fig. 8, a model image PM is generated by capturing an image of a three-dimensional space model MS using a virtual camera installed at an arbitrary position.

[0061] The processing device 140 of the information management device 10 functions as an acquisition unit 141 and acquires a captured image PS of the surroundings of the signage display SD, which is an object, from the imaging device 20 (step S10). The processing device 140 functions as a person model generation unit 142 and generates a skeletal model BM that imitates the posture of the person J in the real space RS based on the captured image PS (step S11).

[0062] The processing device 140 functions as the placement unit 143 and estimates three-dimensional coordinates indicating the position of the person J in the real space RS based on the depth estimated from the captured image PS, the position of the person J on the captured image PS, and the position of the imaging device 20 in the real space RS (step S12). Note that the order of the processing of step S11 and the processing of step S12 may be reversed or may be performed simultaneously. The processing device 140 functions as the placement unit 143 and places the skeletal model BM on a three-dimensional space model MS that simulates the structures surrounding the signage display SD at a position corresponding to the three-dimensional coordinates estimated in step S12 (step S13).

[0063] The processing device 140 functions as the identification unit 144 and determines whether the person J has looked at the signage display SD (whether the person J has performed a related behavior) based on the posture of the skeletal model BM in the three-dimensional space model MS (step S14). If the person J has not looked at the signage display SD (step S14: NO), the processing device 140 proceeds to step S17.

[0064] If the person J is looking at the signage display SD (step S14: YES), the processing device 140 identifies a related position related to the related behavior (step S15). The related position is, for example, the position of the person J while looking at the signage display SD, or the position on the display surface DP1 of the signage display SD that the person J is looking at.

[0065] The processing device 140 functions as the display control unit 145 and performs image processing on the portion of the model image PM corresponding to the relevant position (step S16). Then, the processing device 140 functions as the display control unit 145 and displays the model image PM on the display 110 (step S17). That is, if person J is performing the relevant behavior, the model image PM to which image processing has been applied is displayed at the relevant position, and if person J is not performing the relevant behavior, the model image PM to which image processing has not been applied is displayed. Thereafter, the processing device 140 returns the process to step S10.

[0066] As described above, when a person J located around the signage display SD performs a related behavior, the information management device 10 according to the embodiment displays a model image PM in which image processing has been applied to a portion corresponding to the related position. Therefore, the user can obtain, for example, information about the related position in addition to the fact that the person J performed the related behavior, and can use this information to consider the related behavior. In this embodiment, the related behavior is viewing the signage display SD. Therefore, the user can grasp the behavior of the person J who is interested in the signage display SD (more specifically, the person J who is interested in the information displayed on the signage display SD).

[0067] For example, the information management device 10 identifies the position of person J at the time when person J looks at the signage display SD as the relevant position. Therefore, the user can know from what position the signage display SD is being viewed.

[0068] Furthermore, for example, the information management device 10 further identifies the trajectory of the position of person J in the period before and after person J views the signage display SD. Thus, the user can know how person J moves before and after viewing the signage display SD.

[0069] Furthermore, for example, when person J looks at the signage display SD, the information management device 10 identifies the position on the signage display SD at which the person J is looking as the relevant position, thereby enabling the user to know which part of the signage display SD is being looked at.

[0070] Furthermore, for example, the information management device 10 performs image processing on a floor image PM2 captured from a direction perpendicular to the floor model FM of the three-dimensional space model MS, and then displays the image. This allows the user to accurately grasp the planar distribution of related positions within the station premises where the signage displays SD are installed.

[0071] Furthermore, for example, the information management device 10 performs image processing on a front image PM3 captured from a direction perpendicular to a display surface DP2 on the three-dimensional space model MS corresponding to the display surface DP1 of the signage display SD, and then displays the image. This allows the user to accurately grasp the planar distribution of related positions on the display surface DP1 of the signage display SD.

[0072] B: Modifications The following are modifications of the above-described embodiment. Two or more modifications selected from the following modifications may be combined as appropriate within the scope of not contradicting each other.

[0073] B1: First Modification In the above-described embodiment, the target object is a signage display SD installed in a station. The target object is not limited to this, and for example, the store itself may be the target object. Specifically, the target object may be, for example, a store's show window facing the road, a window through which the interior of the store can be seen, or the exterior of the store. In this case, for example, the store manager can obtain information such as whether person J who sees the store from the road actually enters the store, which parts of the show window are most frequently viewed, etc.

[0074] B2: Second Modification In the above-described embodiment, the target object is a signage display SD, and the image displayed thereon changes over time. However, the target object is not limited to this, and may be, for example, a poster printed on paper or a medium that displays information that does not change over time, such as a display in a shop window.

[0075] C: Others (1) In the above-described embodiment, ROM, RAM, etc. are exemplified as storage device 130, but storage device 130 may also be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or other suitable storage medium.

[0076] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0077] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0078] (4) In the above-described embodiment, the determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a comparison of numerical values ​​(e.g., a comparison with a predetermined value).

[0079] (5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0080] (6) Each function illustrated in FIG. 2 is realized by any combination of hardware and / or software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (e.g., wired, wireless, etc.) and these multiple devices. A functional block may be realized by combining software with the single device or the multiple devices.

[0081] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0082] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0083] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0084] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0085] (10) In the above-described embodiments, the portable device may be a mobile station (MS). Those skilled in the art may also refer to a mobile station as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other appropriate terminology. In this disclosure, the terms "mobile station," "user terminal," "user equipment (UE)," "terminal," etc. may be used interchangeably.

[0086] (11) In the above-described embodiments, the terms "connected," "coupled," or any variation thereof refers to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0087] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0088] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining something as "determining" or "determining," and the like. Furthermore, "judgment" and "decision" may include regarding receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and accessing (e.g., accessing data in memory) as having been "judgment" or "decision." Furthermore, "judgment" and "decision" may include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judgment" or "decision." In other words, "judgment" and "decision" may include regarding some action as having been "judgment" or "decision." Furthermore, "judgment" may be interpreted as "assuming," "expecting," "considering," etc.

[0089] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or," as used in this disclosure, is not intended to be an exclusive or.

[0090] (15) In this disclosure, when articles are added by translation, such as a, an, and the in English, the disclosure may include the noun following these articles being plural.

[0091] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0092] (17) The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to being explicit, but may be implicit (e.g., not notifying the predetermined information).

[0093] 1...system, 10...information management device, 20...imaging device, 100...input device, 110...display, 120...communication device, 130...storage device, 140...processing device, 141...acquisition unit, 142...person model generation unit, 143...placement unit, 144...identification unit, 145...display control unit, BM (BM1 to BM4)...skeletal model, J (J1 to J4)...person, MS...three-dimensional space model, N...network, RS...real space, SD...signage display.

Claims

1. A display control device comprising: an arrangement unit that arranges, based on an image of the periphery of an object, a person image relating to the behavior of at least one person located around the object on a three-dimensional model that reproduces the periphery of the object; an identification unit that, when the at least one person performs a related behavior related to the object, identifies a related position relating to the related behavior; and a display control unit that displays a two-dimensional image of the three-dimensional model after performing image processing on the portion corresponding to the related position.

2. A display control device as described in claim 1, wherein the related behavior is looking at the object, and the identification unit identifies the position of the at least one person at the time the at least one person looks at the object as the related position.

3. The display control device according to claim 2, wherein the identification unit further identifies a trajectory of the position of the at least one person during a period including the time when the person looked at the object.

4. A display control device according to claim 1, wherein the related behavior is looking at the object, and the identification unit identifies a portion of interest on the object that is looked at by the at least one person as the related position.

5. The display control device according to claim 1, wherein the display control unit displays the two-dimensional image captured from a direction perpendicular to the floor surface of the three-dimensional model.

6. The display control device according to claim 1, wherein a reference plane is set for the object, and the display control unit displays the two-dimensional image captured from a direction perpendicular to the reference plane.

7. The display control device according to claim 1, wherein the display control unit performs the image processing by adding an image indicating the relevant position.

8. A display control method comprising: arranging, on a three-dimensional model reproducing the surroundings of an object, a person image relating to the behavior of at least one person positioned around the object based on an image of the surroundings of the object; when the at least one person performs a related behavior related to the object, identifying a related position relating to the related behavior; and displaying a two-dimensional image of the three-dimensional model after performing image processing on the portion corresponding to the related position.

Images