Art appreciation device, art appreciation method, and program
The art appreciation device mimics real-world interaction by using viewer head movements to control the display of a virtual art object, addressing the lack of intuitiveness in existing technologies and providing a similar experience to viewing a real work of art.
Patent Information
- Application Number
- JP2023073346
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-04-27
- Publication Date
- 2024-11-08
AI Technical Summary
Existing art appreciation technologies, such as those described in Non-Patent Document 1, fail to provide an intuitive understanding of how the reflection state of a highly reflective art object changes with viewer interaction, as they rely on touch operations on a touch panel display rather than head movements, which are necessary for real-world viewing.
An art appreciation device and method that utilizes a virtual art object model displayed on a display in conjunction with a virtual camera and light source in a virtual space, capturing viewer head movements through a camera to control the display of a virtual art object, mimicking real-world interaction.
Provides an experience similar to viewing a real work of art by allowing intuitive changes in the reflection state of the virtual art object based on viewer head movements, enhancing the appreciation experience.
Smart Images

Figure 2024158275000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an art appreciation device, an art appreciation method, and a program. [Background technology]
[0002] Conventionally, there is known an art appreciation device that displays 3D CG (Computer Graphics) of real swords on a large touch panel display and allows the user to appreciate them (for example, see Non-Patent Document 1 below). In the art appreciation device described in Non-Patent Document 1 below, the viewer can rotate, enlarge, reduce, etc. the displayed 3D CG by performing touch operations on the touch panel display. [Prior art documents] [Non-patent literature]
[0003] [Non-Patent Document 1] "About the National Treasure "Tachi Mumei Ichimonji (Yamadorige)" - Setouchi City Official Website", [online], [searched on April 17, 2023], Internet<URL:https: / / www.city.setouchi.lg.jp / site / token / 109319.html> Summary of the Invention [Problem to be solved by the invention]
[0004] When appreciating an actual work of art such as a sword in a museum or the like, the viewer moves his / her head to peer into the work of art housed in a glass case, etc., and sees each part of the work of art. In particular, when the work of art to be appreciated is highly reflective, the viewer generally moves his / her head to check how the reflective state of the work of art changes. However, in the technology of Non-Patent Document 1, the action for changing the reflective state of the artwork existing in the CG space is a touch operation on a touch panel display, which is different from the action for changing the reflective state of an artwork that actually exists. Therefore, with the technology of Non-Patent Document 1, it is difficult to intuitively grasp how the reflective state of the artwork changes with each operation, and it is difficult to intuitively confirm the reflective state.
[0005] In view of the above-mentioned problems, an object of the present invention is to provide an art appreciation device, an art appreciation method, and a program that can provide an experience similar to that of appreciating a real work of art. [Means for solving the problem]
[0006] In order to solve the above-mentioned problems, one embodiment of the art appreciation device of the present invention is an art appreciation device that includes an image processing unit that displays a virtual space image of the virtual art model captured by a virtual camera in a virtual space in which a virtual art model, a virtual light source, and a virtual camera are disposed, on a display disposed in a real space, an acquisition unit that acquires an image of an area in front of the display captured by the camera disposed in the real space, and a display control unit that performs processing to control the display of the virtual art model on the display based on movement information that indicates changes in the movement of a person present in front of the display, which is calculated based on the acquired image.
[0007] An art appreciation method according to one embodiment of the present invention is an art appreciation method executed by a computer, the method including: an image processing step of displaying, on a display arranged in a real space, a virtual space image of the virtual art model captured by a virtual camera in a virtual space in which a virtual art model, a virtual light source, and a virtual camera are disposed; an acquisition step of acquiring an image of an area in front of the display captured by the camera arranged in the real space; and a display control step of performing processing to control the display of the virtual art model on the display based on movement information indicating changes in the movement of a person present in front of the display, the image information being calculated based on the acquired image.
[0008] A program according to one aspect of the present invention is a program for causing a computer to function as an image processing means for displaying, on a display arranged in a real space, a virtual space image of the virtual artwork model captured by a virtual camera in a virtual space in which a virtual artwork model, a virtual light source, and a virtual camera are disposed, an acquisition means for acquiring an image captured in front of the display by the camera arranged in the real space, and a display control means for performing processing to control the display of the virtual artwork model on the display based on movement information indicating changes in the movement of a person present in front of the display, which is calculated based on the acquired image. Effect of the Invention
[0009] According to the present invention, it is possible to provide an experience similar to that of appreciating a real work of art. [Brief description of the drawings]
[0010] [Figure 1] 1 is a diagram illustrating an example of the configuration of an art appreciation system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of an imaging range of a camera arranged in a real space according to the present embodiment. [Diagram 3] FIG. 2 is a diagram showing an example of an imaging range of a camera arranged in a real space according to the present embodiment. [Figure 4] 1 is a block diagram showing an example of a functional configuration of an art appreciation device according to an embodiment of the present invention. [Diagram 5] 1 is a diagram showing an example of a virtual space in which a virtual artwork model, a virtual light source, and a virtual camera according to an embodiment of the present invention are arranged. [Figure 6] FIG. 2 is a diagram showing an example of a captured image according to the embodiment; [Figure 7] 1 is a diagram showing an example of the position of the head of a viewer existing in real space according to the present embodiment. FIG. [Figure 8] 11A to 11C are diagrams illustrating an example of control for changing the position of a virtual camera arranged in a virtual space according to the present embodiment. [Figure 9] 11A to 11C are diagrams illustrating an example of control for changing the position of a virtual camera arranged in a virtual space according to the present embodiment. [Figure 10] FIG. 4 is a sequence diagram showing an example of a processing flow according to the embodiment. [Figure 11] 1 is a block diagram showing an example of the functional configuration of an art appreciation device in a first modified example of the present embodiment. FIG. [Figure 12] FIG. 11 is a block diagram showing an example of the functional configuration of an art appreciation device in a second modified example of the present embodiment. [Figure 13] FIG. 13 is a block diagram showing an example of the functional configuration of an art appreciation device in a third modified example of the present embodiment. [Figure 14] FIG. 13 is a block diagram showing an example of the functional configuration of an art appreciation device in a fourth modified example of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In addition, in the drawings, mutually orthogonal X-axis, Y-axis, and Z-axis are shown as necessary. On each axis, the direction in which the arrow extends is called the "positive direction," and the direction opposite to the positive direction is called the "negative direction."
[0012] <1. Configuration of the Art Appreciation System> The configuration of an art appreciation system according to this embodiment will be described with reference to Figures 1 to 3. Figure 1 is a diagram showing an example of the configuration of an art appreciation system according to this embodiment. The art appreciation system 1 shown in FIG. 1 is a system that displays a virtual art model arranged in a virtual space on a display arranged in real space in a facility (such as a museum or art gallery) where art and the like are exhibited for the purpose of appreciation. In addition to the virtual art model, a virtual camera and a virtual light source are arranged in the virtual space. The art appreciation system 1 changes the appearance (e.g., the reflection state) of the virtual art model displayed on the display in conjunction with the head position, etc., of a person who has visited the facility and is appreciating the virtual art model (hereinafter also referred to as the "appreciation person"). In this way, the art appreciation system 1 enables the appreciator to freely appreciate the virtual art model.
[0013] In the following, an embodiment will be described taking as an example an example where the artwork is a sword and the virtual artwork model to be appreciated is a virtual sword model 22 in which a sword is reproduced by 3D CG (Computer Graphics). Note that the object to be appreciated is not limited to the virtual sword model 22. For example, the object to be appreciated may be a 3D model (virtual artwork model) that imitates an actual artwork other than a sword, such as metalware, pottery, or lacquerware. As shown in FIG. 1, the art appreciation system 1 includes an art appreciation device 10, a display 20, and a camera 30.
[0014] (1) Art appreciation device 10 The art appreciation device 10 is a device that changes the appearance of a virtual art model displayed on a display in conjunction with the position of the head of an observer (an example of a human being). The art appreciation device 10 is, for example, a PC (Personal Computer), a server device, or the like. The art appreciation device 10 is communicatively connected to a display 20 and a camera 30 via a wired or wireless connection.
[0015] (2) Display 20 The display 20 is a display device provided in real space that displays an image showing the state of the virtual space (hereinafter also referred to as a "virtual space image") in accordance with an image signal from the art appreciation device 10. As the display 20, for example, a liquid crystal monitor is used. 1, the display 20 displays, for example, a virtual space image showing the appearance of a virtual sword model 22 placed in a virtual space. The virtual space image is an image of the virtual sword model 22 placed in the virtual space captured by a virtual camera placed in the virtual space.
[0016] (3) Camera 30 The camera 30 is an imaging device provided in real space so as to be able to capture an image in front of the display 20. Therefore, the camera 30 generates a captured image capturing an image of the area in front of the display 20. If a viewer is present in front of the display 20, the camera 30 generates a captured image capturing an image of the viewer. The generation of captured images is repeated at predetermined time intervals (e.g., 10 msec.). The camera 30 transmits the generated captured image to the art appreciation device 10.
[0017] Here, the imaging range of the camera 30 will be described with reference to Fig. 2 and Fig. 3. Fig. 2 and Fig. 3 are diagrams showing an example of the imaging range of a camera arranged in real space according to this embodiment. Fig. 2 is a diagram showing the display 20 and the camera 30 viewed from the negative direction to the positive direction of the Z1 axis. That is, Fig. 2 is a diagram showing the display 20 and the camera 30 viewed from the front of the display 20. Fig. 3 is a diagram showing the display 20 and the camera 30 viewed from the positive direction to the negative direction of the X1 axis. That is, Fig. 3 is a diagram showing the display 20 and the camera 30 viewed from the side of the display 20.
[0018] As shown in Fig. 2 and Fig. 3, the imaging range 50 of the camera 30 is set to include the range in which the viewer's head is located. In the example shown in Fig. 2 and Fig. 3, the viewer is located in front of the display 20 and at a predetermined distance L (e.g., 1 m) from the display surface S0 of the display 20 in the normal direction of the display surface S0. A plane S1 is defined as a plane parallel to the display surface S0 at the position of the viewer. The imaging range 50 of the camera 30 is defined as a specific range within the plane S1 (e.g., a rectangular range centered on the intersection with the optical axis of the camera 30). The angle α is the horizontal angle of view of the camera 30 (hereinafter also referred to as the "horizontal angle of view α"), and the angle β is the vertical angle of view of the camera 30 (hereinafter also referred to as the "vertical angle of view β").
[0019] <2. Functional configuration of the art appreciation device> The configuration of the art appreciation system 1 according to this embodiment has been described above. Next, the functional configuration of the art appreciation device 10 according to this embodiment will be described with reference to Fig. 4 to Fig. 9. Fig. 4 is a block diagram showing an example of the functional configuration of the art appreciation device 10 according to this embodiment. As shown in FIG. 4, art appreciation device 10 includes a communication unit 110, a storage unit 120, and a control unit .
[0020] (1) Communications Department 110 The communication unit 110 has a function of transmitting and receiving various information. For example, the communication unit 110 transmits a virtual space image to the display 20. The communication unit 110 also receives a captured image transmitted from the camera 30.
[0021] (2) Storage section 120 The storage unit 120 has a function of storing various information. The storage unit 120 is configured by a storage medium provided as hardware in the art appreciation device 10, such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, an electrically erasable programmable read only memory (EEPROM), a random access read / write memory (RAM), a read only memory (ROM), or any combination of these storage media. The storage unit 120 stores, for example, captured images and virtual space images.
[0022] (3) Control unit 130 The control unit 130 has a function of controlling the overall operation of the art appreciation device 10. The control unit 130 is realized, for example, by causing a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) that the art appreciation device 10 has as hardware to execute a program. As shown in FIG. 4, the control unit 130 includes an image processing unit 131, an acquisition unit 132, a position calculation unit 133, and a display control unit 134.
[0023] (3-1) Image processing unit 131 The image processing unit 131 has a function of performing image processing. The image processing unit 131 performs image processing for displaying a virtual space image on the display 20 arranged in the real space. The image processing unit 131 places a virtual sword model, a virtual light source, and a virtual camera in the virtual space. The virtual sword model is a three-dimensional model that imitates a real sword such as a Japanese sword to be appreciated. The virtual sword model is created based on high-definition image data generated by imaging a real sword, for example. As an example, a flat three-dimensional model imitating the planar shape of a sword is prepared in advance as a virtual sword model, with thickness information omitted. The virtual sword model may be a three-dimensional model in which image data of the front and back of a real sword are attached as texture images to the front and back of the three-dimensional model, respectively. In this case, the image processing unit 131 calculates the reflection of light and shadows caused by the virtual light source on the front and back of the virtual sword model using a normal map.
[0024] Image processing unit 131 generates a virtual space image by having a virtual camera placed in a virtual space capture an image of the virtual sword model. Furthermore, when the position of the virtual camera is controlled by display control unit 134 (described later), image processing unit 131 generates a virtual space image in which the appearance (e.g., reflection state) of the virtual sword model has changed by having the virtual camera capture the image of the virtual sword model at a position after the control. After generating the virtual space image, the image processing unit 131 transmits the generated virtual space image from the communication unit 110 to the display 20, and causes the virtual space image to be displayed on the display 20. This allows the viewer to confirm the change in the appearance of the virtual sword model.
[0025] Here, a virtual space in which a virtual artwork model, a virtual light source, and a virtual camera are arranged will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of a virtual space in which a virtual artwork model, a virtual light source, and a virtual camera according to this embodiment are arranged.
[0026] As shown in FIG. 5, the image processing unit 131 places a virtual sword model 22, a virtual light source 23, and a virtual camera 24 in the virtual space 21. The virtual sword model 22 and the virtual camera 24 in the virtual space 21 are located in a plane S2 parallel to the X2Y2 plane of the virtual space 21 in the orthogonal coordinate system (X2, Y2, Z2) of the virtual space 21. The virtual camera 24 is located at a predetermined distance L' from the plane S2 in the normal direction of the plane S2, and is located in a virtual plane S3 parallel to the plane S2. The virtual camera 24 is movable within the plane S3. Here, the predetermined distance L' is the same as the predetermined distance L between the display surface S0 and the plane S1 of the display 20. The plane S3 of the virtual space 21 is a plane corresponding to the plane S1 (the plane on which the viewer's head moves) of the real space (actual three-dimensional space) as described later. In the virtual space 21, a virtual pedestal model 25, which is a three-dimensional model of the pedestal of the sword, is located below the virtual sword model 22. 5 shows an example in which a frustum-shaped three-dimensional model is used as virtual base model 25. Also, light from virtual light source 23 strikes reflection position 28 on virtual sword model 22 and is reflected. 5, with the virtual space 21 prepared, the image processing unit 131 captures the virtual sword model 22 (a flat three-dimensional model, i.e., a two-dimensional model) and the virtual pedestal model 25 (a frustum-shaped three-dimensional model) with the virtual camera 24, and generates a virtual space image. Then, the image processing unit 131 transmits the generated virtual space image (an image showing the virtual sword model 22 and the virtual pedestal model 25) from the communication unit 110 to the display 20, and causes it to be displayed on the display 20.
[0027] (3-2) Acquisition part 132 The acquisition unit 132 has a function of acquiring a captured image. For example, when the communication unit 110 receives a captured image from the camera 30, the acquisition unit 132 acquires the captured image.
[0028] (3-3) Position calculation unit 133 The position calculation unit 133 has a function of calculating the position of the viewer's head. For example, the position calculation unit 133 calculates the position (A1, B1, C1) of the head of the viewer in front of the display 20 as movement information based on the captured image acquired by the acquisition unit 132. The movement information is information indicating a change in the movement of the viewer in front of the display 20, and is information calculated based on the captured image acquired by the acquisition unit 132.
[0029] Here, a method for calculating the head position of a viewer existing in real space will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a diagram showing an example of a captured image according to this embodiment. Fig. 7 is a diagram showing an example of the head position of a viewer existing in real space according to this embodiment.
[0030] First, the position calculation unit 133 uses an image processing technique to calculate the position (a, b) of the head in the Cartesian coordinate system (x, y) of the captured image 26 based on the captured image 26 as shown in Fig. 6. Next, the position calculation unit 133 calculates the position (A1, B1, C1) of the head in the Cartesian coordinate system (X1, Y1, Z1) of the real space 27 as shown in Fig. 7 based on the calculated position (a, b) of the head. For example, the position calculation unit 133 calculates the position (A1, B1, C1) of the head based on the horizontal angle of view α and vertical angle of view β of the camera 30, and a predetermined distance L from the camera 30 to the plane S1 (see Figs. 2 and 3). Specifically, the position calculation unit 133 assumes that there is an image 26 having a size (for example, a size of 2Ltan (α / 2) in the left-right direction and 2Ltan (β / 2) in the up-down direction) of the imaging range 50 of the camera 30 on a plane S1 of the real space 27 (see Figures 2 and 3), and calculates the position (A1, B1, C1) of the head within the image 26.
[0031] (3-4) Display control unit 134 The display control unit 134 has a function of performing processing for controlling the display of the virtual sword model 22 on the display 20. The display control unit 134 performs processing for controlling the display of the virtual sword model 22 on the display 20, based on operation information calculated based on the captured image acquired by the acquisition unit 132. For example, the display control unit 134 controls the position of the virtual camera 24 in the virtual space 21 based on the position (A1, B1, C1) of the viewer's head calculated by the position calculation unit 133. Specifically, the display control unit 134 changes the position of the virtual camera 24 to a position corresponding to the position of the head calculated by the position calculation unit 133 so that the position of the viewer's head in the real space 27 corresponds to the position of the virtual camera 24 in the virtual space 21. As a result, the image processor 131 regenerates a virtual space image in the virtual space 21 in which the virtual camera 24 has been moved according to the position (A1, B1, C1) of the viewer's head, and updates the display content of the display 20. Such updating of the display content is repeated every predetermined time (for example, 10 msec.).
[0032] Here, control for changing the position of virtual camera 24 arranged in virtual space 21 will be described with reference to Fig. 8 and Fig. 9. Fig. 8 and Fig. 9 are diagrams showing an example of control for changing the position of the virtual camera arranged in the virtual space according to this embodiment.
[0033] It is assumed that the virtual space 21 and the real space 27 are continuous. The display control unit 134 calculates the position (A2, B2, C2) of the viewer's head in the orthogonal coordinate system (X2, Y2, Z2) of the virtual space 21 shown in Fig. 8 based on the position (A1, B1, C1) of the viewer's head in the orthogonal coordinate system (X1, Y1, Z1) of the real space 27 calculated by the position calculation unit 133. The display control unit 134 calculates the position (A2, B2, C2) of the head by assuming that the coordinate value in the Z2 axis direction of the virtual sword model 22 in the orthogonal coordinate system (X2, Y2, Z2) of the virtual space 21 is located on the display surface S0 of the display 20 in the real space 27, for example. That is, the display control unit 134 calculates the position of the viewer's head (A2, B2, C2) assuming that the imaging range 60 of the virtual camera 24 on the plane S2 of the virtual space 21 coincides with the display surface S0 of the display 20 in the real space 27. 9, the display control unit 134 moves the position of the virtual camera 24 to the calculated head position (A2, B2, C2) in the virtual space 21. Here, the plane S1 of the real space 27 (the plane on which the viewer's head moves) corresponds to the plane S3 of the virtual space 21. Therefore, the movement of the virtual camera 24 is performed within the plane S3.
[0034] Furthermore, the display control unit 134 changes the angle of the virtual camera 24 so that the virtual sword model 22 is located at the center of the imaging range 60 of the virtual camera 24. For example, assume that the viewer moves to the left from a state in which he or she faces the center of the display 20 in the real space 27, and the viewer's head moves to the left. In this case, the display control unit 134 moves the virtual camera 24 to the left in the plane S3 from a state in which the virtual camera 24 faces the virtual sword model 22 in the virtual space 21, and turns the virtual camera 24 to the right. As a result, in the virtual sword model 22 displayed on the display 20, the reflection position 28 where the light of the virtual light source 23 hits is moved to the left. Therefore, the viewer can confirm how the reflection state of the virtual sword model 22 (virtual artwork model) changes by moving his or her head in the same way as when viewing a real sword.
[0035] <3. Processing flow> The functional configuration of the art appreciation device 10 according to this embodiment has been described above. Next, the flow of processing according to this embodiment will be described with reference to Fig. 10. Fig. 10 is a sequence diagram showing an example of the flow of processing according to this embodiment. Below, an example will be described in which the viewer checks the virtual sword model initially displayed on the display 20.
[0036] As shown in FIG. 10, first, the image processing unit 131 of the art appreciation device 10 generates a virtual space image to be displayed as an initial display (virtual space image for initial display) (step S101). After generation, image processing unit 131 transmits the generated initial display virtual space image from communication unit 110 to display 20, and causes display 20 to display the initial display virtual space image (step S102). The display 20 receives the initial display virtual space image from the art appreciation device 10 and displays it (step S103).
[0037] The viewer checks the initially displayed virtual sword model from the initially displayed virtual space image displayed on the display 20 (step S104). The viewer can change the confirmation position at any time by moving his / her standing position or moving his / her head (step S105).
[0038] The camera 30 captures an image in front of the display 20 and generates a captured image (step S106). After generation, the camera 30 transmits the generated captured image to the art appreciation device 10 (step S107). The camera 30 captures an image in front of the display 20 at predetermined time intervals, generates a captured image, and transmits the generated captured image to the art appreciation device 10. For ease of explanation, the example shown in Fig. 10 only shows a part in which the camera 30 captures an image after the viewer changes his / her confirmation position, generates a captured image, and transmits the generated captured image to the art appreciation device 10.
[0039] When the communication unit 110 receives the captured image from the camera 30, the acquisition unit 132 of the art appreciation device 10 acquires the captured image (step S108). Next, the position calculation section 133 of the art appreciation device 10 calculates the position of the head of the viewer present in front of the display 20, based on the captured image acquired by the acquisition section 132 (step S109). Next, the display control unit 134 of the art appreciation device 10 controls the position of the virtual camera 24 in the virtual space 21 based on the viewer's head position calculated by the position calculation unit 133, thereby controlling the display of the virtual sword model 22 (step S110). In the example shown in Fig. 10, the position of the viewer's head is calculated based on the captured image captured after the viewer's confirmation position is changed. Therefore, the display control unit 134 moves the position of the virtual camera 24 in accordance with the movement of the viewer's head position. Next, image processing unit 131 generates an update virtual space image based on the position of virtual camera 24 moved by display control unit 134 (step S111). After generation, image processing unit 131 transmits the generated update virtual space image from communication unit 110 to display 20, and causes display 20 to update the display of the virtual space image (step S112).
[0040] Display 20 receives the update virtual space image from art appreciation device 10 and updates the display of the virtual space image (step S113). The viewer checks the updated virtual sword model from the update virtual space image displayed on the display 20 (step S114).
[0041] As described above, the art appreciation device 10 of this embodiment comprises an image processing unit 131 that displays, on the display 20 placed in the real space 27, a virtual space image of the virtual sword model 22 captured by the virtual camera 24 in the virtual space 21 in which the virtual sword model 22 (virtual artwork model), virtual light source 23, and virtual camera 24 are placed; an acquisition unit 132 that acquires an image captured in front of the display 20 by the camera 30 placed in the real space 27; a position calculation unit 133 that calculates the head position of the viewer (human) present in front of the display 20 based on the acquired image; and a display control unit 134 that controls the position of the virtual camera 24 in the virtual space 21 based on the calculated head position.
[0042] With this configuration, the art appreciation device 10 can change the appearance of the virtual sword model 22 displayed on the display 20 when the viewer moves his or her head. Therefore, the art appreciation device 10 can provide the viewer with an experience similar to that of appreciating a real sword (art work).
[0043] Furthermore, unlike conventional methods in which a virtual sword model is displayed on a touch panel display and the viewer operates it by touch, art appreciation device 10 allows the position of the head to be used as input for art appreciation device 10. For this reason, art appreciation device 10 can prevent the viewer from touching the input device, which is preferable from the standpoint of preventing virus infection, etc. Furthermore, the art appreciation device 10 can prevent operations by the viewer from interfering with the appreciation of the virtual sword model 22.
[0044] <4. Modifications> The above describes the embodiments of the present invention. Next, modified examples of the above-described embodiments will be described. Each modified example described below may be applied to the embodiment alone or in combination with the embodiment. Each modified example may be applied in place of the configuration described in the embodiment, or may be applied in addition to the configuration described in the embodiment.
[0045] (1) First Modification In the above embodiment, an example was described in which the virtual sword model 22 is a flat three-dimensional model that omits thickness information and imitates the planar shape of a sword, and image data of the front and back of an actual sword is pasted as texture images on the front and back of the three-dimensional model, respectively, but the present invention is not limited to such an example. For example, the virtual sword model 22 may be a three-dimensional model that imitates the three-dimensional shape (concave and convex shape) of a sword such as an actual Japanese sword.
[0046] Here, the functional configuration of the art work appreciation device 10 in the first modified example will be described with reference to Fig. 11. Fig. 11 is a block diagram showing an example of the functional configuration of the art work appreciation device 10 in the first modified example according to the present embodiment. As shown in FIG. 11, the control unit 130 of the art appreciation device 10 in the first variant example further includes an angle calculation unit 135 in addition to the image processing unit 131, acquisition unit 132, position calculation unit 133, and display control unit 134 shown in FIG.
[0047] The angle calculation unit 135 has a function of calculating the face direction of a viewer (human being). For example, the angle calculation unit 135 calculates the face direction of the viewer present in front of the display 20 as motion information based on the captured image acquired by the acquisition unit 132. The face direction is, for example, the face direction in a motion of shaking the head left and right, and includes the angle of the direction in which the face is facing (left and right direction) with respect to the front direction of the body.
[0048] The display control unit 134 controls the orientation of the virtual camera 24 in the virtual space 21 based on the orientation of the face calculated by the angle calculation unit 135. Specifically, the display control unit 134 changes the orientation (angle) of the virtual camera 24 so that the orientation of the viewer's face in the real space 27 corresponds to the orientation of the virtual camera 24 in the virtual space 21, and the orientation of the virtual camera 24 corresponds to the orientation of the face calculated by the angle calculation unit 135. As an example, it is assumed that the viewer is trying to see the thickness of the virtual sword model 22. When the viewer is standing on the tip side of the virtual sword model 22, the viewer turns his / her face to the tang end side of the virtual sword model 22 so as to look from the tip side of the virtual sword model 22 to the tang end side of the virtual sword model 22. Alternatively, when the viewer is standing on the tang end side of the virtual sword model 22, the viewer turns his / her face to the tip side of the virtual sword model 22 so as to look from the tang end side of the virtual sword model 22 to the tip side. In these cases, the display control unit 134 causes the angle of the virtual camera 24 with respect to the plane S2 on which the virtual sword model 22 is located to match the orientation of the face with respect to the display surface S0 of the display 20 (the angle of the direction the face is facing).
[0049] (2) Second modified example In the above embodiment, an example has been described in which the position of the virtual camera 24 is changed in response to the movement of the viewer. This is because, by changing the position of the virtual camera 24, the line of sight vector in the virtual space 21 used to calculate the reflection of light and shadows in the virtual sword model 22 can be changed in response to the movement of the viewer. This is not limited to the example in which the position of the virtual camera 24 is changed, and the line of sight vector in the virtual space 21 used to calculate the reflection of light and shadows in the virtual sword model 22 may be changed without changing the position of the virtual camera 24 in response to the movement of the viewer.
[0050] In the second modified example, art appreciation device 10 calculates the viewer's line of sight vector based on an image captured by camera 30 placed in real space 27, applies the calculated line of sight vector to a line of sight vector used to calculate reflections and shading by a shader, and determines the light reflections and shading on virtual sword model 22. In this way, art appreciation device 10 can change the light reflection on virtual sword model 22 in response to changes in the viewer's movement without changing the position of virtual camera 24 in response to changes in the viewer's movement.
[0051] Here, the functional configuration of the art work appreciation device 10 in the second modified example will be described with reference to Fig. 12. Fig. 12 is a block diagram showing an example of the functional configuration of the art work appreciation device 10 in the second modified example according to the present embodiment. As shown in FIG. 12, the control unit 130 of the art appreciation device 10 in the second variant example further includes a gaze vector calculation unit 136 and a reflected shadow calculation unit 137 in addition to the image processing unit 131, acquisition unit 132, and display control unit 134 shown in FIG. 4.
[0052] Based on the captured image acquired by the acquisition section 132, the line-of-sight vector calculation section 136 calculates the line-of-sight vector V1 of the viewer as movement information. The reflection shadow calculation unit 137 calculates reflection shadow information related to reflections and shadows in the virtual sword model 22 based on the viewer's line of sight vector V1 calculated by the line of sight vector calculation unit 136. Specifically, the reflection shadow calculation unit 137 performs shader calculation using the viewer's line of sight vector V1 as the line of sight vector V for shader calculation, and calculates the reflection shadow information. The display control unit 134 controls the reflection position of light on the virtual sword model 22 based on the reflection shadow information calculated based on the line of sight vector V1, which is the operation information. Specifically, the display control unit 134 causes the image processing unit 131 to generate a virtual space image in which the reflection position 28 has been changed without changing the display (position, orientation, etc.) of the virtual sword model 22, and causes the display 20 to display the generated virtual space image.
[0053] (3) Third modified example The above-described embodiment and the above-described second modified example may be combined, which makes it possible to control with a higher degree of freedom how the virtual sword model 22 appears in response to the movement of the viewer.
[0054] In the third modified example, the art appreciation device 10 calculates the viewer's line of sight vector V1 based on the captured image captured by the camera 30 arranged in the real space 27. The art appreciation device 10 adds the calculated viewer's line of sight vector V1 to a line of sight vector V2 depending on the position of the virtual camera 24, and uses the result as the line of sight vector V in the shader calculation. When adding, the art appreciation device 10 may adjust the amount of movement of the light reflection position 28 in the virtual sword model 22 by multiplying the line of sight vector V1 and the line of sight vector V2 by a control parameter prepared for each line of sight vector. For example, the art appreciation device 10 can adjust the amount of movement of the light reflection position 28 in the virtual sword model 22 to be larger than the amount of actual movement of the viewer. Therefore, the viewer can move the reflection position 28 by a larger amount with a smaller amount of movement. This allows the viewer to see almost all the light reflections with minimal movement when there is not enough space for the viewer to move around widely in the real space 27 in which the display 20 is placed. Also, the viewer can view the display without having to make any significant bending or stretching movements.
[0055] Here, the functional configuration of the art work appreciation device 10 in the third modified example will be described with reference to Fig. 13. Fig. 13 is a block diagram showing an example of the functional configuration of the art work appreciation device 10 in the third modified example according to the present embodiment. As shown in FIG. 13, the control unit 130 of the art appreciation device 10 in the third modified example includes the image processing unit 131 shown in FIG. 11, an acquisition unit 132, a position calculation unit 133, a display control unit 134, an angle calculation unit 135, the line of sight vector calculation unit 136 shown in FIG. 12, and a reflected shadow calculation unit 137.
[0056] Line-of-sight vector calculation section 136 calculates line-of-sight vector V2 depending on the position of virtual camera 24 in virtual space 21, in addition to line-of-sight vector V1 of the viewer. Based on the control parameters set for each of the viewer's view vector V1 calculated by the view vector calculation unit 136 and the view vector V2 depending on the position of the virtual camera 24, the reflection shadow calculation unit 137 calculates a view vector V that reflects each view vector according to each control parameter. The control parameters are information indicating to what extent the view vectors V1 and V2 are reflected in the calculation of the view vector V. For example, in order to suppress the amount of change in the display of the virtual sword model 22 while allowing the viewer to confirm changes in the reflection of light with a small amount of movement, the control parameter p1 is set so as to suppress the degree of reflection of the view vector V1, and the control parameter p2 is set so as to increase the degree of reflection of the view vector V2.
[0057] The reflected shadow calculation unit 137 calculates the line-of-sight vector V by the following formula (1): In the following formula (1), p1 is a control parameter for the viewer's line-of-sight vector V1, and p2 is a control parameter for the line-of-sight vector V2 depending on the position of the virtual camera 24. V = V1 p1 + V2 p2 (1) The reflected shadow calculation unit 137 uses the calculated line-of-sight vector V to calculate reflected shadow information in which the viewer's line-of-sight vector V1 and the line-of-sight vector V2 depending on the position of the virtual camera 24 are reflected. The display control unit 134 controls the light reflection position 28 on the virtual sword model 22 based on the reflection and shadow information calculated based on the control parameters.
[0058] (4) Fourth modified example In the above-described embodiment, an example in which the position of virtual camera 24 is changed according to the movement of the viewer has been described, but the present invention is not limited to such an example. For example, at least one of the position and the direction of virtual light source 23 may be changed according to the height of the viewer. However, if the position of virtual light source 23 is simply changed according to the height, when the viewer is short and the vertical width of the object to be viewed is small, such as virtual sword model 22, there are cases in which the reflection does not appear in the object to be viewed. This is because when the viewer is short, the position of virtual light source 23 moves downward, and the position of the reflection also moves in the direction accordingly.
[0059] Therefore, in the fourth modified example, art appreciation device 10 controls virtual light source 23 so that optimal lighting is performed on virtual sword model 22 according to the height of the viewer. Optimal lighting refers to lighting (such as the way virtual light source 23 is aimed and its direction) that makes the object of appreciation (for example, virtual sword model 22) look best in the viewer's line of sight. This prevents a situation where a short viewer has to stretch in order for the light to hit the object properly, or where a tall viewer has to crouch in order for the light to hit the object properly.
[0060] Here, the functional configuration of the art work appreciation device 10 in the fourth modified example will be described with reference to Fig. 14. Fig. 14 is a block diagram showing an example of the functional configuration of the art work appreciation device 10 in the fourth modified example according to the present embodiment. As shown in FIG. 14, the control unit 130 of the art appreciation device 10 in the fourth variant further includes a height calculation unit 138 in addition to the image processing unit 131, acquisition unit 132, position calculation unit 133, display control unit 134, and angle calculation unit 135 shown in FIG. 11.
[0061] The acquisition unit 132 acquires an image captured by, for example, a height calculation camera (for example, an RGB-D camera) separate from the camera 30. The height calculation camera captures an image of a viewer standing upright at a position at a specified distance from the camera in advance. The height calculation unit 138 calculates the height of the viewer based on the captured image acquired from the height calculation camera by the acquisition unit 132. The height calculation unit 138 estimates the positions of the viewer's head and feet by, for example, skeletal estimation based on the captured image captured by the height calculation camera, and calculates the distance between the head and feet as the height. The height of the viewer may be determined based on a physical measurement taken in advance.
[0062] Display control unit 134 controls at least one of the position and orientation of virtual light source 23 in virtual space 21, based on the height of the viewer calculated by height calculation unit 138. When virtual light source 23 is a point light source, it is sufficient to change at least the position of virtual light source 23, and when virtual light source 23 is a parallel light source, it may be necessary to change the orientation of virtual light source 23. Specifically, the display control unit 134 changes at least one of the position and the direction of the virtual light source 23 according to the height of the viewer so that the virtual sword model 22 is lit in the same way even when a plurality of viewers of different heights view the same virtual sword model 22 from the same position. When controlling the virtual light source 23 according to the height of the viewer, information for determining the control amount of the virtual light source 23 is prepared in advance. For example, information in which the height of the viewer is associated with the control amount that provides appropriate lighting according to each height (hereinafter also referred to as "correspondence information") is prepared and stored in the storage unit 120. In this case, the display control unit 134 refers to the correspondence information in the storage unit 120 based on the height of the viewer calculated by the height calculation unit 138, and determines the control amount of the virtual light source 23 according to the height of the viewer.
[0063] (5) Fifth Variation In the above-described embodiment, an example in which the position of virtual camera 24 is changed in response to the movement of the viewer has been described, but the present invention is not limited to such an example. In the above-described fourth modified example, an example in which virtual light source 23 is controlled in response to the height of the viewer has been described, but in the fifth modified example, an example in which virtual light source 23 is controlled in response to the standing position of the viewer (Z1-axis direction: depth direction as viewed from display 20) will be described. If the position and orientation of virtual light source 23 depend on the position of the viewer in the Z1-axis direction, when the viewer moves in the Z1-axis direction, the lighting on virtual sword model 22 changes accordingly. This requires the viewer to search for a position with optimal lighting while moving, which takes time and effort before viewing can begin.
[0064] Therefore, in the fifth variant, art appreciation device 10 controls virtual light source 23 independently of the Z1-axis direction so that optimal lighting is always performed on virtual sword model 22 depending on the viewer's position in the Z1-axis direction. This allows the viewer to reduce the time it takes to search for the position with the best lighting when moving along the Z1 axis, making viewing relatively easy. The functional configuration of the art work appreciation device 10 in the fifth modified example may be any of the configurations of the art work appreciation device 10 in the above-described embodiment and each modified example.
[0065] The acquisition unit 132 acquires, for example, a captured image captured by a standing position calculation camera (for example, an RGB-D camera) separate from the camera 30. Note that the height calculation camera in the fourth modified example may be used as the standing position calculation camera. The position calculation unit 133 calculates the standing position of the viewer present in front of the display 20 as movement information, based on the captured image acquired by the acquisition unit 132 from the standing position calculation camera.
[0066] Display control unit 134 controls at least one of the position and orientation of virtual light source 23 in virtual space 21, based on the standing position of the viewer calculated by position calculation unit 133. When virtual light source 23 is a point light source, it is sufficient to change at least the position of virtual light source 23, and when virtual light source 23 is a parallel light source, it may be necessary to change the orientation of virtual light source 23. Specifically, the display control unit 134 changes at least one of the position and the direction of the virtual light source 23 according to the position of the viewer so that the virtual sword model 22 is always optimally lit even when the position of the viewer changes. When controlling the virtual light source 23 according to the position of the viewer, information for determining the control amount of the virtual light source 23 is prepared in advance, as in the case of the fourth modified example. For example, correspondence information in which the position of the viewer and the control amount that results in appropriate lighting according to each position are associated with each other is prepared and stored in the storage unit 120. In this case, the display control unit 134 refers to the correspondence information in the storage unit 120 based on the position of the viewer calculated by the position calculation unit 133, and determines the control amount of the virtual light source 23 according to the position of the viewer.
[0067] The above describes the modified embodiment of the present invention. Note that a part or the whole of the art appreciation device 10 in the above embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in such a case. Furthermore, the above program may be a program for realizing a part of the above-mentioned function, or may be a program that can realize the above-mentioned function in combination with a program already recorded in the computer system, or may be a program that is realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).
[0068] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]
[0069] 1...artwork appreciation system, 10...artwork appreciation device, 20...display, 21...virtual space, 22...virtual sword model, 23...virtual light source, 24...virtual camera, 25...virtual pedestal model, 26...captured image, 27...real space, 28...reflection position, 30...camera, 50, 60...imaging range, 110...communication unit, 120...storage unit, 130...control unit, 131...image processing unit, 132...acquisition unit, 133...position calculation unit, 134...display control unit, 135...angle calculation unit, 136...gaze vector calculation unit, 137...reflection shadow calculation unit, 138...height calculation unit
Claims
1. an image processing unit that displays a virtual space image of the virtual artwork model captured by the virtual camera in a virtual space in which a virtual artwork model, a virtual light source, and a virtual camera are disposed, on a display disposed in a real space; an acquisition unit that acquires a captured image of an area in front of the display by a camera disposed in the real space; a display control unit that performs processing to control the display of the virtual artwork model on the display based on motion information that indicates a change in motion of a person present in front of the display, the motion information being calculated based on the acquired captured image; and An art appreciation device comprising:
2. a position calculation unit that calculates a position of a head of a person present in front of the display as the motion information based on the acquired captured image; Further equipped with The display control unit controls a position of the virtual camera in the virtual space based on the calculated position of the head.
2. The art appreciation device according to claim 1.
3. the display control unit changes a position of the virtual camera to a position corresponding to the calculated position of the head so that a position of the head of the person in the real space corresponds to a position of the virtual camera in the virtual space; 3. The art appreciation device according to claim 2.
4. an angle calculation unit that calculates, as the motion information, a direction of a face of a person present in front of the display based on the acquired captured image; Further equipped with The display control unit controls a direction of the virtual camera in the virtual space based on the calculated direction of the face.
3. The art appreciation device according to claim 2.
5. the display control unit changes the orientation of the virtual camera to an orientation corresponding to the calculated orientation of the face so that the orientation of the face of the person in the real space corresponds to the orientation of the virtual camera in the virtual space; 5. The art appreciation device according to claim 4.
6. a gaze vector calculation unit that calculates a gaze vector of the person as the motion information based on the captured image; a reflection shadow calculation unit that calculates reflection shadow information regarding reflections and shadows in the virtual artwork model based on the calculated line of sight vector of the person; Further equipped with the display control unit controls a reflection position of light in the virtual artwork model based on the reflection shadow information calculated based on the operation information.
3. The art appreciation device according to claim 1 or 2.
7. the line-of-sight vector calculation unit further calculates a line-of-sight vector depending on a position of the virtual camera in the virtual space, the reflected shadow calculation unit calculates the reflected shadow information reflecting each line-of-sight vector in accordance with each control parameter based on control parameters set for each of the calculated line-of-sight vector of the person and the line-of-sight vector depending on the position of the virtual camera; the display control unit controls a reflection position of light in the virtual artwork model based on the reflection shadow information calculated based on the control parameters.
7. The art appreciation device according to claim 6.
8. a height calculation unit for calculating the height of the person; Further equipped with the display control unit controls at least one of a position and a direction of the virtual light source in the virtual space based on the calculated height of the person.
2. The art appreciation device according to claim 1.
9. the display control unit changes at least one of a position and a direction of the virtual light source in accordance with heights of the people so that the virtual artwork model is lit in the same manner even when a plurality of people of different heights view the same virtual artwork model from the same position; 9. The art appreciation device according to claim 8.
10. the position calculation unit calculates, as the movement information, a standing position of a person present in front of the display based on the acquired captured image; the display control unit controls at least one of a position and a direction of the virtual light source in the virtual space based on the calculated standing position of the person.
3. The art appreciation device according to claim 2.
11. the display control unit changes at least one of a position and a direction of the virtual light source in accordance with a standing position of the person so that the virtual artwork model is always optimally lit even when the standing position of the person changes. The art appreciation device according to claim 10.
12. The virtual artwork model is a virtual sword model.
2. The art appreciation device according to claim 1.
13. an image processing step of displaying a virtual space image of the virtual artwork model captured by the virtual camera in a virtual space in which a virtual artwork model, a virtual light source, and a virtual camera are disposed, on a display disposed in a real space; an acquisition step of acquiring a captured image of an area in front of the display by a camera disposed in the real space; a display control process for performing processing to control display of the virtual artwork model on the display based on motion information indicating a change in motion of a person present in front of the display, the motion information being calculated based on the acquired captured image; A computer-implemented art appreciation method comprising:
14. Computer, an image processing means for displaying, on a display disposed in a real space, a virtual space image of the virtual artwork model captured by the virtual camera in a virtual space in which the virtual artwork model, a virtual light source, and a virtual camera are disposed; an acquisition means for acquiring an image of an area in front of the display by a camera disposed in the real space; a display control means for performing processing to control the display of the virtual artwork model on the display based on motion information indicating a change in motion of a person present in front of the display, the motion information being calculated based on the acquired captured image; A program to function as a
