Display control device, display system, and program

Abstract

To allow a sense of immersion to be obtained by matching a user's viewpoint and a display image, despite that the user does not exist in front of a display device.SOLUTION: A display control device includes: a display control part for controlling the displaying of a display object on a display device; an image acquisition part for acquiring a photographed image of a user's face, that is an image photographed by an imaging device rigidly coupled to the display device; a viewpoint relative position calculation part for calculating a relative position relationship between a user's viewpoint and the display device; a gravity information acquisition part for acquiring information related to acceleration in a gravity direction and an azimuth angle thereof; an initial position information acquisition part for acquiring position information where an own device exists; an acceleration information acquisition part for acquiring acceleration information; and a vector calculation part for calculating an absolute position of a projection plane on which the display device projects an image, calculating absolute position coordinates of a user, and calculating a direction in which the projection plane is directed. The display control part causes modification of three-dimensional information of a display object, on the basis of the information calculated by the vector calculation part.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a display control device, a display system, and a program. [Background technology]

[0002] Conventionally, there has been a technology for displaying three-dimensional information of an object existing in a virtual space on a display device such as a smartphone or a tablet terminal device. By viewing an image displayed by a display device using such a technology, a user can get the sensation of a virtual object being present through the display device. For example, as an example of a device using such a technology, an image processing device capable of synthesizing three-dimensional information of an object in a virtual space with an image of the real world has been disclosed (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-113269 A Summary of the Invention [Problem to be solved by the invention]

[0004] According to such a technique, the three-dimensional information of an object in a virtual space is displayed on the display device on the premise that the user views the display device from the front. However, there are cases where the user's viewpoint remains fixed and only the display device moves, or the angle of the display device changes. In such a case, since the user's line of sight is fixed, the user expects to continue to view the same virtual object through the display device. On the other hand, the control performed by the device is based on the premise that the user is in front, so the image expected by the user and the image actually displayed do not match. Therefore, even if the user views a virtual object through the display device, the user cannot get the feeling that the object is actually present there, and it is difficult to get a sense of immersion in the virtual world. In other words, the conventional technique has a problem that the user's viewpoint does not match the displayed image when the user is not in front of the display device.

[0005] The present invention has been made taking these circumstances into consideration, and its purpose is to provide a display control device, display system, and program that enables the user's viewpoint to match the displayed image, even when the user is not in front of the display device, thereby enabling the user to feel immersed in the virtual world. [Means for solving the problem]

[0006] (1) One aspect of the present invention includes a display control unit that controls the display of a display object having three-dimensional information on a display device; an image acquisition unit that acquires an image captured by an imaging device rigidly connected to the display device, the image capturing an image of a face of a user looking at the display object displayed on the display device; a viewpoint relative position calculation unit that calculates a relative positional relationship between the viewpoint of the user and the display device based on the acquired image; a gravity information acquisition unit that acquires information regarding acceleration in the gravity direction and its azimuth angle; an initial position information acquisition unit that acquires position information of the location of the device when a predetermined point is set as the origin; an acceleration information acquisition unit that acquires acceleration information of the device; and a position information acquired by the initial position information acquisition unit and the acceleration information. and a vector calculation unit that calculates an absolute position of a projection plane onto which the display device projects an image based on acceleration information acquired by an information acquisition unit, calculates absolute position coordinates of the user's viewpoint based on the calculated absolute position of the projection plane and the relative positional relationship between the user's viewpoint and the display device calculated by the viewpoint relative position calculation unit, and calculates a vector of the direction in which the projection plane faces based on the calculated absolute position of the projection plane, gravity information acquired by the gravity information acquisition unit, and the acceleration information acquired by the acceleration information acquisition unit, and the display control device deforms three-dimensional information of the object to be displayed based on the information calculated by the vector calculation unit, and controls the display on the display device. (2) Also, one aspect of the present invention is that in the display control device described above in (1), the viewpoint relative position calculation unit calculates the relative positional relationship between the user's viewpoint and the display device based on attributes of the user that have been acquired in advance. (3) Also, one aspect of the present invention is that in the display control device described above in (1), the viewpoint relative position calculation unit measures the distance between the user's pupils based on the image acquired by the image acquisition unit, and calculates the relative positional relationship between the user's viewpoint and the display device based on the measured distance between the user's pupils. (4) Also, in one aspect of the present invention, in the display control device of (3) described above, the viewpoint relative position calculation unit measures the interpupillary distance and interpupillary angle of view of the user based on the image acquired by the image acquisition unit, and calculates the relative positional relationship between the user's viewpoint and the display device based on the measured interpupillary distance and interpupillary angle of view of the user. (5) Also, one aspect of the present invention is that in the display control device described above in (3), the viewpoint relative position calculation unit calculates the relative positional relationship between the user's viewpoint and the display device based on the user's interpupillary distance stored in advance and the user's interpupillary distance measured from the image acquired by the image acquisition unit. (6) Also, one aspect of the present invention is that in any of the display control devices described above in (1) to (5), the viewpoint relative position calculation unit updates the relative positional relationship between the user's viewpoint and the display device in response to a change in the ratio of the sizes of the user's left and right pupils or irises obtained based on the image acquired by the image acquisition unit. (7) Furthermore, one aspect of the present invention is a display control device according to any one of (1) to (6) above, wherein the display control unit displays a plurality of the display objects on the display device and further includes an error assigning unit that assigns different errors to each of the plurality of display objects based on a virtual position of each of the plurality of display objects, and the display control unit controls the display of the plurality of the display objects on the display device by varying the relative relationship between the display objects and the background within the range of the assigned errors. (8) Also, one aspect of the present invention is that in the display control device described above in (7), the display control unit controls the display of a plurality of the display objects on the display device by varying the position at which the display objects are displayed within a given error range. (9) Also, one aspect of the present invention is that in the display control device described above in (7), the display control unit controls the display of a plurality of the display objects on the display device by varying the position at which the display objects are displayed in a predetermined direction within a given error range. (10) Also, one aspect of the present invention is that in the display control device described above in (1), the display control unit displays on the display device a plurality of display objects each located at a different distance from the display device, and the plurality of display objects are displayed at different display timings depending on their distance from the display device. (11) Another aspect of the present invention is a display system comprising: a display control device according to any one of (1) to (10) described above; an imaging device that captures an image of a user's face and provides the captured image to the image acquisition unit; and the display device controlled by the display control unit. (12) Also, one aspect of the present invention includes a display control step of controlling a computer to display a display object having three-dimensional information on a display device; an image acquisition step of acquiring an image captured by an imaging device rigidly connected to the display device, the image being an image of a user's face looking at the display object displayed on the display device; a viewpoint relative position calculation step of calculating a relative positional relationship between the viewpoint of the user and the display device based on the acquired image; a gravity information acquisition step of acquiring information regarding acceleration in the gravity direction and its azimuth angle; an initial position information acquisition step of acquiring position information of the location of the device when a predetermined point is set as the origin; an acceleration information acquisition step of acquiring acceleration information of the device; and a calculation step of calculating the position information acquired by the initial position information acquisition step and the acceleration information. a vector calculation step of calculating an absolute position of a projection plane onto which the display device projects an image based on acceleration information acquired by a gravity information acquisition step, calculating absolute position coordinates of the user's viewpoint based on the calculated absolute position of the projection plane and the relative positional relationship between the user's viewpoint and the display device calculated by the viewpoint relative position calculation step, and calculating a vector of the direction in which the projection plane faces based on the calculated absolute position of the projection plane, the gravity information acquired by the gravity information acquisition step, and the acceleration information acquired by the acceleration information acquisition step, wherein the display control step is a program that deforms three-dimensional information of the object to be displayed based on the information calculated by the vector calculation step, and controls the display on the display device. Effect of the Invention

[0007] According to the present invention, even if the user is not in front of the display device, the user's viewpoint can be aligned with the displayed image, thereby providing the effect of providing a sense of immersion in the virtual world. [Brief description of the drawings]

[0008] [Figure 1] 11A and 11B are diagrams for explaining changes in an image displayed by a display system according to an embodiment. [Diagram 2] 1A and 1B are diagrams for explaining virtual objects displayed by the display system according to the embodiment. [Diagram 3] FIG. 11 is a first diagram for explaining an example of a process of analyzing a user's viewpoint based on an image captured by the display system according to the present embodiment. [Figure 4] FIG. 2 is a second diagram for explaining an example of the analysis process of the user's viewpoint based on an image captured by the display system according to the present embodiment. [Diagram 5] FIG. 1 is a diagram for explaining an image of a display system according to an embodiment of the present invention. [Figure 6] 10A and 10B are diagrams for explaining excess and deficiency of information analyzed by the display system according to the embodiment. [Figure 7] 5A and 5B are diagrams for explaining vectors calculated by the display system according to the embodiment. [Figure 8] FIG. 2 is a block diagram showing an example of a functional block for realizing the display system according to the present embodiment. [Figure 9] 1 is a functional configuration diagram showing an example of a functional configuration of a display control device according to an embodiment of the present invention. [Figure 10] FIG. 11 is a functional configuration diagram showing a modified example of the functional configuration of the display control device according to the embodiment. [Figure 11] 11A and 11B are diagrams for explaining how to handle a case where the user's face is tilted in the display system according to the present embodiment. [Figure 12] 1 is a block diagram showing an example of an internal configuration of a display control device according to an embodiment of the present invention. [Figure 13] 11 is a diagram for explaining a change in coordinate axes in response to a change in the inclination of a smartphone when attempting to display a virtual object on the smartphone using augmented reality or mixed reality technology according to the related art. FIG. [Figure 14] FIG. 13 is a diagram for explaining changes in coordinate axes in response to changes in the inclination of smart glasses when attempting to display a virtual object on the smart glasses using virtual reality technology according to the related art. [Figure 15] 11A and 11B are diagrams for explaining a change in the appearance of a virtual object when the smartphone is tilted in a case where the virtual object is displayed on the smartphone using conventional technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A display control device, a display system, and a program according to the aspects of the present invention will be described in detail below with reference to the accompanying drawings, showing preferred embodiments. The aspects of the present invention are not limited to these embodiments, and include various modifications or improvements. In other words, the components described below include those that a person skilled in the art can easily imagine, and those that are substantially the same, and the components described below can be appropriately combined. In addition, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the present invention. In addition, in the drawings below, the scale and number of each structure may be different from the scale and number of each structure in order to make each structure easier to understand. In addition, due to the relationship with usable characters, those with vector symbols in the formulas shown below (variables expressed in English letters or Greek letters) may be described as "vectors (English letters or Greek letters)" in the text of this specification.

[0010] [Embodiment] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. First, the matters on which the embodiment of the present invention is based will be described. The display control device, display system, and program according to the present invention can be used in technologies such as Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR). The display control device, display system, and program according to the present invention can display a virtual object superimposed on an image (or video, hereinafter simply referred to as an image) of the real world. The image of the real world is an image of the world that the user actually sees, and may be captured by an imaging device. In addition, features may be extracted from an image captured by an imaging device, and a virtual object may be superimposed according to the extracted features. In addition, according to this embodiment, the position and direction of the device are detected based on position information, orientation information, gyro information, and the like, and the display is controlled based on the detected position and direction. By performing such a display, the user can obtain a sense of immersion.

[0011] However, this embodiment is not limited to this example, and it is also possible to display only virtual objects. A virtual object may be anything that has three-dimensional information, and broadly includes objects, backgrounds, and the like. In the following description, an embodiment in which a virtual object is simply displayed will be described. In this case, the background may be from the real world or may be from the virtual world.

[0012] The technology according to the present embodiment can be applied to devices having displays such as smartphones and tablet terminals, as well as wearable devices such as smart glasses and smart goggles. In the following description, an embodiment using a smartphone will be described as an example.

[0013] First, problems associated with the conventional technology will be described in detail with reference to FIGS.

[0014] 13 is a diagram for explaining a change in coordinate axes according to a change in the inclination of a smartphone when a virtual object is to be displayed on the smartphone using augmented reality or mixed reality technology according to the related art. With reference to the figure, an example of a case where a user U views a virtual object 92 using a device 91 capable of displaying virtual objects will be explained. The device 91 may be, for example, a smartphone.

[0015] FIG. 13(A) shows an example in which the user U holds the device 91 at a first angle. In this case, an object 92 exists at the point where the user U's viewpoint and the device 91 are connected. The device 91 captures an image or video of the real world and displays the object 92 by superimposing it on the captured image or video. The display of the object 92 is drawn, for example, based on a reference feature on the captured image. By looking at the display of the device 91 from the front, the user U can get a sense that the object 92 exists in the real world. However, according to the conventional technology, this is not the case when the angle of the device 91 is tilted by the user U and the user looks at the object 92 from a position other than the front of the display.

[0016] FIG. 13B shows an example in which the user U holds the device 91 at a second angle. In the example shown in FIG. 13B, the relationship between the coordinate axes of the real world and the coordinate axes of the device 91 is different from the example shown in FIG. 13A. As a result of the device 91 being tilted by the user U, the coordinate axes of the device 91 are also tilted, and the object 92 falls outside the imaging angle of the device 91. In this case, even though the object 92 should exist at the point where the viewpoint of the user U is connected to the device 91, the object 92 is not displayed on the display of the device 91. Therefore, the user U cannot feel immersed in the virtual space in which the virtual object is displayed, depending on the tilt of the device 91.

[0017] FIG. 14 is a diagram for explaining a change in coordinate axes according to a change in the inclination of smart glasses when a virtual object is to be displayed on the smart glasses using a virtual reality technology according to the related art. The example shown in FIG. 14 is an example of a case where smart glasses are used. In this case, as in the case shown in FIG. 13, the user U has multiple problems, such as a problem that it is difficult to obtain a sense of immersion in the virtual space in which the virtual object is displayed, and a problem for obtaining a sense of immersion. With reference to the same figure, an example of a case where the user U views a virtual object 92 using a device 93 capable of displaying a virtual object will be described. The device 93 may be, for example, smart glasses that the user U can wear.

[0018] FIG. 14(A) shows an example of a case where a user U wears a device 93 and views a virtual object 92 at a first angle. FIG. 14(B) shows an example of a case where a user U wears a device 93 and views a virtual object 92 at a second angle. In the case of smart glasses, the device 93 performs perspective drawing with a fixed operator's viewpoint as a focal point. In the case of smart glasses, the viewpoint and the coordinate axis of the device are fixed, so as shown in FIG. 13, a situation does not occur in which only the coordinate axis of the device changes while the viewpoint of the user U is fixed. Therefore, in the case of smart glasses, there is an effect that a greater sense of immersion can be obtained compared to a smartphone. However, smart glasses block the field of view, and there are safety issues. In addition, smart glasses require a lot of effort to wear and calibrate. Furthermore, there is a problem that it is not easy to miniaturize smart glasses with current technology. If smart glasses are not small, a sense of discomfort occurs when wearing them, and it is difficult to obtain a sense of immersion in the virtual world. For these reasons, it has to be said that the penetration rate of smart glasses is low compared to that of smartphones.

[0019] 15 is a diagram for explaining a change in a virtual object when the smartphone is tilted in a case where the virtual object is displayed on the smartphone using the conventional technology. With reference to the same figure, a change in the appearance of a virtual object when the smartphone is tilted in the conventional technology will be explained. In the illustrated example, a device 91 capable of displaying virtual objects displays an object that exists in the direction in which the device 91 is facing, among a plurality of virtual objects 92-1, 92-2, and 92-3.

[0020] FIG. 15(A) shows an example of a case where the user U holds the device 91 at a first angle. In this case, the device 91 faces the direction in which the object 92-2 exists, and therefore displays an image 92-2A of the object 92-2. Here, FIG. 15(B) shows an example of a case where only the device 91 is tilted while the viewpoint of the user U remains fixed. As shown in FIG. 15(B), when the device 91 is tilted to the right, the object 92-2 and the object 92-3 exist ahead of the device 91. Therefore, the device 91 displays an image 92-2A of the object 92-2 and an image 92-3A of the object 92-3. However, only the object 92-2 should exist ahead of the line of sight of the user U via the device 91, and if such a display is performed, the user U cannot get a sense of immersion in the virtual world.

[0021] Next, a display control device, a display system, and a program according to the present invention will be described with reference to Figs. 1 to 12. In the following description, an example of a case where a display control device, a display system, and a program according to the present embodiment are applied to a smartphone SP will be described. However, the present embodiment is not limited to this example, and can be widely applied to devices that have a display and display images to a user. The smartphone SP shown below as an example is a device that can display virtual objects. First, an overview of a display control device, a display system, and a program according to the present embodiment will be described with reference to Figs. 1 and 2.

[0022] 1 is a diagram for explaining a change in an image displayed by a display system according to an embodiment. With reference to the same figure, a change in the appearance of a virtual object in relation to the viewpoint of a user U when a smartphone SP is tilted in a display system according to this embodiment will be explained. In the illustrated example, the smartphone SP displays an object that exists in the direction in which the smartphone SP is facing, among a plurality of virtual objects OBJ1, object OBJ2, and object OBJ3. The positional relationship between the positions of these virtual objects relative to each other on the coordinate axes of the virtual world is predetermined.

[0023] In the positional relationship shown in FIG. 1(A), the smartphone SP faces the direction in which the object OBJ2 exists. Therefore, the smartphone SP displays an image OBJ2A of the object OBJ2. Here, in FIG. 1(B), the positional relationship between the user U and the center of gravity of the smartphone SP remains unchanged, but the direction in which the smartphone SP faces changes as the smartphone SP rotates to the right. According to the conventional technology, the display object was determined by detecting only the rotation of the smartphone SP, so that the display object changed with the rotation of the smartphone SP. However, according to the present embodiment, in addition to detecting the rotation of the smartphone SP, information such as the viewpoint of the user U is further taken into consideration, and thus a display is performed according to the positional relationship between the viewpoint of the user U, the smartphone SP, and the virtual object.

[0024] Here, FIG. 1(C) shows an example of information displayed by the smartphone SP when the user U moves from the positional relationship shown in FIG. 1(B) to the positional relationship shown in FIG. 1(C), there is no change in the positional relationship between the smartphone SP and the virtual object. However, in FIG. 1(C), the user U changes his / her viewpoint and looks into the screen from the left side (from the front of the smartphone SP). In such a case, the smartphone SP displays objects OBJ2 and OBJ3 that are in the line of sight of the user U through the smartphone SP. That is, according to this embodiment, by considering information such as the viewpoint of the user U, a display is performed according to the positional relationship between the viewpoint of the user U, the smartphone SP, and the virtual object, so that the user U can more easily get a sense of immersion in the virtual world. By changing the relative positions of multiple objects rendered in accordance with the movement of the viewpoint, a motion-based cue for depth perception (motion parallax) is obtained, resulting in depth perception that is independent of other perceptual psychological cues (e.g. binocular disparity), and a sense of three-dimensionality for the object can be obtained.

[0025] FIG. 2 is a diagram for explaining a virtual object displayed by the display system according to the present embodiment. The diagram shows the positional relationship between the user U, the smartphone SP, and the virtual object OBJ. The diagram also shows the coordinate axes of the real world in which the user U exists, and the coordinate axes of the smartphone SP. The coordinate axes of the smartphone SP can be said to be depiction reference coordinate axes for depicting a virtual object on the smartphone SP. FIG. 2(A) shows an example of a case in which the user U holds the smartphone SP at a first tilt, and FIG. 2(B) shows an example of a case in which the user U holds the smartphone SP at a second tilt. According to this embodiment, as shown in the figure, even if the tilt of the smartphone SP changes, the coordinate axes of the smartphone SP remain fixed. That is, according to this embodiment, it is possible to draw the object OBJ so that it is transparent through the screen, based on the viewpoint of the user U, regardless of the position or angle of the screen. Moreover, by adopting such a configuration, the user U can get an experience as if the virtual object OBJ is fixed in the space of the real world.

[0026] Next, the operation principle of the display control device 10, the display system 1, and the program according to this embodiment will be described with reference to FIGS.

[0027] FIG. 3 is a first diagram for explaining an example of a process for analyzing a user's viewpoint based on an image captured by the display system according to this embodiment. FIG. 4 is a second diagram for explaining an example of a process for analyzing a user's viewpoint based on an image captured by the display system according to this embodiment. First, the display system 1 captures an image of the face of the user U and analyzes the position of the user U's pupils. In the figure, an image IMG in which the face of the user U is captured is shown. The image IMG may be one of consecutive frame images included in the moving image data. The display system 1 analyzes the image IMG and specifies the positions of the right eye ER and the left eye EL of the user U. Based on the specified positions of the right eye ER and the left eye EL of the user U, the display system 1 calculates a distance D from the right eye ER to the left eye EL of the user U. base The distance Dbase from the right eye ER to the left eye EL of the user U may be calculated by image processing, or may be measured by the user U himself / herself and input in advance.

[0028] The display system 1 also determines an angle η between a line connecting the center O of the image IMG to the center of gravity of the pupil and a reference line of the image IMG. Furthermore, the display system 1 determines an interpupillary angle of view φ. Here, the reference length L0=D base / sin(φ / 2), the distance D from the right eye ER to the left eye EL of the user U is a point away from the center of gravity COGR of the right eye ER and the center of gravity COGL of the left eye EL by a reference length L0. base The point at the midpoint of these points can be defined as point P. The interpupillary angle of view φ is the angle formed by a line connecting point P and the center of gravity COGR of the right eye ER, and a line connecting point P and the center of gravity COGL of the left eye EL. The display system 1 also specifies the angle of view θ from the center O of the image IMG to the pupil center of gravity. Note that known image processing techniques, such as object detection techniques, may be used to specify these angles and calculate the values.

[0029] 5 is a diagram for explaining an image of the display system according to this embodiment. With reference to the figure, a calculation process of a vector V indicating the coordinates of a perspective plane in a virtual space, a plane vector W of the perspective plane in a virtual space, and a vector U indicating a viewpoint coordinate in a virtual space, which is performed by the display system 1, will be explained. Note that the perspective plane is a display surface of a display (display device) equipped in the smartphone SP. The calculation of the vector U indicating the viewpoint coordinate in a virtual space, the vector V indicating the coordinates of the perspective plane in a virtual space, and the plane vector W of the perspective plane in a virtual space requires the above-mentioned distance D from the right eye ER to the left eye EL of the user U. base In addition to the angle η, which is the angle between the line connecting the centers of the pupils and the reference line of the image IMG, the interpupillary angle of view φ, and the angle of view θ from the center O of the image IMG to the center of the pupil, a vector S indicating the initial coordinates in the virtual space and a vector ζ indicating the angle between the display and the direction of gravity are used.

[0030] A vector S indicating an initial coordinate in the virtual space is an initial coordinate (reference coordinate) in the virtual space. A virtual object to be displayed is generated based on the vector S indicating the initial coordinate in the virtual space. The vector S indicating the initial coordinate in the virtual space may be determined in advance and stored in a predetermined storage unit.

[0031] A vector ζ indicating the angle between the display and the direction of gravity is acquired from a gravity sensor Gs. The gravity sensor Gs is provided in the smartphone SP and detects acceleration in the direction of gravity. The gravity sensor Gs also detects the azimuth angle of the direction of gravity. The display system 1 acquires information on the acceleration in the direction of gravity and its azimuth angle from the gravity sensor Gs, and calculates a vector ζ indicating the angle between the display and the direction of gravity based on the acquired information.

[0032] The vector U indicating the viewpoint coordinates in the virtual space may be calculated using the following equation (1).

[0033]

number

[0034] Here, f in equation (1) is a coordinate transformation function that determines the position of vector U, which indicates the viewpoint coordinate in the virtual space, with respect to vector S, which indicates the initial coordinate in the virtual space, from the relative positional relationship between the camera and the pupil.

[0035] Moreover, the following formula (2) may be used to calculate a vector V indicating the coordinates of a perspective plane in the virtual space.

[0036]

number

[0037] Furthermore, the following formula (3) may be used to calculate a vector W indicating a plane vector of a perspective surface in the virtual space.

[0038]

number

[0039] The display system 1 perspectively renders a three-dimensional object in a virtual space on a plane represented by vector V indicating the coordinates of the perspective plane in the virtual space / plane vector W of the perspective plane in the virtual space, with vector U indicating the viewpoint coordinates in the virtual space as the focus.

[0040] In addition, the vector U indicating the viewpoint coordinates in the virtual space, the vector V indicating the coordinates of the perspective plane in the virtual space, and the plane vector W of the perspective plane in the virtual space are calculated using the distance D from the right eye ER to the left eye EL of the user U. base Instead of using the above, a base line length L0 as shown in the figure may be used. A distance sensor such as a LiDAR (Light Detection And Ranging) sensor or a ToF (Time of Flight) sensor may be used to calculate the base line length L0.

[0041] When calculating the vector U indicating the viewpoint coordinates in the virtual space, the vector V indicating the coordinates of the perspective plane in the virtual space, and the plane vector W of the perspective plane in the virtual space using the base line length L0, the following equations (4), (5), and (6) are used.

[0042]

number

[0043]

number

[0044]

number

[0045] FIG. 6 is a diagram for explaining excess and deficiency of information analyzed by the display system according to this embodiment. With reference to the figure, excess and deficiency of the amount of information used by the display system 1 according to this embodiment will be described. The vectors shown in the figure are unknown information sought by the display system 1. Known information includes the vector ζ indicating the angle between the display and the direction of gravity, and the distance D from the right eye ER to the left eye EL of the user U.base and the amount of information is 4. The acquired information includes the angle η between the line connecting the centers of gravity of the pupils and the reference line of the image IMG, the interpupillary angle of view φ, the angle of view θ from the center O of the image IMG to the center of gravity of the pupils, and the vector ζ indicating the angle between the display and the direction of gravity, and the amount of information is 6. That is, of the information used by the display system 1 according to this embodiment, the amount of known information is 10.

[0046] On the other hand, the information (unknown information) that the display system 1 must determine is a vector (three-dimensional) indicating the viewpoint coordinates in the virtual space, a vector (three-dimensional) indicating the perspective plane coordinates in the virtual space, and a plane vector (three-dimensional) of the perspective plane in the virtual space, and the amount of unknown information is 9.

[0047] Therefore, since there are 9 unknowns compared to 10 knowns, it can be said that there is an excess of information. However, since the baseline length L0 (one piece of information) is calculated using two pieces of information, the interpupillary distance and the interpupillary angle of view, the known number is 9, so there is no excess or deficiency of information and this is the minimum configuration.

[0048] 7 is a diagram for explaining vectors calculated by the display system according to this embodiment. With reference to the same figure, a method for calculating the above-mentioned unknowns (three pieces of three-dimensional information) will be described. Note that the conditions for the method described below are, first, that the display surface (projection surface) of the display device, the selfie camera that captures the face of the user U, and the gravity sensor are rigidly coupled, second, that the display surface, the selfie camera focal point, and the gravity sensor are arranged in approximately the same plane, and third, that the distance between the display surface, the selfie camera focal point, and the gravity sensor is within the range of the viewpoint relative position E L It can be noted that it is sufficiently small compared to

[0049] As shown in the figure, the unknowns (three pieces of three-dimensional information) necessary for display by the display system 1 can be calculated by combining known vectors. For example, the absolute viewpoint position Ez can be calculated by the following equation (7).

[0050]

number

[0051] In addition, P and Q in the formula (7) are correction values ​​based on the coupling angle between the selfie camera axis and the gravity sensor axis.

[0052] In addition, the gravity orientation sensor input G θ For the first or regular measurement, θ0 As, G θ =G θ0 +Δ θ It is also possible to use the following.

[0053] When calculating the unknowns (three pieces of three-dimensional information) by combining known vectors, the unknowns can be found by the following equations (8), (9), and (10).

[0054]

number

[0055]

number

[0056]

number

[0057] The vector W shown in equation (10) can also be obtained by using equation (11).

[0058]

number

[0059] Next, an example of a specific configuration for realizing the above-described display system 1 will be described with reference to FIGS.

[0060] 8 is a block diagram showing an example of a functional block for realizing the display system according to this embodiment. With reference to the diagram, an example of the functional configuration of the display system 1 will be described. The display system 1 includes an imaging device 2, a gravity sensor 3, an acceleration sensor 4, a storage device 5, a display device 6, and a display control device 10. Note that the display system 1 may include a gyro sensor and the like in addition to the configuration shown in the figure.

[0061] The imaging device 2 captures an image of the face of the user U. That is, the imaging device 2 may be an in-camera (selfie camera) attached to a smartphone SP or the like. The user U is a user of the display system 1. Specifically, the imaging device 2 may be a CCD camera using a CCD (Charge Coupled Devices) image sensor, or a CMOS camera using a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0062] The gravity sensor 3 detects acceleration in the direction of gravity and also detects the relative orientation of the device with respect to an absolute reference such as the earth's axis.

[0063] The acceleration sensor 4 detects the acceleration of the device itself. The acceleration sensor 4 may be capable of detecting gravitational acceleration, in which case the gravity sensor 3 and the acceleration sensor 4 may have the same configuration. The acceleration sensor 4 detects acceleration in three axial directions (X-axis, Y-axis, and Z-axis).

[0064] The storage device 5 stores parameter information used in the calculation. The parameter information includes information on attributes of the user U and information on the initial position of the display system 1. The information stored in the storage device 5 may be received from the user U by an operation receiving unit (not shown), or may be updated by the display control device 10.

[0065] The display device 6 displays various information according to the control of the display control device 10. The display device 6 acquires a control signal from the display control device 10, and displays information according to the acquired control signal. The display device 6 may be, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or the like. It is preferable that the imaging device 2 and the display device 6 are rigidly connected to each other.

[0066] The display control device 10 acquires information from the imaging device 2, the gravity sensor 3, the acceleration sensor 4, the storage device 5, etc., generates information to be displayed on the display device 6 based on the acquired information, and controls the display on the display device 6. An example of the functional configuration of the display control device 10 will be described in detail below with reference to FIG. 9.

[0067] FIG. 9 is a functional configuration diagram showing an example of the functional configuration of the display control device according to this embodiment. The display control device 10 includes an image acquisition unit 11, a viewpoint relative position calculation unit 12, a gravity information acquisition unit 13, a display device angle calculation unit 14, an acceleration information acquisition unit 15, an initial position information acquisition unit 16, a vector calculation unit 17, and a display control unit 18. Each of these functional units is realized, for example, by using an electronic circuit. Each functional unit may include a storage means such as a semiconductor memory or a magnetic hard disk device, as necessary. Each function may be realized by a computer having a CPU (Central Processing Unit) and software. All or a part of each functional unit may be realized by using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field-Programmable Gate Array). All or a part of each functional unit may be realized by a combination of software and hardware.

[0068] The image acquiring unit 11 acquires an image captured by the imaging device 2. The image acquired by the image acquiring unit 11 is an image of the face of a user U of the display system 1. Note that the image acquiring unit 11 may acquire successive frame images included in moving image data captured by the imaging device 2.

[0069] The viewpoint relative position calculation unit 12 calculates the relative positional relationship between the viewpoint of the user U and the display device 6 based on the image acquired by the image acquisition unit 11. The viewpoint relative position calculation unit 12 includes, for example, a pupil position identification unit 121, an interpupillary angle of view identification unit 122, and a viewpoint relative position identification unit 123, thereby calculating the relative positional relationship between the viewpoint of the user U and the display device 6. Note that the specific calculation method of the viewpoint relative position calculation unit 12 is not limited to this example, and other methods may be used.

[0070] The pupil position specifying unit 121 specifies the position coordinates of the left eye EL and the right eye ER of the user U based on the image acquired by the image acquiring unit 11. By specifying the position coordinates of the left eye EL and the right eye ER of the user U, the interpupillary distance can be specified. In addition, a known image processing technique such as an object detection technique may be used to specify the position coordinates. Next, the interpupillary angle of view specifying unit 122 detects the interpupillary angle of view φ based on the specified position coordinates of the left eye EL and the right eye ER of the user U. Furthermore, the viewpoint relative position specifying unit 123 specifies the relative position between the viewpoint of the user U and the display device 6 based on the specified interpupillary distance, the detected interpupillary angle of view φ, and the image acquired by the image acquiring unit 11. In addition, the method of specifying the relative position between the viewpoint of the user U and the display device 6 can be performed by the method described with reference to FIG. 3 to FIG. 7.

[0071] The relative position between the viewpoint of the user U and the display device 6 may be specified based on the attributes of the user U. The attributes of the user U may be, for example, the actual value of the interpupillary distance, gender, age, race, etc. The attributes of the user U are input in advance based on the operation of the user U and stored in a predetermined storage unit. In this case, the viewpoint relative position calculation unit 12 calculates the relative positional relationship between the viewpoint of the user U and the display device 6 based on the attributes of the user U acquired in advance. For example, the viewpoint relative position calculation unit 12 may calculate the relative positional relationship between the viewpoint of the user U and the display device 6 based on the interpupillary distance of the user U stored in advance and the interpupillary distance of the user U measured from the image acquired by the image acquisition unit 11.

[0072] The gravity information acquisition unit 13 acquires from the gravity sensor 3 information on the acceleration in the direction of gravity and the relative orientation of the device with respect to an absolute reference such as the earth's axis.

[0073] The display device angle calculation unit 14 calculates the angle of the display device 6 based on the acceleration in the direction of gravity acquired by the gravity information acquisition unit 13 and information related to the azimuth angle. Here, since the display device 6 is rigidly coupled to the gravity sensor 3, the display device angle calculation unit 14 can calculate the angle of the display device 6 based on the acceleration in the direction of gravity acquired by the gravity information acquisition unit 13 and information related to the relative orientation of the device with respect to an absolute reference such as the earth's axis.

[0074] The acceleration information acquisition unit 15 acquires acceleration information of the own device from the acceleration sensor 4. Specifically, the acceleration information acquisition unit 15 acquires acceleration information in three axial directions (X-axis, Y-axis, and Z-axis).

[0075] The initial position information acquisition unit 16 acquires the acceleration information of the own device acquired by the acceleration information acquisition unit 15, and acquires an origin coordinate OC, which is three-dimensional coordinate information of a predetermined point. The initial position information acquisition unit 16 acquires position information of the own device when the origin coordinate OC is set as the origin. The initial position information is the initial position vector S (Ix, Iy, Iz) shown in FIG. 7.

[0076] The vector calculation unit 17 calculates unknown information based on known information. The unknown information calculated by the vector calculation unit 17 is the three three-dimensional vectors described with reference to Figs. 6 and 7. Specifically, the unknown information calculated by the vector calculation unit 17 is the absolute position (perspective plane position vector) of the projection plane onto which the display device 6 projects an image, the absolute position coordinates (viewpoint position vector) of the user U, and a vector of the direction in which the projection plane faces (perspective plane vector). A specific calculation method will be described below.

[0077] The absolute position of the projection plane onto which the display device 6 projects an image (the perspective plane position vector) is the projection plane absolute position vector V(P x, P y, P z The vector is calculated based on the position information acquired by the initial position information acquisition unit 16 and the acceleration information acquired by the acceleration information acquisition unit 15.

[0078] The absolute position coordinates (eye point position vector) of the user U are the eye point absolute position vector U(E x, E y, E z ) The vector is the calculated projection plane absolute position vector V(P x, P y, P z ) and the relative positional relationship between the viewpoint of the user U and the display device 6 calculated by the viewpoint relative position calculation unit 12.

[0079] The vector in the direction in which the projection plane faces (the perspective plane vector) is the projection plane vector (P θ, P Φ, P Ψ ) The vector is the projection plane absolute position vector V(P x, P y, P z ), the gravity information acquired by the gravity information acquisition unit 13, and the acceleration information acquired by the acceleration information acquisition unit 15.

[0080] The display control unit 18 transforms the three-dimensional information of the display object based on the information calculated by the vector calculation unit 17, and controls the display on the display device 6.

[0081] By adopting such a configuration, the display system 1 deforms and displays the display target object, which is a virtual object, according to the position of the user U's face, the inclination of the smartphone SP, and the like. Therefore, the user U can experience as if he or she were looking into a virtual space through a window, even without using a wearable device. Furthermore, the sense of immersion is enhanced by the motion parallax that is generated by the user's minute movements.

[0082] According to the above-described embodiment, when the positional relationship between the viewpoint of the user U and the smartphone SP does not change, the display of the virtual object remains fixed on the display device 6. When there is no movement on the screen, it may be difficult to grasp the three-dimensional positional relationship, and it may be difficult to obtain a sense of immersion in the virtual space. Therefore, hereinafter, as a modification of the display control device 10 according to this embodiment, a display control device 10A that can obtain a sense of immersion in the virtual space even when the positional relationship between the viewpoint of the user U and the smartphone SP does not change will be described.

[0083] 10 is a functional configuration diagram showing a modified example of the functional configuration of the display control device according to the present embodiment. With reference to the same figure, an example of the functional configuration of the display control device 10A will be described. In the description of the display control device 10A, the same components as those in the display control device 10 may be denoted by the same reference numerals and description thereof may be omitted. The display control device 10A differs from the display control device 10 in that it further includes an error providing unit 19.

[0084] The error assigning unit 19 assigns different errors to each of the multiple display objects based on the virtual position of each of them. The errors may be given as distance information, for example. The error assigning unit 19 may assign a large error to an object that exists near the calculated viewpoint position, and a small error to an object that exists far from the calculated viewpoint position, for example. Conversely, the error assigning unit 19 may assign a small error to an object near the smartphone SP, and a large error to an object that exists far from the smartphone SP.

[0085] The display control unit 18 controls the display of a plurality of display objects on the display device 6 by varying the relative relationship between the display object and the background within the range of the error given by the error giving unit 19. That is, according to the present embodiment, the relative relationship between the display object and the background is varied according to the distance from the smartphone SP to the position where the virtual object exists. By varying the relative relationship, the user U can get the sensation that the virtual object is trembling. In addition, since the amount of trembling varies according to the distance, the user U can easily grasp the three-dimensional positional relationship and can be immersed in the three-dimensional virtual world.

[0086] The change in the relative relationship between the display objects and the background may be realized by changing the position information of the virtual object, or by changing the position information of the background. When the change is realized by changing the position information of the virtual object, the display control unit 18 controls the display of the multiple display objects on the display device 6 by changing the positions at which the display objects are displayed within the given error range.

[0087] The relative relationship between the display object and the background is preferably changed in a predetermined direction. The predetermined direction is preferably either the horizontal direction or the vertical direction. For example, it is possible to change the display object and the background in a mixed manner in the vertical direction and the horizontal direction, but excessive changes may burden the user U and cause a feeling of intoxication.

[0088] Furthermore, when multiple display objects are displayed on the display device 6 and the display is deformed by the movement of any one of the absolute viewpoint position, the display absolute position, or the display orientation vector, it is preferable to delay the display update by a delay time of different lengths. By adopting such a configuration, it is possible to give the illusion of a larger displacement than the actual one, making it easier to perceive depth. In this case, the error providing unit 19 may provide a delay time of different lengths for each.

[0089] In addition, when the display positions of objects change due to a change in vector, a more realistic change may be expressed by displaying them with different delay times. For example, in FIG. 1C, when the user U moves leftward, the object OBJ2 in front of the user U and the object OBJ3 behind the user U move faster and slower as seen by the user as a physical law. Therefore, following such physical laws, the smartphone SP may also display the objects on the smartphone SP with a delay in the display timing between the front and the back. In this case, the display control unit 18 displays a plurality of display objects that are at different distances from the display device 6 on the display device 6, and the plurality of display objects are displayed at different display timings according to their respective distances from the display device 6.

[0090] FIG. 11 is a diagram for explaining the concept of the case where the user's face is tilted in the display system according to the present embodiment. With reference to the same figure, processing of the change in apparent distance of the interpupillary angle of view due to the left-right inclination of the face will be explained. FIG. 11(A) shows the interpupillary angle of view φ when the user U views the display device 6 from the front. FIG. 11(B) shows the interpupillary angle of view φ when the user U looks at the display device 6 from an angle (sideways). In this way, the interpupillary angle of view φ changes according to the inclination of the user U's face. In addition, the interpupillary distance also varies according to the inclination of the user U's face. In order to solve such a problem, the relative position between the user U's viewpoint and the display device 6 may be specified based on the ratio of the size of the left and right pupils or irises of the user U. In this case, the viewpoint relative position calculation unit 12 may update the relative positional relationship between the user U's viewpoint and the display device 6 according to the change in the ratio of the size of the left and right pupils or irises of the user U obtained based on the image acquired by the image acquisition unit 11.

[0091] The relative position between the viewpoint of the user U and the display device 6 may be specified using a three-dimensional face model on which the characteristic positions of the eyes, nose, mouth, etc. are arranged. The simplest three-dimensional face model can be represented by a spherical model, but other suitable models may also be used. In this case, the characteristic positions of the eyes, nose, mouth, etc. are arranged on the three-dimensional face model, and the relative positional relationship can be updated by calculating the position and angle of the three-dimensional face model that best matches the apparent positions.

[0092] In addition, in an actual use case, there may be cases where two people simultaneously look into the display device 6. In such a case, one of the users U may be identified using iris authentication technology, and a suitable display may be provided to the identified user U.

[0093] FIG. 12 is a block diagram showing an example of the internal configuration of the display control device 10 of the present embodiment. At least some of the functions of the display control device 10 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902 or the like. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations according to each instruction. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with an external input / output device or the like. The input / output devices 904 and 905 are input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data from the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906. Also, all or part of each functional unit provided in the display control device 10 may be realized using hardware such as an ASIC, a PLD, or an FPGA. Also, all or part of each functional unit may be realized by a combination of software and hardware.

[0094] [Summary of the embodiment] According to this embodiment, the display control device 10 is provided with a display control unit 18, which controls the display of a display object, which is a virtual object having three-dimensional information, on the display device 6; the image acquisition unit 11 acquires an image captured by an imaging device 2 rigidly connected to the display device 6, the image capturing an image of the face of a user U looking at the display object displayed on the display device 6; the viewpoint relative position calculation unit 12 calculates the relative positional relationship between the viewpoint of the user U and the display device 6 based on the acquired image; the gravity information acquisition unit 13 acquires information regarding acceleration in the direction of gravity and its azimuth angle; the initial position information acquisition unit 16 acquires position information at which the device is located when a specified point is set as the origin; and the acceleration information acquisition unit 15 acquires acceleration information of the device.

[0095] Further, the display control device 10 calculates the absolute position (perspective plane position vector) of the projection plane onto which the display device 6 projects the image based on the position information acquired by the initial position information acquisition unit 16 and the acceleration information acquired by the acceleration information acquisition unit 15, calculates the absolute position coordinates (viewpoint position vector) of the user U based on the calculated absolute position of the projection plane and the relative positional relationship between the viewpoint of the user U and the display device 6 calculated by the viewpoint relative position calculation unit 12, and calculates a vector (perspective plane vector) of the direction in which the projection plane faces based on the calculated absolute position of the projection plane, the gravity information acquired by the gravity information acquisition unit 13, and the acceleration information acquired by the acceleration information acquisition unit 15. Furthermore, the display control unit 18 transforms the three-dimensional information of the display object based on the information calculated by the vector calculation unit 17, and controls the display on the display device 6. By adopting such a configuration, even if the user U is not in front of the display device 6, the display control device 10 can match the viewpoint of the user U with the display image, so that a sense of immersion can be obtained.

[0096] Furthermore, according to the above-described embodiment, the viewpoint relative position calculation unit 12 calculates the relative positional relationship between the viewpoint of the user U and the display device 6 based on attributes of the user U acquired in advance. Therefore, the display control device 10 can accurately calculate the relative positional relationship between the viewpoint of the user U and the display device 6. Therefore, the display control device 10 can accurately match the viewpoint of the user U with the display image, thereby providing a greater sense of immersion.

[0097] Furthermore, according to the above-described embodiment, the viewpoint relative position calculation unit 12 measures the interpupillary distance of the user U based on the image acquired by the image acquisition unit 11, and calculates the relative positional relationship between the viewpoint of the user U and the display device 6 based on the measured interpupillary distance of the user U. Thus, the display control device 10 can accurately match the viewpoint of the user U with the displayed image, thereby providing a greater sense of immersion.

[0098] Furthermore, according to the above-described embodiment, the viewpoint relative position calculation unit 12 measures the interpupillary distance and interpupillary angle of view of the user U based on the image acquired by the image acquisition unit 11, and calculates the relative positional relationship between the viewpoint of the user U and the display device 6 based on the measured interpupillary distance and interpupillary angle of view of the user U. Thus, the display control device 10 can accurately match the viewpoint of the user U with the displayed image, thereby providing a greater sense of immersion.

[0099] Furthermore, according to the above-described embodiment, the viewpoint relative position calculation unit 12 calculates the relative positional relationship between the viewpoint of the user U and the display device 6, based on the pre-stored interpupillary distance of the user U and the interpupillary distance of the user U measured from the image acquired by the image acquisition unit 11. Therefore, when the interpupillary distance of the user U is known, the display control device 10 can accurately match the viewpoint of the user U with the displayed image, thereby providing a greater sense of immersion.

[0100] Furthermore, according to the above-described embodiment, the viewpoint relative position calculation unit 12 updates the relative positional relationship between the viewpoint of the user U and the display device 6 in accordance with a change in the ratio of the sizes of the left and right pupils or irises of the user U obtained based on the image acquired by the image acquisition unit 11. Therefore, even if the face of the user U is tilted, the display control device 10 can accurately match the viewpoint of the user U with the display image, thereby providing a greater sense of immersion.

[0101] Also, according to the above-described embodiment, in the display control device 10A, the display control unit 18 displays a plurality of display objects on the display device 6. The display control device 10A further includes an error imparting unit 19 that imparts different errors to the plurality of display objects based on the virtual positions of the respective display objects, and the display control unit 18 controls the display of the plurality of display objects on the display device 6 by varying the relative relationship between the display objects and the background within the range of the imparted errors. That is, the display control device 10A intentionally moves the display objects in a jiggling manner to provide a visual effect that makes the objects appear more three-dimensional, and can give the user U a sense of immersion in the virtual world.

[0102] Furthermore, according to the above-described embodiment, the display control unit 18 controls the display of a plurality of display objects on the display device 6 by varying the positions at which the display objects are displayed within the range of the error given by the error giving unit 19. That is, the display control unit 18 varies the relative relationship between the display objects and the background by varying the positions of the display objects among the background and the display objects. Therefore, the display control device 10A intentionally moves the display objects against the background in a jiggling manner, thereby providing a visual effect that makes the objects appear more three-dimensional, and can give the user U a sense of immersion in the virtual world.

[0103] Further, according to the above-described embodiment, The display control unit 18 controls the display of a plurality of display objects on the display device 6 by varying the position at which the display object is displayed in a predetermined direction within the range of error given by the error giving unit 19. According to this embodiment, the display control device 10A does not cause shaking by mixing up, down, left, and right, so that the user U is less likely to feel dizzy when looking at the screen displayed by the display control device 10A.

[0104] Furthermore, this will make it possible to improve the quality of services that present users with virtual three-dimensional information via smartphones, tablet devices, etc., thereby contributing to Goal 8 of the United Nations Sustainable Development Goals (SDGs) which is to "Promote inclusive and sustainable economic growth, employment and decent work for all."

[0105] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like that do not deviate from the gist of the present invention are also included.

[0106] In addition, a computer program for implementing the functions of each of the above-mentioned devices may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Note that the "computer system" referred to here may include hardware such as an OS and peripheral devices. In addition, the term "computer-readable recording medium" refers to writable non-volatile memory such as a flexible disk, an optical magnetic disk, a ROM, or a flash memory, a portable medium such as a DVD (Digital Versatile Disc), or a storage device such as a hard disk built into a computer system.

[0107] Furthermore, "computer-readable recording medium" also includes a medium that holds a program for a certain period of time, such as a volatile memory (e.g., DRAM (Dynamic Random Access Memory)) inside a computer system that becomes a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. The above program may also be transmitted from a computer system that stores the program in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for implementing some of the functions described above, or may be a so-called differential file (differential program) that can implement the functions described above in combination with a program already recorded in the computer system. [Explanation of symbols]

[0108] 1...display system, 2...imaging device, 3...gravity sensor, 4...acceleration sensor, 5...storage device, 6...display device, 10...display control device, 11...image acquisition unit, 12...viewpoint relative position calculation unit, 13...gravity information acquisition unit, 14...display device angle calculation unit, 15...acceleration information acquisition unit, 16...initial position information acquisition unit, 17...vector calculation unit, 18...display control unit, 19...error assignment unit, 121...pupil position determination unit, 122...interpupillary angle of view determination unit, 123...viewpoint relative position determination unit, OC...origin coordinates, DOI...display object information

Claims

1. a display control unit that controls displaying a display object having three-dimensional information on a display device; an image acquisition unit that acquires an image captured by an imaging device rigidly connected to the display device, the image being an image of a face of a user looking at the display object displayed on the display device; a viewpoint relative position calculation unit that calculates a relative positional relationship between the viewpoint of the user and the display device based on the acquired image; a gravity information acquisition unit that acquires information about the acceleration in the gravity direction and its azimuth angle; an initial position information acquisition unit that acquires position information of the location of the own device when a predetermined point is set as an origin; an acceleration information acquisition unit that acquires acceleration information of the own device; a vector calculation unit that calculates an absolute position of a projection plane onto which the display device projects an image, based on the position information acquired by the initial position information acquisition unit and the acceleration information acquired by the acceleration information acquisition unit, calculates absolute position coordinates of the user's viewpoint, based on the calculated absolute position of the projection plane and the relative positional relationship between the user's viewpoint and the display device calculated by the viewpoint relative position calculation unit, and calculates a vector of the direction in which the projection plane faces, based on the calculated absolute position of the projection plane, the gravity information acquired by the gravity information acquisition unit, and the acceleration information acquired by the acceleration information acquisition unit; Equipped with The display control unit transforms three-dimensional information of the display object based on the information calculated by the vector calculation unit, and controls display on the display device. Display control device.

2. The viewpoint relative position calculation unit calculates a relative positional relationship between the viewpoint of the user and the display device based on attributes of the user acquired in advance. The display control device according to claim 1 .

3. The viewpoint relative position calculation unit measures a distance between the pupils of the user based on the image acquired by the image acquisition unit, and calculates a relative positional relationship between the viewpoint of the user and the display device based on the measured distance between the pupils of the user. The display control device according to claim 1 .

4. The viewpoint relative position calculation unit measures an interpupillary distance and an interpupillary angle of view of the user based on the image acquired by the image acquisition unit, and calculates a relative positional relationship between the viewpoint of the user and the display device based on the measured interpupillary distance and interpupillary angle of view of the user. The display control device according to claim 3 .

5. The viewpoint relative position calculation unit calculates a relative positional relationship between the viewpoint of the user and the display device based on a pre-stored interpupillary distance of the user and the interpupillary distance of the user measured from the image acquired by the image acquisition unit. The display control device according to claim 3 .

6. The viewpoint relative position calculation unit updates a relative positional relationship between the viewpoint of the user and the display device in accordance with a change in a ratio of sizes of the left and right pupils or irises of the user obtained based on the image acquired by the image acquisition unit. The display control device according to claim 1 .

7. The display control unit displays a plurality of the display objects on the display device, an error providing unit that provides different errors based on the virtual positions of the plurality of display objects, The display control unit controls display of the plurality of display objects on the display device by varying a relative relationship between the display objects and a background within a given error range. The display control device according to claim 1 .

8. The display control unit controls display of the plurality of display objects on the display device by varying positions at which the display objects are displayed within a given error range. The display control device according to claim 7.

9. The display control unit controls display of the plurality of display objects on the display device by varying, in a predetermined direction, the positions at which the display objects are displayed within the given error range. The display control device according to claim 7.

10. the display control unit displays, on the display device, a plurality of the display objects that are present at different distances from the display device; The plurality of display objects are displayed at different display timings depending on the distance from the display device. The display control device according to claim 1 .

11. A display control device according to any one of claims 1 to 10, the imaging device that captures an image of a user's face and provides the captured image to the image acquisition unit; the display device controlled by the display control unit; A display system comprising:

12. On the computer, a display control step of controlling the display of a display object having three-dimensional information on a display device; an image capturing step of capturing an image captured by an imaging device rigidly connected to the display device, the image being an image of a face of a user looking at the display object displayed on the display device; a viewpoint relative position calculation step of calculating a relative positional relationship between the viewpoint of the user and the display device based on the acquired image; a gravity information acquiring step of acquiring information about an acceleration in a gravity direction and an azimuth angle thereof; an initial position information acquisition step of acquiring position information of a location where the own device is located when a predetermined point is set as an origin; an acceleration information acquisition step of acquiring acceleration information of the own device; a vector calculation step of calculating an absolute position of a projection plane onto which the display device projects an image, based on the position information acquired in the initial position information acquisition step and the acceleration information acquired in the acceleration information acquisition step, calculating absolute position coordinates of the user's viewpoint, based on the calculated absolute position of the projection plane and the relative positional relationship between the user's viewpoint and the display device calculated in the viewpoint relative position calculation step, and calculating a vector of the direction in which the projection plane faces, based on the calculated absolute position of the projection plane, the gravity information acquired in the gravity information acquisition step, and the acceleration information acquired in the acceleration information acquisition step; A program for executing The display control step transforms three-dimensional information of the object to be displayed based on the information calculated in the vector calculation step, and controls display on the display device. program.

Images