Information display device and display method therefor

Abstract

To provide an information display device and a display method for the information display device that make menu display and the like easy to be viewed so as to be usable even when a display section is inclined.SOLUTION: An information display device having a display section comprises: an object data acquisition section that acquires information related to an AR object and generates the AR object; an additional object generation section that generates an additional object related to the AR object; an attitude detection section that detects an inclination of the information display device relative to a gravity direction in a real space; and a display control section that displays the AR object so that the AR object is inclined relative to the display section in accordance with the inclination of the information display device, displays the additional object so that the additional object is inclined relative to the display section in accordance with the inclination of the information display device when the inclination of the information display device is less than a predetermined value, and displaying the additional object using the same vertical reference as for the display section so that the additional object is not inclined relative to the display section regardless of the inclination of the information display device when the inclination is equal to or greater than the predetermined value.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an information display device and a display method thereof. [Background technology]

[0002] One type of information display device is a head-mounted display device (hereinafter referred to as HMD), which is a device worn on the head to view information. HMDs can superimpose images of virtual objects (AR objects: Argument Reality Objects) such as images and text onto real space, and can be visually recognized. In recent years, their use has been expanding in fields such as gaming and work support.

[0003] Here, the virtual space that the user views by wearing the HMD needs to change in accordance with the movement of the user's head in order to create a state of immersion in the virtual world.

[0004] Patent Document 1 is a background art in this technical field. Patent Document 1 discloses an image processing device that presents a stereoscopic image using an HMD, and includes a reference image acquisition unit that acquires a reference image consisting of a pair of left-eye and right-eye images that represent an object viewed from two viewpoints that are horizontally spaced apart, a tilt angle acquisition unit that acquires the tilt angle of the vertical axis of the head of a user wearing the HMD from the vertical direction in a plane parallel to the screen, an image conversion unit that performs conversion processing to rotate the left-eye and right-eye images around a common axis in a virtual space that includes the object based on the tilt angle, and an output unit that outputs data of the converted left-eye and right-eye images to the HMD. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2019 / 163031 Summary of the Invention [Problem to be solved by the invention]

[0006] In Patent Document 1, display image control is performed to rotate the image on the display unit, which also tilts in accordance with the tilt angle of the user's head. Therefore, when the user tilts his or her head, the entire image displayed on the display unit as seen by the user is tilted. Therefore, if there is a menu display, for example, the menu display will also be tilted, and no consideration is given to the fact that the menu display will be difficult to see.

[0007] In view of the above-mentioned problems, an object of the present invention is to provide an information display device and a display method therefor that are easy to use and that make it possible to make menu displays and the like easily visible even when the display unit is tilted. [Means for solving the problem]

[0008] As one example, the present invention is an information display device having a display unit, comprising an object data acquisition unit that acquires information about an AR object and generates the AR object; an additional object generation unit that generates an additional object related to the AR object; an attitude detection unit that detects the tilt of the information display device with respect to the direction of gravity in real space; and a display control unit that displays the AR object on the display unit so that it is tilted in accordance with the tilt of the information display device, and that displays the additional object on the display unit so that it is tilted in accordance with the tilt of the information display device when the tilt of the information display device is less than a predetermined value, and displays the additional object on the display unit so that it is not tilted, with the vertical reference being the same as that of the display unit, regardless of the tilt of the information display device when the tilt is equal to or greater than the predetermined value. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an information display device and a display method that are easy to use and that make it possible to make menu displays and the like easily visible even when the display unit is tilted. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 is a block diagram showing the hardware configuration of the HMD according to the first embodiment. [Figure 2] 1A and 1B are diagrams illustrating the appearance and wearing of an HMD according to a first embodiment. [Figure 3] FIG. 2 is a functional configuration diagram of an HMD according to the first embodiment. [Figure 4A] FIG. 2 is a conceptual diagram illustrating a visual recognition state of a virtual space using an HMD in the first embodiment. [Figure 4B] 4B is a diagram illustrating an image displayed on the display unit of the HMD in the state of FIG. 4A. FIG. [Figure 5A] FIG. 10 is a conceptual diagram illustrating a viewing state of a virtual space through an HMD when a user tilts their head by 30 degrees in Example 1. [Figure 5B] 5B is a diagram illustrating an image displayed on the display unit of the HMD with the AR object as the reference in the state of FIG. 5A. FIG. [Figure 5C] FIG. 5B is a diagram illustrating an image displayed on the display unit of the HMD in the state of FIG. 5A, with the display unit of the HMD as the reference; [Figure 6A] FIG. 10 is a conceptual diagram illustrating a viewing state of a virtual space through an HMD when a user tilts their head at 45 degrees in Example 1. [Figure 6B] 6B is a diagram illustrating an image displayed on the display unit of the HMD with the AR object as the reference in the state of FIG. 6A. FIG. [Figure 6C] FIG. 6B is a diagram illustrating an image displayed on a display unit based on the display unit of the HMD in the state of FIG. 6A. [Figure 7] 4 is an explanatory diagram showing the relationship between the tilt angle of the user's head and the image displayed on the display unit in the first embodiment. FIG. [Figure 8] 10 is a flowchart of an object display control process according to the first embodiment. [Figure 9A] FIG. 10 is an explanatory diagram of additional object display control in the first embodiment. [Figure 9B] 9B is a table summarizing display control of AR objects and additional objects according to the tilt angle α of the user's head in the state of FIG. 9A. [Figure 10]10A and 10B are diagrams illustrating display control of an AR object and an additional object according to a head tilt angle α of a user in the second embodiment. [Figure 11A] 10A and 10B are diagrams illustrating an image displayed on the display unit of the HMD in a state where a user wearing the HMD in Example 3 tilts his / her head to the right. [Figure 11B] 11B is a diagram illustrating an image displayed on the display unit when the user tilts his / her head further to the right from the state of FIG. 11A and the tilt of the user's head reaches a predetermined value or more. FIG. [Figure 11C] FIG. 11 is an explanatory diagram of additional object display control in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that in the following embodiment, an HMD will be described as an example of an information display device. [Example]

[0012] Fig. 1 is a block diagram showing the hardware configuration of an HMD 100 in this embodiment. In Fig. 1, 101 is a main control unit (CPU / MCU, etc.), 102 is a bus which is a path for sending and receiving commands and data, 103 is a RAM which serves as a work area when executing basic operation programs and other operation programs, and 110 is a storage unit which is a non-volatile storage medium such as Flash ROM / EEPROM / SSD / HDD.

[0013] Reference numeral 120 denotes an operation input unit that is a user operation interface, and includes operation keys 121 such as switches, a power key, and a volume key, and a touch sensor 122 such as a touch pad.

[0014] 130 is an image processing unit such as an image (video) processor, and has a display unit (display) 131, an image signal processing unit (image (video) signal processor) 132, a first image input unit 133 which is an external camera for capturing images in the forward direction, and a second image input unit 134 which is an internal camera for detecting the line of sight.

[0015] Reference numeral 140 denotes an audio processor, which includes an audio output unit (speaker) 141, an audio signal processing unit (audio signal processor) 142, and an audio input unit (microphone) 143.

[0016] Reference numeral 150 denotes a location information acquisition unit (for receiving location information signals such as GPS signals), and 160 denotes a sensor unit, which includes a gyro sensor 161, a geomagnetic sensor 162, and an acceleration sensor 163.

[0017] 170 is a communication unit (communication interface) consisting of a LAN (Wi-Fi (registered trademark)) communication unit, a mobile communication unit, a Bluetooth (registered trademark) communication unit, etc., and 180 is an expansion interface unit, such as a USB interface, which is used for data transmission and reception, charging, etc.

[0018] Fig. 2 shows the appearance and wearing of the HMD in this embodiment. In Fig. 2, the upper diagram is a top view of the HMD 100 worn on the head of user U1, the middle diagram is a front view, and the lower diagram is a side view. As shown in Fig. 2, the HMD 100 in this embodiment is a goggle-type, non-transparent HMD.

[0019] 2, the display unit 131 is of a non-transparent type and is placed inside the housing of the HMD 100 at a position visible to the user U1, and an image of the user's field of view range acquired by a first image input unit 133, which is an external camera, is displayed on the display unit 131. Note that the display unit 131 may be of a transparent type, in which case the outside world can be directly viewed through the display unit 131.

[0020] 133L is a first image input unit (L) which is an external camera on the left side, and 133R is a first image input unit (R) which is an external camera on the right side. Note that there may be only one first image input unit, and for example, a configuration may be such that one first image input unit 133 is provided in the center.

[0021] Furthermore, 122L is a touch sensor (L), 122R is a touch sensor (R) (not shown), and only one touch sensor may be provided on either the left or right side. Furthermore, 141L is an audio output unit (L) (stereo speaker L), and 141R is an audio output unit (R) (stereo speaker R) (not shown). Although not shown, a monaural microphone may be provided as the audio input unit 143.

[0022] Fig. 3 is a functional configuration diagram of the HMD in this embodiment. In Fig. 3, the storage unit 110 is composed of a basic operation program 1001, which is a basic program such as an OS, an application 1002 including applications (hereinafter abbreviated as apps) that execute the functions of this embodiment and other apps, and a various data storage area 1009, which is an area for storing various operation setting values ​​and various information (video / still images / audio, etc.).

[0023] The programs / applications stored in the storage unit 110 are deployed (loaded) into the RAM 103, and the main control unit 101 executes the deployed programs / applications through software processing, thereby executing the functional units of this embodiment and other functional units.

[0024] In FIG. 3, the executable programs to be loaded are written in RAM 103 by function. That is, the RAM 103 has a basic operation function unit 1101 that executes the basic operation program of the HMD 100, an object data acquisition unit 1111 that acquires information about an AR object (hereinafter, the object may be referred to as an OBJ) and generates an AR object to be displayed on the display unit 131 of the HMD 100 based on the acquired information, an attitude detection unit 1112 that detects the inclination of the HMD 100 relative to the direction of gravity in real space (the inclination of the head of the user U1), an additional object generation unit 1113 that generates additional information (additional object) related to the AR object generated by the object data acquisition unit 1111, and a display control unit 1114 that controls the display manner on the display unit 131 of the AR object generated by the object data acquisition unit 1111 and the additional object generated by the additional object generation unit 1113 based on the direction of gravity set in the virtual space in which the AR object is placed, the inclination of the HMD 100 relative to the direction of gravity in real space detected by the attitude detection unit 1112, and the operation mode of the HMD 100.

[0025] The RAM 103 also has a temporary storage area 1199 that is a temporary storage area for various pieces of information created / acquired by the application.

[0026] 4A and 4B are conceptual diagrams illustrating the viewing situation of the virtual space through the HMD in this embodiment.

[0027] 4A shows a case where user U1 is wearing HMD 100 with his / her head standing upright without tilting relative to the vertical direction, which is the direction of gravity G in real space. S1 is a virtual space that user U1 views by wearing HMD 100. In FIG. 4A, virtual space S1 is a space in which OBJ1 to OBJ3 and other objects are placed, and is not limited to the inside of an ellipse.

[0028] OBJ1 is an AR object of a person, OBJ2 is an AR object of a cup, and OBJ3 is an AR object of a table, each of which is positioned based on the world coordinate system. 131V is the visible range in the virtual space S1 that is viewed by the user U1 wearing the HMD 100, and moves in conjunction with the movement of the HMD 100 worn by the user U1. Note that g is the direction of gravity set in the virtual space S1 in which the AR objects OBJ1 to OBJ3 are positioned.

[0029] By displaying the AR objects OBJ1 to OBJ3 on the display unit 131 of the HMD 100, the user U1 is placed in a state of visually recognizing a predetermined range of the virtual space S1 including the AR objects OBJ1 to OBJ3 through the viewable range 131V.

[0030] Fig. 4B is a diagram illustrating an image displayed on the display unit of the HMD in the state of Fig. 4A. In Fig. 4B, image 131V1 displayed in the state of Fig. 4A is displayed on display unit 131. Note that OBJM is an additional object mainly composed of text, such as a menu or a pop-up for explaining an object.

[0031] 5A, 5B, and 5C are conceptual diagrams illustrating the viewing state of the virtual space through the HMD when the user tilts their head by 30 degrees in this embodiment.

[0032] 5A, 5B, and 5C, the same components as those in FIGS. 4A and 4B are denoted by the same reference numerals, and their description will be omitted. FIG. 5A shows a case where user U1 tilts his / her head 30 degrees to the right from the state shown in FIG. 4A, i.e., the HMD 100 rotates 30 degrees clockwise within the XY plane of the real space. In other words, the angle of the vertical axis of the head relative to the vertical direction on a plane parallel to the display unit 131 is 30 degrees.

[0033] When user U1 tilts his / her head 30 degrees to the right, HMD 100 also tilts, and viewable range 131V also tilts 30 degrees to the right. That is, viewable range 131V rotates 30 degrees clockwise within the xy plane of virtual space S1. To create the feeling of being immersed in the virtual world, the AR objects OBJ1-3 and additional objects OBJM arranged based on the world coordinate system are fixed in position within virtual space S1 even if user U1 or viewable range 131V moves. That is, the AR objects OBJ1-3 and additional objects OBJM remain displayed with the gravity direction g of the virtual space as the vertical reference.

[0034] FIG. 5B shows a state in which an image 131V3 displayed within the display unit 131 in the state of FIG. 5A is drawn with reference to each of the AR objects OBJ1 to OBJ3.

[0035] Fig. 5C shows the state in Fig. 5A, in which image 131V3 displayed within display unit 131 is drawn with display unit 131 as the reference. That is, it shows the image displayed on display unit 131 as seen by user U1 with his head tilted.

[0036] When user U1 tilts his / her head to the right, the display unit 131 and viewable range 131V of the HMD 100 worn on the user U1's head also tilt to the right, but the AR objects OBJ1-3 arranged in the virtual space S1 maintain their orientation based on the direction of gravity g of the virtual space S1 to create the feeling of being immersed in the virtual world. Therefore, as shown in Fig. 5C, the AR objects OBJ1-3 and additional object OBJM displayed on the display unit 131 are displayed tilted to the left, opposite to the right to which user U1 tilts his / her head. This allows the objects themselves to appear fixed even when user U1 tilts his / her head, giving the user the feeling that they are actually present in the virtual world.

[0037] 6A, 6B, and 6C are conceptual diagrams illustrating the viewing state of the virtual space through the HMD when the user tilts their head at 45 degrees in this embodiment.

[0038] In Figures 6A, 6B, and 6C, the same components as those in Figures 5A, 5B, and 5C are denoted by the same reference numerals, and their description will be omitted. Figure 6A shows a case where user U1 tilts his / her head further to the right from the state in Figure 5A, tilting it 45 degrees. In other words, it shows a case where the HMD 100 is rotated 45 degrees clockwise within the XY plane in real space.

[0039] 6B and 6C show the same display as in Fig. 5B and 5C within the display unit 131. Fig. 6B and 6C differ from Fig. 5B and 5C in that the additional object OBJM, unlike the AR objects OBJ1 to OBJ3, has the same tilt as the user's head, and is displayed facing the user directly.

[0040] In this embodiment, the predetermined value for the tilt angle of the user's head is set to 40 degrees, and if the tilt angle is less than 40 degrees, the AR objects OBJ1 to 3 and the additional object OBJM are both displayed on the display unit 131 in a state tilted in accordance with the tilt of the head, as shown in Fig. 5C. On the other hand, if the tilt angle is 40 degrees or more, the additional object OBJM, unlike the AR objects OBJ1 to 3, has the same tilt as the user's head, and is displayed so as to face the user U1 who has tilted his or her head, as shown in Fig. 6C.

[0041] This is because if the difference between the tilt of the user's head and the tilt of the additional object is large, the additional object becomes difficult to read. Therefore, in this embodiment, the predetermined value of the tilt angle is set to 40 degrees, and if the head tilt angle is 40 degrees or more, the tilt of the additional object is made the same as the tilt of the user's head. This has the effect of making the additional object easier to read when it is a character or the like. Note that the predetermined value of the tilt angle is preferably a maximum of approximately 30 to 40 degrees. The predetermined value of the tilt angle may also be 0 degrees. In other words, the additional object OBJM may always be tilted at the same angle as the user's head, and may be displayed so as to face the user U1 who has tilted his or her head. The predetermined value of the tilt angle may also be changeable by user setting.

[0042] Fig. 7 is an explanatory diagram showing the relationship between the tilt angle of the user's head and the image displayed on the display unit in this embodiment. Fig. 7 shows the display state of the AR object and the additional object displayed on the display unit when the tilt angle of the user's head is changed in 15-degree increments, and shows the case where the predetermined tilt angle is set to 40 degrees.

[0043] 7, at tilt angles of 15 degrees and 30 degrees, which are less than the predetermined value of 40 degrees, the AR objects OBJ1 to 3 and the additional object OBJM are displayed in a similar state tilted to the left, opposite to the right to which the user has tilted their head. On the other hand, at tilt angles of 45 degrees to 90 degrees, which are the predetermined value of 40 degrees or more, the AR objects OBJ1 to 3 are displayed in a state tilted to the left, opposite to the right to which the user has tilted their head, but the additional object OBJM is displayed in a state tilted in the same way as the head tilt, i.e., facing directly from the user who has tilted their head.

[0044] 8 is a flowchart of the object display control process in this embodiment. In FIG. 8, first, in step S101, the display control unit 1114 checks whether the state of the HMD 100 is the AR object display On mode. If it is the AR object display On mode, the process proceeds to step S102. If it is not the AR object display On mode, the process ends.

[0045] In step S102, the posture detection unit 1112 detects the position and orientation direction of the HMD 100 in real space, and the tilt of the HMD 100. Here, the tilt of the HMD 100 refers to the angle between the vertical direction of the display unit 131 and the vertical direction in a plane parallel to the display unit 131, in other words, the angle between the vertical axis of the head and the vertical direction in a plane parallel to the display unit 131, and is 0° when the head is upright.

[0046] Then, in step S103, the object data acquisition unit 1111 acquires information about the AR object to be placed in the virtual space S1 from a server device on the network. Then, in step S104, the object data acquisition unit 1111 generates an AR object to be displayed on the display unit 131.

[0047] Then, in step S105, the display control unit 1114 displays the AR object generated in the processing of S104 on the display unit 131. Note that the AR object displayed here is displayed with the gravity direction g of the virtual space as the vertical reference, regardless of the tilt of the HMD 100 detected in the processing of S102. In other words, the AR object is displayed so as to maintain an orientation based on the gravity direction g of the virtual space S1. That is, a rotation process is performed so that the AR object is tilted in the opposite direction by the same amount as the tilt of the HMD 100 detected in the processing of S102, and then displayed.

[0048] Next, in step S106, the additional object generation unit 1113 checks whether the state of the HMD 100 is the additional object display on mode. If it is the additional object display on mode, the process proceeds to step S107. If it is not the additional object display on mode, the process returns to step S101.

[0049] In step S107, the additional object generating unit 1113 generates an additional object related to the AR object generated in the process of S104.

[0050] Then, in step S108, it is confirmed whether the tilt of the HMD 100 with respect to the vertical direction detected in the process of S102 is less than a predetermined value. If the tilt with respect to the vertical direction is less than the predetermined value, the process proceeds to step S109. If the tilt with respect to the vertical direction is not less than the predetermined value, the process proceeds to step S110.

[0051] In step S109, the display control unit 1114 displays the additional object generated in S107 using the gravity direction g of the virtual space as a vertical reference, regardless of the tilt of the HMD 100 detected in the process of S102. That is, like the AR object displayed in the process of S105, the additional object is displayed in a state where it is tilted in the opposite direction to the tilt of the HMD 100 detected in the process of S102 by the same amount.

[0052] In step S110, the display control unit 1114 displays the additional object generated in S107 with the same vertical reference as the display unit 113. That is, the additional object is displayed in the display unit 131 in a posture that does not have any tilt.

[0053] 9A and 9B are explanatory diagrams of additional object display control in this embodiment. Fig. 9A is a diagram showing a state in which the user U1 tilts his / her head α degrees to the right (clockwise). That is, α is the tilt angle of the user U1's head with respect to the vertical direction, which is the direction of gravity G in the real space, and shows a case in which the HMD 100 rotates α degrees clockwise within the XY plane in the real space.

[0054] FIG. 9B is a table summarizing the display control of the AR object and the additional object according to the tilt angle α of the user's head in this embodiment in the state of FIG. 9A.

[0055] 9B, when the tilt angle α of the user's head, i.e., the tilt angle α of the HMD, is 0≦α<a predetermined value, the AR object is displayed based on the gravity direction g of the virtual space S1, regardless of the tilt of the HMD 100. Similarly, the additional object is displayed based on the gravity direction g of the virtual space S1, regardless of the tilt of the HMD 100.

[0056] On the other hand, when the tilt α of the HMD is a predetermined value ≦ α, the AR object is similarly displayed based on the gravity direction g of the virtual space S1 regardless of the tilt of the HMD 100, but the additional object is displayed in accordance with the tilt of the HMD 100 so that the relative tilt difference with the HMD 100 is 0.

[0057] As described above, according to this embodiment, in order to create a state of immersion in the virtual world, the AR object, which is the main image, is tilted according to the tilt angle of the user's head, i.e., the tilt angle of the display unit, but additional objects such as menu displays are not tilted, thereby making it possible to make menu displays and the like easily viewable even when the display unit is tilted, and a user-friendly information display device and display method therefor can be provided. [Example]

[0058] In this embodiment, a modified example of the display control of the additional object when the user tilts his / her head will be described.

[0059] FIG. 10 is an explanatory diagram of display control of the AR object and the additional object according to the tilt angle α of the user's head in this embodiment.

[0060] 10, similar to the state in FIG. 9A, it is assumed that user U1 tilts his / her head to the right (clockwise) by α degrees. When multiple predetermined values ​​are set such that the relationship is predetermined value 1<predetermined value 2<predetermined value 3<...<predetermined value n, if the tilt α of the HMD is 0≦α<predetermined value 1, the AR object is displayed based on the gravity direction g of the virtual space S1 regardless of the tilt of the HMD 100. Furthermore, the additional object is displayed with a tilt of (predetermined value 1÷2) with respect to the vertical direction. For example, if predetermined value 1 is 30 degrees and α is 0≦α<30, the additional object is displayed with a tilt of (30÷2)=15 degrees.

[0061] Furthermore, when the tilt α of the HMD is predetermined value 1≦α<predetermined value 2, the AR object is similarly displayed based on the gravity direction g of the virtual space S1 regardless of the tilt of the HMD 100, but the additional object is displayed with a tilt of [predetermined value 1+{(predetermined value 2-predetermined value 1)÷2}] with respect to the vertical direction. For example, if predetermined value 1 is 30 degrees and predetermined value 2 is 60 degrees, and α is 30≦α<60, the additional object is displayed with a tilt of [30+{(60-30)÷2}]=45 degrees.

[0062] Similarly, when the tilt α of the HMD is predetermined value 2≦α<predetermined value 3, the AR object is similarly displayed based on the gravity direction g of the virtual space S1 regardless of the tilt of the HMD 100, but the additional object is displayed with a tilt of [[predetermined value 2+{(predetermined value 3-predetermined value 2)÷2}] with respect to the vertical direction. For example, if predetermined value 2 is 60 degrees and predetermined value 3 is 90 degrees, and α is 60≦α<90, the additional object is displayed with a tilt of [60+{(90-60)÷2}]=75 degrees.

[0063] Finally, when the tilt α of the HMD is a predetermined value n≦α≦180, the AR object is similarly displayed based on the gravity direction g of the virtual space S1 regardless of the tilt of the HMD 100, but the additional object is displayed with a tilt of [predetermined value n+{(180-predetermined value n)÷2}] relative to the vertical direction.

[0064] In this way, in this embodiment, the additional object is displayed with a stepwise inclination angle according to the inclination α of the user's head, that is, the inclination α of the display unit of the HMD.

[0065] As a result, according to this embodiment, the difference between the tilt of the user's head and the tilt of the additional object can be minimized without increasing the difference in tilt between the AR object and the additional object. Also, it is possible to prevent the display mode of the additional object from fluctuating minutely in response to small changes in the tilt of the user's head, which produces a state of immersion in the virtual world and, if the additional object is text, makes it easier to read. [Example]

[0066] In this embodiment, a further modification of the display control of the additional object when the user tilts his / her head will be described.

[0067] 11A, 11B, and 11C are diagrams illustrating images displayed on the display unit of the HMD in this embodiment when the user wearing the HMD tilts his / her head to the right (clockwise).

[0068] 11A, an image 131V5 is displayed on the display unit 131 of the HMD. Similar to FIG. 4A, the image 131V5 displays AR objects OBJ1 to OBJ3 and an additional object OBJP1 which is an explanatory pop-up related to the AR objects (for example, a diagram, table, or comment explaining AR object OBJ2). FIG. 11A shows a case where the tilt of the user's head is less than a predetermined value, and the additional object OBJP1 also maintains its position based on the direction of gravity in the virtual space S1.

[0069] 11B shows a case where the user tilts his / her head further to the right from the state shown in FIG. 11A, and the tilt of the user's head reaches a predetermined value or more. In FIG. 11B, because the tilt of the user's head reaches a predetermined value or more, the display control described in the first embodiment causes the additional object OBJP1 to be displayed with the same vertical reference as the HMD. However, because the AR objects OBJ1 to OBJ3 maintain their positions based on the direction of gravity in the virtual space S1, the display of the additional object OBJP1 and the display of the AR objects may overlap, as shown in FIG. 11B.

[0070] Therefore, in this embodiment, as shown in Fig. 11C, in order to eliminate the overlap between the display of the additional object OBJP1 and the display of each AR object in the state of Fig. 11B, the additional object OBJP1 is moved and displayed to a position where it does not overlap with the display of each AR object. That is, the additional object can be simultaneously subjected to not only the rotation process as in the first embodiment but also the movement process. This makes it possible to make the menu display and the like easier to see even when the user tilts their head.

[0071] The destination of the additional object OBJP1 during the above processing may be a position a predetermined distance away in a predetermined direction. Alternatively, since the display control unit 1114 can grasp the area occupied by each AR object within the display unit 131 based on the shape of each AR object, the additional object OBJP1 may be moved to a position that avoids the area occupied by each AR object within the display unit 131.

[0072] Furthermore, when the process of moving the additional object OBJP1 is performed, in order to prevent the destination position of the additional object from fluctuating minutely in response to small changes in the tilt of the user's head, control may be performed so that the relative positional relationship between the destination position of the additional object and the AR object is fixed when the change in the tilt of the user's head is slight. This makes it possible to prevent the visibility of the additional object from being impaired even if the user's head moves slightly.

[0073] As described above, according to this embodiment, when the user tilts their head, i.e., tilts the display unit, the AR object is tilted according to the tilt angle of the user's head, i.e., the tilt angle of the display unit, and additional objects such as menu displays are not tilted. In addition, by moving and displaying them in a position that does not overlap with the AR object, it is possible to make menu displays and the like easier to see, and a more user-friendly information display device and display method can be provided.

[0074] Although the embodiments have been described above, the present invention is not limited to the above-described embodiments and includes various modifications. For example, in the embodiments, an HMD has been used as an example of an information display device, but the present invention is not limited to an HMD and can also be applied to information display devices such as smartphones in which the angle of the display screen relative to the vertical direction can be changed.

[0075] Furthermore, in addition to the process of tilting the object according to the tilt angle of the user's head or the tilt angle of the display unit, the user may be allowed to select, using the operation input unit, to fix the tilt of the object. For example, when an AR object and an additional object are being displayed tilted according to the tilt of the display unit, if the user selects to fix the display, the display will continue to be at the selected tilt regardless of the subsequent tilt of the display unit. Some users may not prefer that the object be rotated in accordance with the tilt of the display unit. Therefore, allowing the user to select to fix the display can improve user convenience.

[0076] Furthermore, although the functions of the present invention described in the embodiments are implemented in software by a CPU or the like interpreting and executing an operating program that implements each function, some or all of them may be implemented in hardware, for example, by designing an integrated circuit, a general-purpose processor, or an application-specific processor. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. Hardware and software may also be used in combination. Some or all of the functions may also be implemented by a server. Note that the server may be any type of server, as long as it can communicate with other components via communications to execute its functions. For example, the server may be a local server, a cloud server, an edge server, or an online service. Information such as programs, tables, and files that implement each function may be stored in a memory, a recording device such as a hard disk or solid-state drive (SSD), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communications network.

[0077] The programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each process is performed may also be changed.

[0078] Furthermore, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. [Explanation of symbols]

[0079] 100: HMD (head mounted display device), 101: main control unit, 103: RAM, 110: storage unit, 120: operation input unit, 130: image processing unit, 131: display unit, 131V: visual range, 140: audio processing unit, 150: position information acquisition unit, 160: sensor unit, 170: communication unit, 180: extension interface unit, 1001: basic operation program, 1002: application, 1009: various data storage area, 1101: basic operation function unit, 1111: object data acquisition unit, 1112: posture detection unit, 1113: additional object generation unit, 1114: display control unit, 1199: temporary storage area, U1: user, S1: virtual space, OBJ1 to 3: AR object, OBJM, OBJP1: additional object

Claims

1. An information display device having a display unit, an object data acquisition unit that acquires information about an AR object and generates an AR object to be displayed on the display unit based on the acquired information; an additional object generating unit that generates an additional object that is additional information related to the AR object generated by the object data obtaining unit; an attitude detection unit that detects an inclination of the information display device with respect to the direction of gravity in real space; a display control unit that controls display of the AR object and the additional object on the display unit based on a direction of gravity set in a virtual space in which the AR object is placed and a tilt of the information display device, The display control unit displays the AR object on the display unit so that it is tilted in accordance with the tilt of the information display device, and displays the additional object on the display unit so that it is tilted in accordance with the tilt of the information display device when the tilt of the information display device is less than a predetermined value, and displays the additional object on the display unit so that it is not tilted, with the vertical reference being the same as that of the display unit, regardless of the tilt of the information display device, when the tilt of the information display device is equal to or greater than the predetermined value.

2. 2. The information display device according to claim 1, The information display device, wherein the display control unit displays the additional object on the display unit with a gradual inclination angle according to the inclination of the information display device.

3. 3. The information display device according to claim 2, The information display device is characterized in that, when the relationship of predetermined value 1 < predetermined value 2 holds, the display control unit displays the additional object on the display unit with a slope of (predetermined value 1 ÷ 2) when the slope α of the information display device is 0≦α<predetermined value 1, and displays the additional object on the display unit with a slope of [predetermined value 1 + {(predetermined value 2 - predetermined value 1) ÷ 2}] when the slope α of the information display device is 0≦α<predetermined value 2.

4. 2. The information display device according to claim 1, The information display device is characterized in that the display control unit moves and displays the additional object to a position where it does not overlap with the AR object.

5. 2. The information display device according to claim 1, The information display device is a head-mounted display device, the tilt of the information display device is a tilt angle of the head of a user wearing the head-mounted display device, the control of displaying the information display device so that the display unit is tilted in accordance with the tilt of the information display device is control of displaying the information display device so that the display unit is tilted in the opposite direction to the tilt angle of the head by the same amount, The control of displaying the additional object on the display unit without tilting it regardless of the tilt of the information display device is a control of displaying the additional object at the same tilt as the tilt of the head and facing directly at the user. to this point

6. A display method for an information display device having a display unit, comprising: Acquire information about the AR object, and generate the AR object based on the acquired information; generating an additional object that is additional information related to the generated AR object; Detecting a tilt of the information display device with respect to a direction of gravity in real space; displaying the AR object on the display unit in a tilted state in accordance with a tilt of the information display device; a display method characterized in that, when the tilt of the information display device is less than a predetermined value, the additional object is displayed in a tilted state on the display unit in accordance with the tilt of the information display device, and when the tilt of the information display device is equal to or greater than a predetermined value, the additional object is displayed in a non-tilted state on the display unit, with the vertical reference being the same as that of the display unit, regardless of the tilt of the information display device.

7. 7. The display method according to claim 6, A display method, characterized in that the additional object is displayed on the display unit with a gradual inclination angle according to the inclination of the information display device.

8. 8. The display method according to claim 7, A display method characterized by the fact that, when the relationship is predetermined value 1<predetermined value 2, if the tilt α of the information display device is 0≦α<predetermined value 1, the additional object is displayed on the display unit with a tilt of (predetermined value 1 ÷ 2), and if the relationship is predetermined value 1≦α<predetermined value 2, the additional object is displayed on the display unit with a tilt of [predetermined value 1 + {(predetermined value 2 - predetermined value 1) ÷ 2}].

9. 7. The display method according to claim 6, A display method characterized in that the additional object is moved to a position where it does not overlap with the AR object and is displayed.

10. 7. The display method according to claim 6, The information display device is a head-mounted display device, the tilt of the information display device is a tilt angle of the head of a user wearing the head-mounted display device, the control of displaying the information display device so that the display unit is tilted in accordance with the tilt of the information display device is control of displaying the information display device so that the display unit is tilted in the opposite direction to the tilt angle of the head by the same amount, A display method characterized in that the control of displaying the additional object on the display unit without tilting it regardless of the tilt of the information display device is control of displaying the additional object at the same tilt as the tilt of the head and facing directly at the user.

11. An information display device, a display, a sensor that detects the tilt of the information display device, and a processor; The processor: generating an AR object and an additional object which is additional information related to the AR object; when the tilt of the information display device detected by the sensor is less than a predetermined value, control is performed so that the AR object and the additional object are displayed on the display in a tilted state according to the tilt of the information display device; When the tilt of the information display device detected by the sensor is equal to or greater than a predetermined value, the AR object is displayed on the display so as to be tilted in accordance with the tilt of the information display device, and the additional object is controlled so as to be displayed on the display without being tilted regardless of the tilt of the information display device.

12. 12. The information display device according to claim 11, The information display device is characterized in that the additional object is composed of characters.

13. 12. The information display device according to claim 11, the AR object is an object representing a person or an object, The information display device is characterized in that the additional object is an object representing a menu including characters.

14. 12. The information display device according to claim 11, An information display device comprising an interface that is used by a user to set the predetermined value.

Images