Information display device and display method thereof

Abstract

The invention provides an information display device and a display method thereof. The information display device is easy to use and enables menu display and the like to be easily viewed even when a display part is inclined. To this end, an information display device having a display unit is configured to include: an object data acquisition unit that acquires information about an AR object and generates the AR object; an additional object generation unit that generates an additional object associated with the AR object; a posture detection unit that detects an inclination angle of the information display device with respect to the gravity direction of the real space; and a display control unit that displays the AR object in an inclined state on the display unit in accordance with the inclination angle of the information display device, and displays the additional object in an inclined state on the display unit in accordance with the inclination angle of the information display device when the inclination angle of the information display device is less than a predetermined value. When the predetermined value is greater than or equal to the predetermined value, the information is displayed on the display unit in a non-tilt manner without causing the vertical reference to be the same as the display unit regardless of the tilt angle of the information display device.

Description

Technical Field

[0001] The invention relates to an information display device and a display method thereof. Background Art

[0002] As an information display device, there is a head mounted display device (hereinafter referred to as HMD) that is worn on the head to view and listen to information. HMD can display images of virtual objects (AR objects: Argument Reality Objects) such as images and texts superimposed on real space for people to watch. In recent years, its application in the fields of gaming, work assistance, etc. is expanding.

[0003] Here, in order for the user to express a state of being immersed in the virtual world by wearing the HMD, the virtual space seen by the user needs to change according to the movement of the user's head.

[0004] Background technology in this technical field includes Patent Document 1. Patent Document 1 discloses an image processing device for presenting a stereoscopic image through an HMD, the structure of which includes: a reference image acquisition unit that acquires a reference image composed of a pair of left-eye images and right-eye images, the pair of images representing an object seen from two viewpoints spaced apart in the horizontal direction; an acquisition unit that acquires an inclination angle between a vertical axis of a head of a user wearing the HMD and a vertical direction in a plane parallel to the screen; an image conversion unit that performs conversion processing to rotate the left-eye image and the right-eye image around a common axis in a virtual space containing the object based on the inclination angle; and an output unit that outputs data of the left-eye image and the right-eye image after the conversion processing to the HMD.

[0005] Prior art literature Patent Literature Patent Document 1: International Publication No. 2019 / 163031 Summary of the invention

[0006] Technical problem to be solved by the invention Patent Document 1 performs display image control so that the image on the display unit that is tilted is rotated according to the tilt angle of the user's head. As a result, when the user tilts his head, all the images displayed by the user on the display unit become tilted. Therefore, for example, when a menu is displayed, the menu will also become tilted, and the fact that the menu will become difficult to see is not considered.

[0007] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an easy-to-use information display device and a display method thereof, which can make menu displays and the like easy to see even when a display unit is tilted.

[0008] Technical means of solving problems An example of the present invention is an information display device with a display unit, which is configured to include: an object data acquisition unit, which acquires information about an AR object and generates an AR object; an additional object generation unit, which generates an additional object associated with the AR object; a posture detection unit, which detects the inclination of the information display device relative to the gravity direction of the real space; and a display control unit, which displays the AR object in a tilted state on the display unit according to the inclination of the information display device, and displays the additional object in a tilted state on the display unit according to the inclination of the information display device when the inclination of the information display device is less than a specified value, and displays the additional object in a tilted state on the display unit according to the inclination of the information display device when the inclination of the information display device is greater than the specified value, making the vertical reference the same as the display unit regardless of the inclination of the information display device so that the object is displayed in a non-tilted manner on the display unit.

[0009] Effects of the Invention According to the present invention, it is possible to provide an easy-to-use information display device and a display method thereof that can make menu displays and the like easy to see even when the display unit is tilted. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a block diagram showing the hardware configuration of the HMD in the first embodiment.

[0011] Figure 2 This is a diagram showing the appearance and wearing of the HMD in Example 1.

[0012] Figure 3 This is a functional configuration diagram of the HMD in the first embodiment.

[0013] Figure 4A This is a conceptual diagram for explaining a viewing situation of a virtual space formed by the HMD in the first embodiment.

[0014] Figure 4B It is explained in Figure 4A FIG. 1 is a diagram of an image displayed on the display unit of the HMD in a state of .

[0015] Figure 5A This is a conceptual diagram for explaining a viewing state of a virtual space formed by the HMD when the user in Example 1 tilts his head by 30 degrees.

[0016] Figure 5B It is explained in Figure 5A FIG. 1 is a diagram of an image displayed on the display unit of the HMD with the AR object as a reference in a state.

[0017] Figure 5C It is explained in Figure 5AFIG. 1 is a diagram of an image displayed on a display unit based on the display unit of the HMD in a state.

[0018] Fig. 6A This is a conceptual diagram for explaining a viewing state of a virtual space formed by the HMD when the user in Example 1 tilts his head by 45 degrees.

[0019] Figure 6B It is explained in Fig. 6A FIG. 1 is a diagram of an image displayed on the display unit of the HMD with the AR object as a reference in a state.

[0020] Figure 6C It is explained in Fig. 6A FIG. 1 is a diagram of an image displayed on a display unit based on the display unit of the HMD in a state.

[0021] Figure 7 This is an explanatory diagram showing the relationship between the tilt angle of the user's head and the image displayed on the display unit in the first embodiment.

[0022] Figure 8 This is a flowchart of the object display control process in the first embodiment.

[0023] Fig. 9A This is an explanatory diagram of the additional object display control in Example 1.

[0024] Fig. 9B is Fig. 9A A summary table of display control of AR objects and additional objects corresponding to the inclination angle α of the user's head in the state.

[0025] Fig.10 This is a diagram for explaining display control of an AR object and an additional object according to the tilt angle α of the user's head in the second embodiment.

[0026] Fig.11A This is a diagram for explaining an image displayed on the display unit of the HMD when the user wearing the HMD tilts his head to the right in the third embodiment.

[0027] Fig. 11B It is explained from Fig.11A , and the user further tilts his head to the right and the tilt angle of the user's head reaches a predetermined value or more.

[0028] Fig. 11C This is an illustration of the additional object display control in Example 3. DETAILED DESCRIPTION

[0029] Hereinafter, embodiments of the present invention will be described using the drawings. In the following embodiments, HMD will be described as an example of an information display device.

[0030] Example 1 Figure 1 : is a block diagram showing the hardware configuration of the HMD 100 in this embodiment. Figure 1 In the figure, 101 is a main control unit (CPU / MCU, etc.), 102 is a bus as a path for sending and receiving commands and data, 103 is a RAM as a working area when the basic action program and other action programs are executed, and 110 is a non-volatile storage medium such as Flash ROM / EEPROM / SSD / HDD, i.e., a storage unit.

[0031] Reference numeral 120 denotes an operation input unit as a user operation interface, and includes operation buttons 121 such as a switch, a power button, and a volume button, and a touch sensor 122 such as a touch pad.

[0032] Reference numeral 130 denotes an image processing unit such as an image (video) processor, and includes a display unit (display) 131, an image signal processing unit (image (video) signal processor) 132, a first image input unit 133 which is an external camera for shooting forward, and a second image input unit 134 which is an internal camera for detecting line of sight.

[0033] Reference numeral 140 denotes a sound processing unit (audio processor), which includes a sound output unit (speaker) 141 , a sound signal processing unit (audio signal processor) 142 , and a sound input unit (microphone) 143 .

[0034] Reference numeral 150 denotes a position information acquisition unit (for receiving a position information signal such as a GPS signal). Reference numeral 160 denotes a sensor unit including a gyro sensor 161 , a geomagnetic sensor 162 , and an acceleration sensor 163 .

[0035] Reference numeral 170 denotes a communication unit (communication interface) such as a LAN (Wi-Fi (registered trademark)) communication unit and / or a mobile communication unit, a Bluetooth (registered trademark) communication unit, etc. Reference numeral 180 denotes an expansion interface unit such as a USB interface for transmitting and receiving data and charging.

[0036] Figure 2 1 is the appearance and wearing diagram of the HMD in this embodiment. Figure 2 In the figure, the upper figure is a top view of the state where the HMD 100 is worn on the head of the user U1, the middle figure is a front view, and the lower figure is a side view. Figure 2 The one shown is a goggle type, which is a non-see-through HMD.

[0037] Figure 2In the embodiment, the display unit 131 is a non-transparent type, and is arranged at a position visible to the user U1 inside the housing of the HMD 100, and an image of the user's visual field obtained by an external camera, i.e., the first image input unit 133, is displayed on the display unit 131. Alternatively, the display unit 131 may be a transparent type, in which case the outside world can be directly seen through the display unit 131.

[0038] 133L is the left external camera, that is, the first image input unit (L), and 133R is the right external camera, that is, the first image input unit (R). In addition, there may be only one first image input unit, for example, a structure in which one first image input unit 133 is provided in the center.

[0039] In addition, 122L is a touch sensor (L), 122R is a touch sensor (R) (illustration omitted), and the touch sensor may be only one of the left and right. In addition, 141L is a sound output unit (L) (stereo speaker L), and 141R is a sound output unit (R) (stereo speaker R) (illustration omitted). In addition, although the illustration is omitted, a monaural microphone may be provided as the sound input unit 143.

[0040] Figure 3 : is a functional structure diagram of the HMD in this embodiment. Figure 3 In the embodiment, the storage unit 110 includes: basic programs such as OS, namely basic action programs 1001, applications 1002 including applications that execute the functions of this embodiment (hereinafter referred to as applications) and other applications, and various data storage areas 1009 that store various action setting values ​​and various information (moving images / still images / sounds, etc.).

[0041] The program / application stored in the storage unit 110 is loaded into the RAM 103 , and the main control unit 101 executes the functional unit of the present embodiment and other functional units by executing software processing of the loaded program / application.

[0042] Figure 3In the example, the execution programs loaded in the RAM 103 are shown by function. That is, the RAM 103 has a basic action function unit 1101 for executing the basic action program of the HMD 100, an object data acquisition unit 1111, a posture detection unit 1112, an additional object generation unit 1113, and a display control unit 1114. The basic action function unit 1101 is used to execute the basic action program of the HMD 100, and the object data acquisition unit 1111 is used to acquire information about an AR object (hereinafter sometimes referred to as an object as OBJ) and generate an AR object (AR object) to be displayed on the display unit 131 of the HMD 100 based on the acquired information. The display control unit 1114 controls the display mode of the AR object generated by the object data acquisition unit 1111 and the additional object generated by the additional object generation unit 1113 on the display unit 131 based on the gravity direction set in the virtual space where the AR object is located, the inclination of the HMD 100 detected by the posture detection unit 1112 with respect to the gravity direction of the real space, and the action mode of the HMD 100.

[0043] In addition, the RAM 103 has a temporary storage area 1199 which is a temporary storage area for various information generated / acquired by applications.

[0044] Figure 4A , Figure 4B 1 is a conceptual diagram for explaining a viewing state of a virtual space formed by an HMD in this embodiment.

[0045] Figure 4A , the user U1 is shown with his head standing upright without tilting with respect to the vertical direction, that is, the gravity direction G in the real space, and wearing the HMD 100 . S1 is a virtual space viewed by the user U1 wearing the HMD 100 . Figure 4A In the example, virtual space S1 is the configuration space of OBJ1 to 3 and other objects, and is not limited to the ellipse.

[0046] OBJ1 is an AR object of a person, OBJ2 is an AR object of a cup, and OBJ3 is an AR object of a table, which are respectively configured based on the world coordinate system. 131V is the viewing range of the virtual space S1 seen by the user U1 wearing the HMD 100, and moves in conjunction with the movement of the HMD 100 worn by the user U1. In addition, g is the gravity direction set in the virtual space S1 where the AR objects OBJ1 to 3 are located.

[0047] By displaying the AR objects OBJ1 to 3 on the display unit 131 of the HMD 100 , the user U1 is in a state of viewing a predetermined range of the virtual space S1 including the AR objects OBJ1 to 3 via the viewing range 131V.

[0048] Figure 4B It is explained in Figure 4A FIG. 1 is a diagram of an image displayed on the display unit of the HMD in a state of . Figure 4B In the example, the display unit 131 displays the following Figure 4A OBJM is an additional object mainly composed of text, such as a menu or a pop-up box for object description.

[0049] Figure 5A , Figure 5B , Figure 5C This is a conceptual diagram for explaining a viewing state of a virtual space formed by the HMD when the user tilts his head 30 degrees in this embodiment.

[0050] Figure 5A , Figure 5B , Figure 5C In Figure 4A , Figure 4B The same structures are denoted by the same symbols, and their description is omitted. Figure 5A Shows from Figure 4A In other words, the vertical axis of the head is at an angle of 30 degrees to the vertical direction in a plane parallel to the display unit 131.

[0051] Since the user U1 tilts his head 30 degrees to the right, the HMD 100 also tilts, and the viewing range 131V also tilts 30 degrees to the right. That is, the viewing range 131V rotates 30 degrees clockwise in the xy plane of the virtual space S1. For each AR object OBJ1 to 3 and additional object OBJM configured based on the world coordinate system, in order to show the state of immersion in the virtual world, even if the user U1 and the viewing range 131V move, the position of each object in the virtual space S1 is fixed. That is, each AR object OBJ1 to 3 and additional object OBJM remains displayed with the gravity direction g of the virtual space as the vertical reference.

[0052] Figure 5B is Figure 5A In the state, the state of the image 131V3 displayed in the display unit 131 is drawn based on each AR object OBJ1~3.

[0053] Figure 5C is Figure 5AIn the state of , the state of the image 131V3 displayed in the display unit 131 is drawn with the display unit 131 as a reference. That is, the image displayed on the display unit 131 as seen by the user U1 who tilts his head (tilts his head) is shown.

[0054] Since the user U1 tilts his head to the right, the display unit 131 and the viewing range 131V of the HMD 100 worn on the head of the user U1 also tilt to the right, but the AR objects OBJ1 to OBJ3 arranged in the virtual space S1 maintain their postures based on the gravity direction g of the virtual space S1 in order to express the state of immersion in the virtual world. Figure 5C As shown, each AR object OBJ1 to OBJ3 and the additional object OBJM displayed on the display unit 131 are displayed in a state of being tilted to the left, which is opposite to the right direction in which the user U1 tilts his head. Thus, even if the user U1 tilts his head, the object itself appears to be fixed, so that the user can feel that the object actually exists in the virtual world.

[0055] Fig. 6A , Figure 6B , Figure 6C This is a conceptual diagram for explaining a viewing state of a virtual space formed by an HMD when a user tilts his head 45 degrees in this embodiment.

[0056] Fig. 6A , Figure 6B , Figure 6C In Figure 5A , Figure 5B , Figure 5C The same structures are denoted by the same symbols, and their description is omitted. Fig. 6A In the figure, it is shown that Figure 5A In other words, the HMD 100 is shown to be rotated 45 degrees clockwise in the XY plane of the real space.

[0057] Figure 6B , Figure 6C Representation and Figure 5B , Figure 5C The display in the display unit 131 is the same. Figure 6B , Figure 6C Zhongyu Figure 5B , Figure 5C The difference is that, unlike the AR objects OBJ1 to 3, the inclination angle of the additional object OBJM is the same as the inclination angle of the user's head, and the additional object OBJM is displayed facing the user.

[0058] In this embodiment, the prescribed value of the tilt angle of the user's head is set to 40 degrees. If it is less than 40 degrees, Figure 5CAs shown in FIG. 1 , the AR objects OBJ1 to OBJ3 and the additional object OBJM are displayed on the display unit 131 in a state in which they are tilted according to the tilt angle of the head. On the other hand, if the tilt angle is greater than 40 degrees, as shown in FIG. Figure 6C As shown, the additional object OBJM is different from the AR objects OBJ1 to 3 in that its inclination angle is the same as the inclination angle of the user's head and is displayed so as to be directly opposite to the user U1 who is tilting his head.

[0059] This is because when the difference between the inclination angle of the user's head and the inclination angle of the attached object is large, the attached object becomes difficult to read, so in this embodiment, the specified value of the inclination angle is set to 40 degrees, and when the inclination angle of the head is more than 40 degrees, the inclination angle of the attached object is made the same as the inclination angle of the user's head. As a result, it has an effect of being easy to read when the attached object is text, etc. In addition, the specified value of the inclination angle can be a maximum of about 30 to 40 degrees. In addition, the specified value of the inclination angle can also be 0 degrees. That is, the inclination angle of the attached object OBJM can always be the same as the inclination angle of the user's head, and displayed in a manner facing the user U1 who is tilting his head. In addition, the specified value of the inclination angle can also be changed according to the user's setting.

[0060] Figure 7 It is an explanatory diagram showing the relationship between the tilt angle of the user's head and the image displayed on the display unit in this embodiment. Figure 7 , a display mode of an AR object and an additional object displayed on the display unit when the tilt angle of the user's head is changed in units of 15 degrees is shown, and a case where the prescribed value of the tilt angle is set to 40 degrees is shown.

[0061] like Figure 7 As shown, at the tilt angle of 15 degrees and 30 degrees less than the specified value of 40 degrees, the AR objects OBJ1 to 3 and the additional object OBJM are displayed in the same state of tilting to the left direction, which is opposite to the right direction when the user tilts his head. On the other hand, at the tilt angle of 45 degrees to 90 degrees, which is greater than the specified value of 40 degrees, the AR objects OBJ1 to 3 are displayed in a state of tilting to the left direction, which is opposite to the right direction when the user tilts his head, but the additional object OBJM is displayed at the same tilt angle as the tilt angle of the head, that is, in a manner facing the user who tilts his head.

[0062] Figure 8 4 is a flowchart of the object display control process in this embodiment. Figure 8 In step S101, the display control unit 1114 first checks whether the state of the HMD 100 is the AR object display on mode. If it is the AR object display on mode, the process proceeds to step S102. If it is not the AR object display on mode, the process ends.

[0063] In step S102, the posture detection unit 1112 detects the position and pointing direction of the HMD 100 in the real space, and the inclination angle of the HMD 100. Here, the inclination angle of the HMD 100 is an angle formed by the longitudinal direction of the display unit 131 with respect to the vertical direction on a plane parallel to the display unit 131, in other words, an angle formed by the vertical axis of the head with respect to the vertical direction on a plane parallel to the display unit 131, and is 0° when the head is upright.

[0064] Then, in step S103 , the object data acquisition unit 1111 acquires information on an AR object to be arranged in the virtual space S1 from a server device on the network. Next, in step S104 , the object data acquisition unit 1111 generates an AR object to be displayed on the display unit 131 .

[0065] Then, in step S105, the display control unit 1114 displays the AR object generated in the process of S104 on the display unit 131. In addition, the AR object displayed here is displayed with the gravity direction g of the virtual space as a vertical reference, regardless of the inclination of the HMD 100 detected in the process of S102. In other words, it is displayed in a manner that maintains the posture based on the gravity direction g of the virtual space S1. That is, the AR object is displayed on the basis of performing a rotation process so that it becomes a state in which it is tilted in the opposite direction by the same amount as the inclination of the HMD 100 detected in the process of S102.

[0066] Next, in step S106, the additional object generation unit 1113 checks whether the state of the HMD 100 is the additional object display on mode. If it is the additional object display on mode, the process proceeds to step S107. If it is not the additional object display on mode, the process returns to step S101.

[0067] In step S107 , the additional object generation unit 1113 generates an additional object associated with the AR object generated in the process of S104 .

[0068] Then, in step S108, it is confirmed whether the inclination angle of HMD 100 with respect to the vertical direction detected in the process of S102 is less than a predetermined value. If the inclination angle with respect to the vertical direction is less than the predetermined value, the process proceeds to step S109. If the inclination angle with respect to the vertical direction is not less than the predetermined value, the process proceeds to step S110.

[0069] In step S109, the display control unit 1114 displays the additional object generated in S107 with the gravity direction g in the virtual space as a vertical reference, regardless of the inclination of the HMD 100 detected in the process of S102. That is, like the AR object displayed in the process of S105, the additional object is displayed in a state in which it is tilted in the opposite direction by the same amount as the inclination of the HMD 100 detected in the process of S102.

[0070] In step S110 , the display control unit 1114 displays the additional object generated in step S107 using the same vertical reference as the display unit 113 . That is, the additional object is displayed in a non-inclined posture on the display unit 113 .

[0071] Fig. 9A , Fig. 9B It is an explanatory diagram of the additional object display control in this embodiment. Fig. 9A This figure shows a state where the user U1 tilts his head to the right (clockwise) by α degrees. That is, α is the tilt angle of the user U1's head relative to the gravity direction G in the real space, that is, the vertical direction, and shows a state where the HMD 100 is rotated clockwise by α degrees in the XY plane in the real space.

[0072] Fig. 9B is Fig. 9A In the state, a summary table of display control of AR objects and additional objects corresponding to the inclination angle α of the user's head in this embodiment.

[0073] Fig. 9B In the case where the tilt angle α of the user's head, i.e., the tilt angle α of the HMD satisfies 0≤α<a predetermined value, the AR object is displayed with reference to the gravity direction g of the virtual space S1 regardless of the tilt angle of the HMD 100. In addition, the additional object is also displayed with reference to the gravity direction g of the virtual space S1 regardless of the tilt angle of the HMD 100.

[0074] On the other hand, when the inclination angle α of the HMD satisfies the specified value ≤α, the AR object is similarly displayed with reference to the gravity direction g of the virtual space S1 regardless of the inclination angle of HMD 100, but the additional object is displayed in a manner such that the relative tilt difference with HMD 100 is 0 according to the inclination angle of HMD 100.

[0075] As described above, according to the present embodiment, in order to express the state of immersion in the virtual world, a process is adopted in which the AR object as the main image is tilted according to the tilt angle of the user's head, that is, the tilt angle of the display unit, but the additional objects such as the menu display are not tilted. This makes it possible to provide an easy-to-use information display device and a display method thereof that can make the menu display and the like easy to view even when the display unit is tilted.

[0076] Example 2 In this modification, a modification of the display control of the additional object when the user tilts his head will be described.

[0077] Fig.10 It is an explanatory diagram of display control of an AR object and an additional object according to the inclination angle α of the user's head in the present embodiment.

[0078] Fig.10 In, with Fig. 9A Similarly, assuming that the user U1 tilts his head to the right (clockwise) by α degrees, when multiple specified values ​​satisfying the relationship of specified value 1 < specified value 2 < specified value 3 < ... < specified value n are set, when the inclination angle α of the HMD satisfies 0≤α<prescribed value 1, the AR object is displayed with reference to the gravity direction g of the virtual space S1 regardless of the inclination angle of the HMD 100. In addition, the additional object is displayed with an inclination of (prescribed value 1÷2) relative to the vertical direction. For example, when the specified value 1 is 30 degrees, when α satisfies 0≤α<30, it is displayed with an inclination of (30÷2)=15 degrees.

[0079] In addition, when the inclination angle α of the HMD satisfies the prescribed value 1≤α<prescribed value 2, the AR object is displayed with reference to the gravity direction g of the virtual space S1 regardless of the inclination angle of the HMD 100, but the additional object is displayed with an inclination angle of {prescribed value 1+[(prescribed value 2-prescribed value 1)÷2]} relative to the vertical direction. For example, when the prescribed value 1 is 30 degrees and the prescribed value 2 is 60 degrees, when α satisfies 30≤α<60, it is displayed with an inclination angle of {30+[(60-30)÷2]}=45 degrees.

[0080] Similarly, when the inclination angle α of the HMD satisfies the prescribed value 2≤α<prescribed value 3, the AR object is displayed with reference to the gravity direction g of the virtual space S1 regardless of the inclination angle of the HMD 100, but the additional object is displayed with an inclination angle of {prescribed value 2+[(prescribed value 3-prescribed value 2)÷2]} relative to the vertical direction. For example, when the prescribed value 2 is 60 degrees and the prescribed value 3 is 90 degrees, when α satisfies 60≤α<90, it is displayed with an inclination angle of {60+[(90-60)÷2]}=75 ​​degrees.

[0081] Finally, when the inclination angle α of the HMD satisfies the specified value n≤α≤180, the AR object is displayed based on the gravity direction g of the virtual space S1 regardless of the inclination angle of HMD100, but the additional object is displayed with an inclination of {specified value n+[(180-specified value n)÷2]} relative to the vertical direction.

[0082] As described above, in the present embodiment, the additional object is displayed with a stepwise inclination according to the inclination angle α of the user's head, that is, the inclination angle α of the display unit of the HMD.

[0083] Therefore, according to this embodiment, the difference between the tilt angle of the user's head and the tilt angle of the additional object can be minimized without increasing the tilt angle difference between the AR object and the additional object. In addition, the display mode of the additional object can be prevented from changing slightly in conjunction with a slight change in the tilt angle of the user's head, which has the effect of being able to express a state of immersion in the virtual world and making it easy to read when the additional object is text or the like.

[0084] Example 3 In the present embodiment, another modification example of the display control of the additional object when the user tilts his head will be described.

[0085] Fig.11A , Fig. 11B , Fig. 11C This is a diagram for explaining an image displayed on the display unit of the HMD in a state where the user wearing the HMD in this embodiment tilts his head to the right (clockwise direction).

[0086] Fig.11A In the example, the image 131V5 is displayed on the display unit 131 of the HMD. In the image 131V5, Figure 4A Similarly, AR objects OBJ1 to OBJ3 and a pop-up box for explaining the AR object (for example, a diagram, a table, or a comment explaining the AR object OBJ2) are the additional object OBJP1. Fig.11A ] shows a case where the inclination angle of the user's head is smaller than a predetermined value, and the additional object OBJP1 also maintains the configuration based on the gravity direction of the virtual space S1.

[0087] Fig. 11B Indicates from Fig.11A , the user further tilts his head to the right and the tilt angle of the user's head reaches a predetermined value or more. Fig. 11B In the embodiment, because the tilt angle of the user's head exceeds the specified value, the additional object OBJP1 is displayed using the same vertical reference as the HMD through the display control described in the first embodiment. However, the AR objects OBJ1 to OBJ3 are arranged based on the gravity direction of the virtual space S1, so Fig. 11B As shown, there is a case where the display of the additional object OBJP1 overlaps with the display of the AR object.

[0088] Therefore, in this embodiment, if Fig. 11C As shown, in order to eliminate Fig. 11BIn the state where the display of the additional object OBJP1 overlaps with the display of each AR object, the additional object OBJP1 is moved to a position where it does not overlap with the display of each AR object. That is, for the additional object, not only the rotation processing in Example 1 but also the movement processing can be performed simultaneously. As a result, even if the user tilts his head, it is easier to view the menu display, etc.

[0089] In addition, the moving target of the additional object OBJP1 during the above processing can be set to a position away from a predetermined distance in a predetermined direction. Alternatively, the display control unit 1114 can grasp the area occupied by each AR object in the display unit 131 determined by the shape of each AR object, so the additional object OBJP1 can be moved to a position avoiding the area occupied by each AR object in the display unit 131.

[0090] In addition, when performing the above-mentioned processing of moving the additional object OBJP1, in order to prevent the position of the additional object moving target from changing slightly in conjunction with a slight change in the tilt angle of the user's head, control can be performed so that the relative position relationship between the position of the additional object moving target and the AR object is fixed when the tilt angle of the user's head changes slightly. In this way, even when the user's head moves slightly, it is possible to prevent the visibility of the additional object from being impaired.

[0091] As described above, according to the present embodiment, when the user tilts his head, that is, when the display unit is tilted, the AR object is tilted corresponding to the tilt angle of the user's head, that is, the tilt angle of the display unit, and additional objects such as menu displays are processed so as not to be tilted and moved to a position that does not overlap with the AR object for display, thereby providing an information display device and a display method thereof that can make menu displays, etc. easier to view and easier to use.

[0092] The above embodiments are described, but the present invention is not limited to the above embodiments, and includes various modified examples. For example, in the embodiments, HMD is used as an example of an information display device, but the present invention is not limited to HMD, and can also be applied to information display devices such as smart phones whose display screen angle relative to the vertical direction can be changed.

[0093] In addition, in addition to the process of tilting the object according to the tilt angle of the user's head or the tilt angle of the display unit, the user can also use the operation input unit to choose to fix the tilt angle of the object. For example, when the AR object and the additional object are displayed tilted according to the tilt angle of the display unit, if the user selects the display to be fixed, the tilt angle at the time of selection is maintained and displayed continuously regardless of the subsequent tilt angle of the display unit. Depending on the user, there are cases where the object does not like to be rotated according to the tilt angle of the display unit. Therefore, by enabling the user to select the display to be fixed, the user's ease of use can be improved.

[0094] In addition, the functions of the present invention described in the embodiments are described as being implemented by software according to the action program that implements each function by interpreting and executing the CPU, etc., but part or all of it can also be implemented by hardware, for example, by designing an integrated circuit, a general-purpose processor, or a dedicated processor. The processor includes transistors and other circuits, which can be regarded as circuitry or processing circuitry. In addition, hardware and software can also be used at the same time. In addition, part or all of each function can also be implemented by a server. In addition, the server can collaborate with other components via communication and perform functions, for example, it can be a local server, a cloud server, an edge server, a network service, etc., and the method is arbitrary. Information such as programs, tables, files, etc. that implement each function can be stored in a recording device such as a memory, a hard disk, an SSD (Solid State Drive) or a recording medium such as an IC card, an SD card, and a DVD, or can be stored in a device on a communication network.

[0095] In addition, the programs described in each processing example may be independent programs, or a plurality of programs may constitute one application program. In addition, the order of each processing may be switched and executed.

[0096] In addition, the above-mentioned embodiments are described in detail for the purpose of explaining the present invention in an easy-to-understand manner, and are not limited to all structures that must be described. In addition, a part of the structure of a certain embodiment can be replaced with the structure of another embodiment, and the structure of another embodiment can be added to the structure of a certain embodiment. In addition, for a part of the structure of each embodiment, other structures can be added, deleted, or replaced.

[0097] Description of Reference Numerals 100: HMD (head mounted display device), 101: main control unit, 103: RAM, 110: storage unit, 120: operation input unit, 130: image processing unit, 131: display unit, 131V: viewing range, 140: sound processing unit, 150: position information acquisition unit, 160: sensor unit, 170: communication unit, 180: expansion interface unit, 1001: basic action program, 1002: application, 1009: various data storage areas, 1101: basic action function unit, 1111: object data acquisition unit, 1112: posture detection unit, 1113: additional object generation unit, 1114: display control unit, 1199: temporary storage area, U1: user, S1: virtual space, OBJ1~3: AR objects, OBJM, OBJP1: additional objects.

Claims

1. An information display device having a display unit, characterized in that: include: an object data acquisition unit that acquires information about an AR object and generates an AR object to be displayed on the display unit based on the acquired information; an additional object generating unit configured to generate an additional object, wherein the additional object is additional information associated with the AR object generated by the object data acquiring unit; a posture detection unit, which detects the inclination angle of the information display device relative to the gravity direction of the real space; and a display control unit that controls display of the AR object and the additional object on the display unit based on a gravity direction set in a virtual space in which the AR object is arranged and an inclination angle of the information display device, wherein: In the display control unit, the AR object is displayed in a tilted state on the display unit corresponding to the inclination angle of the information display device, and the additional object is displayed in a tilted state on the display unit corresponding to the inclination angle of the information display device when the inclination angle of the information display device is less than a specified value, and is displayed in a non-tilted manner on the display unit by making the vertical reference the same as that of the display unit regardless of the inclination angle of the information display device when the inclination angle of the information display device is greater than a specified value.

2. The information display device according to claim 1, characterized in that: The display control unit displays the additional object on the display unit with a stepwise inclination according to the inclination of the information display device.

3. The information display device according to claim 2, characterized in that: In the display control unit, when there is a relationship of prescribed value 1 < prescribed value 2, for the additional object, when the inclination angle α of the information display device satisfies 0≤α<prescribed value 1, the additional object is displayed on the display unit in a manner having an inclination angle of (prescribed value 1÷2), and when the prescribed value 1≤α<prescribed value 2 is satisfied, the additional object is displayed on the display unit in a manner having an inclination angle of {prescribed value 1+[(prescribed value 2-prescribed value 1)÷2]}.

4. The information display device according to claim 1, characterized in that: The display control unit moves the additional object to a position where it does not overlap with the AR object and displays the additional object.

5. The information display device according to claim 1, characterized in that: The information display device is a head mounted display device, The inclination angle of the information display device is an inclination angle of the head of a user wearing the head mounted display device, The control of displaying on the display unit in a tilted state according to the tilt angle of the information display device is control of displaying on the display unit in a state tilted in the opposite direction by the same amount as the tilt angle of the head, The control for displaying the additional object on the display unit in a non-tilted manner regardless of the inclination of the information display device is to display the additional object in a manner that has the same inclination as the head and is directly opposite to the user.

6. A display method of an information display device having a display unit, characterized in that: Acquire information about the AR object, and generate the AR object based on the acquired information; generating an additional object, wherein the additional object is additional information associated with the generated AR object; Detecting the inclination angle of the information display device relative to the gravity direction of the real space; The AR object is displayed on the display unit in a tilted state according to the tilt angle of the information display device; For the additional object, when the inclination angle of the information display device is less than a specified value, it is displayed on the display unit in a tilted state corresponding to the inclination angle of the information display device, and when the inclination angle of the information display device is greater than a specified value, the vertical reference is made the same as the display unit regardless of the inclination angle of the information display device so that it is displayed on the display unit in a non-tilted manner.

7. The display method according to claim 6, characterized in that: The additional object is displayed on the display unit with a stepwise inclination according to the inclination of the information display device.

8. The display method according to claim 7, characterized in that: When there is a relationship of specified value 1 < specified value 2, for the additional object, when the inclination angle α of the information display device satisfies 0≤α<specified value 1, the additional object is displayed on the display portion with an inclination angle of (specified value 1÷2); when the specified value 1≤α<specified value 2 is satisfied, the additional object is displayed on the display portion with an inclination angle of {specified value 1+[(specified value 2-specified value 1)÷2]}.

9. The display method according to claim 6, wherein: The additional object is moved to a position where it does not overlap with the AR object for display.

10. The display method according to claim 6, characterized in that: The information display device is a head mounted display device, The inclination angle of the information display device is an inclination angle of the head of a user wearing the head mounted display device, The control of displaying on the display unit in a tilted state according to the tilt angle of the information display device is control of displaying on the display unit in a state tilted in the opposite direction by the same amount as the tilt angle of the head, The control for displaying the additional object on the display unit in a non-tilted manner regardless of the inclination of the information display device is to display the additional object in a manner that has the same inclination as the head and is directly opposite to the user.

11. An information display device, characterized in that: A display, a sensor for detecting the inclination of the information display device, and a processor are provided. The processor: generating an AR object and an additional object, wherein the additional object is additional information associated with the AR object, When the inclination of the information display device detected by the sensor is smaller than a predetermined value, control is performed to display the AR object and the additional object on the display in a state tilted according to the inclination of the information display device. When the inclination angle of the information display device detected by the sensor is greater than a specified value, control is performed to display the AR object on the display in a state tilted corresponding to the inclination angle of the information display device, and to display the additional object on the display without tilt regardless of the inclination angle of the information display device.

12. The information display device according to claim 11, characterized in that: The additional object is composed of text.

13. The information display device according to claim 11, characterized in that: The AR object is an object representing a person or an object. The additional object is an object representing a menu including text.

14. The information display device according to claim 11, characterized in that: An interface is provided, which is used in an operation in which a user sets the prescribed value.

Images