Display device, control method thereof, program, and recording medium thereof

Abstract

To provide a display device and a control method thereof, and a program and a recording medium thereof, which can accurately and quickly move a display position of a pointer to a position of an object selected by an operator as an operation object.SOLUTION: A display device 110 is connected to a controller 100 held by an operator in a communicable manner, and displays a pointer in a three-dimensional display region. If a direction indication is not given to the controller 100 by the operator, the display device 110 is controlled to enter a first operating state to move a display position of the pointer in the display region in accordance with tilt and displacement of the controller 100 which are acquired from the controller 100. If the direction indication is given to the controller 100 by the operator, the display device 110 is controlled to enter a second operating state to move the display position of the pointer in the display region in response to the direction indication.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a display device, a control method thereof, a program, and a recording medium thereof, and particularly to a display device that superimposes and displays an operation position image whose display position moves by an operation of an operator on an external control device in a display area, a control method thereof, a program, and a recording medium thereof.

Background Art

[0002] Conventionally, there is known a display device that moves the display position of a pointer in a display area according to the inclination and displacement of an external control device held by an operator. The operator can select and operate, as an operation target, an object whose position coincides with the pointer among the objects displayed in the display area.

[0003] However, the external control device may be provided with an input reception unit such as a button, and the operator may input to the input reception unit. In this case, the posture of the external control device changes, and there is a risk that the position of the pointer deviates from the position of the object to be operated, resulting in an erroneous operation.

[0004] To prevent such an erroneous operation, in Patent Document 1, when the operator presses a determination button on the external control device, the display device shifts to an operation state in which the position of the pointer is not moved even if the external control device inclines or displaces. In addition, in Patent Document 1, the external control device is further provided with a movement change button, and when the operator long-presses the movement change button, the movement amount of the pointer according to the inclination and displacement of the external control device is changed.

[0005] Furthermore, in a conventional display device, when a plurality of selectable object groups are arranged in the depth direction as operation targets, the object in the back is completely hidden from view due to the object displayed in the front. For this reason, it becomes difficult for the operator to select an object in the back as an operation target with the pointer.

[0006] On the other hand, in Patent Document 2, binocular parallax is used to determine the selection position in the depth direction within the virtual space at the point where the operator's lines of sight intersect.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0008] However, in Patent Document 1, since the pointer is moved by the inclination and displacement of the external control device, if the position of the object to be operated is far from the operator, even a slight tremor by the operator will cause a large deviation in the display position of the operation position image.

[0009] Also, in Patent Document 2, the process of determining the selection position in the depth direction within the virtual space using binocular parallax makes it difficult for the operator to consciously select an arbitrary object as the operation target, such as when the distance between objects arranged in the depth direction is very short.

[0010] Therefore, an object of the present invention is to provide a display device and its control method, as well as a program and its recording medium, capable of accurately and quickly moving the display position of the pointer to the position of the object selected by the operator as the operation target.

Means for Solving the Problems

[0011] The display device according to claim 1 of the present invention is a display device that is communicably connected to an external control device that receives an operation by an operator and displays a pointer in a three-dimensional display area, and includes acquisition means for acquiring, as detection information, at least one of the inclination and displacement of the external control device and a two-dimensional direction instruction by the operator with respect to the external control device from the external control device, and control means for controlling so that when the direction instruction is not given to the external control device, it is set as a first operation state in which the display position of the pointer in the display area is moved according to the acquired inclination and displacement, and when the direction instruction is given to the external control device, two coordinate axes used for moving the display position of the pointer are made different according to the inclination of the external control device, and it is set as a second operation state in which the display position of the pointer in the display area is moved based on the two coordinate axes according to the acquired direction instruction. When the control means has not acquired the direction instruction as the detection information by the acquisition means for a certain period of time or longer after shifting to the second operating state, it shifts to the first operating state. It is characterized by this. The display device according to claim 5 of the present invention is a display device that is communicably connected to an external control device that receives an operation by an operator and displays a pointer in a three-dimensional display area, and includes at least one of the inclination / displacement of the external control device and a two-dimensional direction instruction by the operator with respect to the external control device. Acquisition means for acquiring from the external control device as detection information, and when no direction instruction is given to the external control device, a first operation state in which the display position of the pointer in the display area is moved according to the acquired inclination / displacement; When a direction instruction is given to the external control device, two coordinate axes used for moving the display position of the pointer are made different according to the inclination of the external control device, and the display position of the pointer is moved based on the two coordinate axes in the display area according to the acquired direction instruction. Control means for controlling to be in a second operating state, and the control means displays an image of the real space captured in the display area, and when the virtual object hidden by the object in the real space exists in the display area in the second operating state, the silhouette of the virtual object is displayed in the display area.

Effect of the Invention

[0012] According to the present invention, the display position of the pointer can be accurately and quickly moved to the position of the object selected by the operator as the operation target.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0014] (First Embodiment) Hereinafter, with reference to FIG. 1, the internal configurations of the display device 110 according to the first embodiment of the present invention and the controller 100 as an external control device communicably connected thereto will be described.

[0015] The controller 100 is, for example, a remote control type control device held by an operator's hand, a ring type control device worn on a finger, or the like. However, the controller 100 is not limited to these as long as it can be held by an operator's hand and has the internal configuration shown in FIG. 1.

[0016] As shown in FIG. 1, the controller 100 includes a control unit 101, a direction instruction unit 102, an angular velocity detection unit 103, a non-volatile memory 104, a volatile memory 105, and a communication unit 106, and these perform data exchange with each other via an internal bus 107.

[0017] The control unit 101 executes a program stored in the non-volatile memory 104 and controls the processing of the entire controller 100. The control unit 101 is composed of a single or a plurality of processors.

[0018] The direction indication unit 102 receives a direction indication from the operator. The direction indication unit 102 is, for example, an analog stick such as a joystick, and receives a two-dimensional direction indication from the operator as shown in FIG. 4. Note that the direction indication unit 102 is not limited to an analog stick as long as it can receive a two-dimensional direction indication from the operator. For example, as in the second embodiment described later, the direction indication unit 102 may be an OTP (Optical Track Pad) that optically detects the movement of the user's finger on the surface of a pushable operation button to receive a direction indication. The direction indication unit 102 may also be a touch panel.

[0019] The angular velocity detection unit 103 detects the inclination and displacement of the controller 100. The angular velocity detection unit 103 is, for example, a gyro sensor.

[0020] The non-volatile memory 104 stores data such as programs executed by the control unit 101.

[0021] The volatile memory 105 is a memory that the control unit 101 uses as a buffer or a working memory during program execution.

[0022] The communication unit 106 transmits the information acquired by the direction indication unit 102 or the angular velocity detection unit 103 to the communication unit 114 (described later) provided in the display device 110.

[0023] Also, the display device 110 is, for example, an HMD (Head Mounted Display). However, the display device 110 is not limited to these as long as the operator can view its display area and it has the internal configuration shown in FIG. 1.

[0024] As shown in FIG. 1, the display device 110 includes a control unit 111, an imaging unit 112, a display unit 113, a communication unit 114, a non-volatile memory 115, and a volatile memory 116, and these perform data exchange with each other via an internal bus 117.

[0025] The control unit 111 executes the program stored in the non-volatile memory 115 and controls the processing of the entire display device 110. The control unit 111 is composed of a single or multiple processors.

[0026] The imaging unit 112 is composed of an imaging lens, an imaging element such as a CMOS sensor, and an A / D converter, etc. It converts the analog signal input through the imaging lens into digital data to acquire image data, and expands it in the volatile memory 116. Thereby, an image of the real space is captured by the imaging unit 112.

[0027] The display unit 113 displays in real time the image data expanded in the volatile memory 116 by the imaging unit 112. That is, an image of the real space is LV-displayed in the three-dimensional display area of the display unit 113. Also, a plurality of virtual objects generated according to the processing of the control unit 111 and an operation position image indicating the operation position determined by the control unit 111 according to the operation of the controller 100 by the operator are superimposed on the LV display of the real space.

[0028] Here, the operation position image has, for example, the shape of a laser pointer that displays a line connecting the position of the controller 100 to the operation position in the display area. However, the operation position image only needs to be an image that specifies the operation position, and may have, for example, an arrow shape indicating only the operation position or any other arbitrary shape. Hereinafter, the operation position specified by the operation position image is simply referred to as a pointer.

[0029] The communication unit 114 receives the information transmitted from the communication unit 106 provided in the controller 100.

[0030] The non-volatile memory 115 stores data such as programs executed by the control unit 111.

[0031] The volatile memory 116 is a memory that the control unit 111 uses as a buffer or a working memory during program execution.

[0032] Also, the communication between the communication unit 106 and the communication unit 114 is, for example, a wireless connection such as Bluetooth (registered trademark) or LAN, or a wired connection using an interface such as USB (registered trademark) or PCIe (registered trademark).

[0033] As described above, with reference to FIG. 1, the necessary internal configurations of the controller 100 and the display device 110 have been described.

[0034] Hereinafter, with reference to the flowchart shown in FIG. 2, the detection information transmission process in the controller 100 according to the first embodiment of the present invention will be described. This process is executed by the control unit 101 expanding the program in the non-volatile memory 104 into the volatile memory 105.

[0035] First, in step S201, the control unit 101 transmits the angular velocity information acquired by the angular velocity detection unit 103 as detection information in real time via the communication unit 106.

[0036] Next, in step S202, the control unit 101 determines whether there is a direction instruction received from the operator by the direction instruction unit 102. If there is a direction instruction, the process proceeds to step S203. On the other hand, if there is no direction instruction, this process ends.

[0037] In step S203, the control unit 101 transmits the information on the instruction direction received from the operator by the direction instruction unit 102 as detection information in real time via the communication unit 106, and ends this process.

[0038] As described above, the detection information transmission process in the controller 100 according to the first embodiment of the present invention has been described.

[0039] Hereinafter, with reference to the flowchart shown in FIG. 3, the pointer display control process in the display device 110 according to the first embodiment of the present invention will be described. This process is executed by the control unit 111 expanding the program in the non-volatile memory 115 into the volatile memory 116.

[0040] Furthermore, this process is started in a state where a plurality of virtual objects and operation position images are superimposed on an image of the real space that is LV-displayed on the display unit 113. Also, in this process, the controller 100 is a remote control type controller as shown in FIG. 4, and a case where the operation position image has the shape of a laser pointer will be described.

[0041] First, in step S301, the control unit 111 (acquisition means) receives in real time, via the communication unit 114, the detection information transmitted in steps S201 and S203.

[0042] Next, in step S302, the control unit 111 determines whether there is a direction instruction in the detection information. Specifically, the control unit 111 determines that there is a direction instruction in the detection information if there is direction instruction information in the information received by the communication unit 114 in step S301.

[0043] As a result of the determination in step S302, if there is no direction instruction in the detection information, the process proceeds to step S310, and if there is a direction instruction, the process proceeds to step S320.

[0044] In step S310, the control unit 111 (control means) shifts the operating state of the display device 110 to the first operating state. Here, the first operating state is an operating state in which the display position of the pointer in the display area is moved according to only the inclination and displacement detected by the angular velocity detection unit 103 as shown in FIG. 5. Note that the virtual controller 301 shown in FIG. 5 is an image that is superimposed and displayed in the display area of the display unit 113 when the controller 100 exists within the imaging range of the imaging unit 112. The virtual controller 301 may not be displayed in the display area of the display unit 113. Further, the virtual object group 302 is an image showing a plurality of virtual objects that are superimposed and displayed in the display area of the display unit 113. By such control, when the virtual object displayed in the display area of the display unit 113 is at a distant position as shown in FIG. 5, the pointer can be quickly and largely moved by tilting the controller 100, and one of the virtual object groups 302 can be selected. Therefore, for example, one of the virtual object groups 302 can be selected without performing operations on the direction indicator 102 many times (with few operations).

[0045] Next, in step S311, the control unit 111 changes the shape and color of the pointer displayed on the display unit 113 in order to indicate that the first operating state has been transitioned to. Note that in step S321 described later, the shape and color of the pointer displayed on the display unit 113 are changed to a shape and color different from those in step S311 in order to indicate that the second operating state has been transitioned to. Thereby, the operator can grasp which of the first and second operating states the current operating state is. The shape of the pointer in the first operating state is, for example, circular and displayed in gray. At this time, not only the shape and color but also the transparency may be changed. Further, instead of changing the pointer display, the selected object itself in the virtual object group 302 may be highlighted.

[0046] In step S312, the control unit 111 determines whether the inclination or displacement of the controller 100 has been detected by the angular velocity detection unit 103 from the angular velocity information received by the communication unit 114. As a result, if it is determined that the inclination or displacement has been detected, the process proceeds to step S313, and if it is determined that the inclination or displacement has not been detected, this process ends.

[0047] In step S313, the control unit 111 moves the pointer according to the inclination or displacement detected by the angular velocity detection unit 103, and ends this process.

[0048] On the other hand, in step S320, the control unit 111 (control means) shifts the operating state of the display device 110 to a second operating state. Here, as shown in FIG. 6, the second operating state is an operating state in which the display position of the pointer in the display area is moved only in response to the direction instruction received by the direction instruction unit 102 from the operator. That is, even if the angular velocity detection unit 103 detects an inclination or displacement, unlike the first operating state, the display position of the pointer is not moved according to the detected inclination or displacement. By such control, when the virtual object group 302 displayed in the display area of the display unit 113 is close to the pointer as shown in FIG. 6, the pointer can be moved more finely by operating the direction instruction unit 102. As a result, since the pointer does not move significantly to a position different from the virtual object group 302, one of the virtual object groups 302 can be selected quickly and more accurately.

[0049] Next, in step S321, the control unit 111 changes the shape and color of the pointer displayed on the display unit 113 in order to indicate that the transition to the second operating state has been made as described above in step S311. The shape of the pointer in the second operating state is, for example, circular and displayed in red. As described above in step S311, the transparency of the pointer display in the first operating state may be made different.

[0050] In step S322, the control unit 111 determines whether the inclination of the controller 100 has been detected by the angular velocity detection unit 103 from the angular velocity information received by the communication unit 114. As a result, if it is determined that the inclination has been detected, the process proceeds to step S323, and if it is determined that the inclination has not been detected, the process proceeds to step S324.

[0051] In step S323, the control unit 111 changes the coordinate axes of the direction instruction in conjunction with the inclination of the controller 100 detected by the angular velocity detection unit 103. For example, as shown in FIG. 7, consider a case where the operator holds the controller 100 in a horizontal posture (i.e., a plane composed of the x-axis and the z-axis) with respect to the display area and gives a direction instruction to the direction instruction unit 102. At this time, the control unit 111 moves the position of the pointer in the X-axis direction and the Z-axis direction of the display area according to the direction instruction. Further, as shown in FIG. 8, consider a case where the operator holds the controller 100 in a vertical posture (i.e., a plane composed of the x-axis and the y-axis) with respect to the display area and gives a direction instruction. At this time, the control unit 111 moves the position of the pointer in the X-axis direction and the Y-axis direction, which is the depth direction, according to the direction instruction. Note that the virtual object group 303 shown in FIGS. 7 and 8 is an image showing a plurality of virtual objects having the same position in the Y-axis direction, and the virtual object 304 is an image showing a virtual object located deeper than the virtual object group 303 in the Y-axis direction. By this process, an operation of extending the pointer in the Y-axis direction becomes possible. Therefore, even when the controller 100 has the direction instruction unit 102 that accepts only two-dimensional direction instructions from the operator as shown in FIG. 4, the operator can give direction instructions in all directions of the three-dimensional display area.

[0052] In step S324, the control unit 111 moves the pointer only in response to the direction instruction to the direction instruction unit 102.

[0053] In step S325, the control unit 111 determines whether the operation position image penetrates the object surface. Specifically, if the X and Z coordinate values of the pointer match one of the virtual objects currently being displayed in the display area, but the Y coordinate value of the pointer is located behind the virtual object, it is determined that the operation position image penetrates the object surface.

[0054] If it is determined that the operation position image penetrates the object surface (YES in step S325), the process proceeds to step S326. If it is determined that the operation position image does not penetrate the object surface (NO in step S325), this process ends.

[0055] In step S326, the control unit 111 invalidates the contact determination between the virtual object (the first virtual object) for which it was determined in step S325 that the operation position image penetrates the surface and the pointer. As a result, when an object deeper inside contacts the pointer, the operator can perform an operation targeting that object. At this time, the control unit 111 displays the virtual object through which the operation position image penetrates in a transparent manner. Thereby, the operator can determine that the contact determination between the virtual object through which the operation position image penetrates the surface and the pointer has been invalidated.

[0056] For example, among a plurality of virtual objects 305a of the same shape arranged linearly in the Y-axis direction as shown in FIG. 9, the virtual object 305a closest to the controller 100 (the closest to the front) has an operation position image that penetrates its surface. Therefore, the virtual object 305a is transparently displayed in step S326 and the contact determination with the pointer is invalidated. Also, the virtual object 305b (the second virtual object) located behind it and not completely hidden by the virtual object 305a is displayed in the display area with the transparent display of the virtual object 305a. Furthermore, since the virtual object 305b is in contact with the pointer, it is in an operable state.

[0057] In step S327, the control unit 111 moves the pointer to the position of the virtual object (hereinafter referred to as the "next nearest object") that has come to be displayed by the process of step S326, makes the next nearest object in an operable state, and then ends this process. As a result, even if the next nearest object is very far from the pointer, a large number of operations for reaching the pointer to the next nearest object become unnecessary. Conversely, even when the next nearest object is very close to the pointer and it is difficult to move the pointer in the Y-axis direction to the extent that it does not penetrate the surface of the next nearest object manually, the pointer can be moved accurately.

[0058] As described above, according to the first embodiment of the present invention, when the operation of tilting the controller 100 is performed, the pointer can be quickly and largely moved with respect to the virtual object displayed at a distant position, and the virtual object can be selected with few turns. The pointer moves only in accordance with the information of the direction instruction received when an instruction is given to the direction instruction unit 102 of the controller 100, and does not move in accordance with the detected tilt or displacement even when the angular velocity detection unit 103 detects the tilt or displacement. As a result, the display position of the pointer can be accurately and finely moved to the position of the object selected by the operator as the operation target, and can be moved promptly.

[0059] Also, by repeating steps S325 to S327, the operator can intuitively select the next nearest object whose display is completely hidden by the virtual object in front.

[0060] (Second Embodiment) Hereinafter, with reference to the flowchart shown in FIG. 10, the pointer display control process in the display device 110 according to the second embodiment of the present invention will be described. In this embodiment, the description of the parts overlapping with the first embodiment will be omitted, and only the parts different from the first embodiment will be described.

[0061] In the first embodiment, by transitioning the operating state of the display device 110 to the second operating state in step S320, it is possible to accurately move the display position of the pointer without being affected by the operator's hand tremor.

[0062] Here, in the first embodiment, the operating state of the display device 110 was transitioned from the second operating state to the first operating state only based on the determination condition in step S302, that is, whether the direction indicating unit 102 has received a direction instruction from the operator.

[0063] However, if the direction indicating unit 102 is an OTP with a limited operable amount that can be input in only one direction instead of an analog stick, when the operator tries to secure the operable amount by momentarily lifting a finger from the OTP, the operating state of the display device 110 will transition to the first operating state due to that movement. In this case, if the inclination and displacement of the controller 100 change due to the operator's hand tremor, the pointer will move in an unintended direction.

[0064] Therefore, in this embodiment, as shown in FIG. 10, a determination process of step S303 is added between step S302 and step S310. This prevents the operating state of the display device 110 from immediately transitioning to the first operating state even when the direction indicating unit 102 has not received a direction instruction from the operator. That is, after transitioning to the second operating state, when the control unit 111 has not received a direction instruction as detection information for a certain period or more (YES in step S302 and YES in step S303), it proceeds to step S310 and transitions the display device 110 to the first operating state.

[0065] Note that the processing after step S310 and the processing after step S320 are the same as those in the flowchart of FIG. 3 in the first embodiment.

[0066] According to the above-described embodiment, when the operator interrupts the direction instruction to the direction instruction unit 102 for some reason for only a short period of time, the control unit 111 does not immediately transition the operation state of the display device 110 to the first operation state. Thereby, it is possible to prevent the pointer accurately position-adjusted in the second operation state from moving unintentionally.

[0067] (Third Embodiment) Hereinafter, with reference to the flowchart shown in FIG. 11, the display control process of the pointer in the display device 110 according to the third embodiment of the present invention will be described. In the present embodiment, the description of the overlapping parts with the first embodiment will be omitted, and only the parts different from the first embodiment will be described.

[0068] According to the first embodiment, by transparently displaying the virtual object whose operation position image penetrates the surface in step S326, the next nearest object whose display was completely hidden by the front virtual object is displayed.

[0069] However, unlike virtual objects, real objects existing in the real space cannot be transparently displayed. For this reason, when the display of the next nearest object is completely hidden by the front real object, the next nearest object cannot be displayed by the method shown in the first embodiment. Therefore, even if the pointer moves to the position of the next nearest object in step S327, the operator cannot see the next nearest object and it becomes difficult to perform the operation.

[0070] Therefore, in the present embodiment, the display device 110 further includes a three-dimensional distance sensor that acquires distance information of the imaging region of the imaging unit 112. The three-dimensional distance sensor is, for example, a TOF (Time of Flight) sensor. The control unit 111 creates distance map information based on the distance information acquired by the distance measurement sensor.

[0071] In addition, the object to be determined in step S325 was only a virtual object in the first embodiment, but in this embodiment, it also includes real objects. Specifically, when the X and Z coordinate values of the pointer match one of the real objects whose LV display is on, but the Y coordinate value of the pointer is located behind the real object, the control unit 111 determines that the operation position image penetrates the object surface. Note that this determination is made based on the distance map information described above.

[0072] Furthermore, as shown in FIG. 11, a determination process of step S328 is added between step S326 and step S327. As a result, when the object through which the operation position image penetrates the surface is a virtual object (NO in step S328), the process proceeds to step S327, and when it is a real object (YES in step S328), the process proceeds to step S329.

[0073] In step S329, as shown in FIG. 12, the control unit 111 displays the silhouette of the next nearest object whose display is completely hidden by the real object in the front, and then proceeds to step S327.

[0074] That is, after the operator confirms that the pointer has automatically moved to the position of the next nearest object in step S327, the operator can perform an operation on the next nearest object while checking its silhouette.

[0075] For example, as shown in FIG. 12, when the operation position image penetrates the surface of the real object 306 (YES in step S325), the contact determination between the real object 306 and the pointer is invalidated (step S326).

[0076] Also, the object to be determined in step S325 here is the real object 306 (YES in step S328). Therefore, the process proceeds to step S329, and the virtual object 307 (the third virtual object) hidden behind the real object 306 (invisible to the user) on the side opposite to the user with respect to the real object 306, which is the next nearest object, is displayed in silhouette.

[0077] According to this embodiment, when it is desired to check whether a virtual object hidden behind a predetermined real object exists in the display area, the operator may give a direction instruction to penetrate the surface of the predetermined real object with an operation position image by the direction instruction unit 102. If a virtual object hidden behind the predetermined real object exists, the silhouette of the virtual object is displayed in the display area, making it easily confirmable.

[0078] Furthermore, since the pointer automatically moves to the position of the virtual object whose silhouette is displayed in step S327, the operator can quickly perform an operation on the virtual object.

[0079] Also, since the virtual object hidden behind the real object is silhouette-displayed only during the second operating state, it is also possible to obtain a display of the original object arrangement in which the object behind the real object is hidden by transitioning to the first operating state.

[0080] As described above, the preferred embodiments of the present invention have been described. However, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

[0081] (Other Embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a recording medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be executed by a circuit (for example, ASIC) that realizes one or more functions.

Description of Reference Numerals

[0082] 100 Controller 101 Control Unit 102 Direction Instruction Unit 103 Angular Velocity Detection Unit 104 Non-Volatile Memory 105 Volatile memory 106 Communication unit 107 Internal bus 110 Display device 111 Control unit 112 Imaging unit 113 Display section 114 Communication unit 115 Non-volatile memory 116 Volatile memory 117 Internal bus

Claims

1. A display device that is communicably connected to an external control device that receives an operation by an operator and displays a pointer in a three-dimensional display area, an acquisition means for acquiring, as detection information, at least one of the inclination / displacement of the external control device and a two-dimensional direction instruction by the operator with respect to the external control device from the external control device; When the direction instruction is not given to the external control device, it is a first operating state in which the display position of the pointer in the display area is moved according to the acquired inclination / displacement, and when the direction instruction is given to the external control device, two coordinate axes used for moving the display position of the pointer are made different according to the inclination of the external control device, and control means for controlling so as to be a second operating state in which the display position of the pointer is moved based on the two coordinate axes in the display area according to the acquired direction instruction. The display device according to claim 1, wherein the control means shifts to the first operating state when the direction instruction is not acquired as the detection information by the acquisition means for a certain period of time or more after shifting to the second operating state.

2. When in the second operating state, the control means makes the first virtual object be transparently displayed according to the positions of the pointer and the first virtual object displayed in the display area, and invalidates the contact determination between the pointer and the first virtual object. The display device according to claim 1.

3. Before making the first virtual object be transparently displayed, when a second virtual object that was hidden by the first virtual object exists in the display area, the position of the pointer is moved to the position of the second virtual object when the contact determination is invalidated. The display device according to claim 2.

4. The control means is characterized in that at least one of the shape and color of the pointer is made different according to whether the operating state is the first or the second operating state, for the display device according to any one of claims 1 to 3.

5. A display device that is communicably connected to an external control device that receives an operation by an operator and displays a pointer in a three-dimensional display area, an acquisition means for acquiring, as detection information, at least one of the inclination / displacement of the external control device and a two-dimensional direction instruction by the operator with respect to the external control device, from the external control device; When the direction instruction is not given to the external control device, it is set as a first operating state in which the display position of the pointer in the display area is moved according to the acquired inclination / displacement, and when the direction instruction is given to the external control device, two coordinate axes used for moving the display position of the pointer are made different according to the inclination of the external control device, and control means for controlling so as to be a second operating state in which the display position of the pointer is moved based on the two coordinate axes in the display area according to the acquired direction instruction; the control means displays an image of the real space captured in the display area, A display device, characterized in that, in the second operating state and when a virtual object hidden by an object in the real space exists in the display area, a silhouette of the virtual object is displayed in the display area.

6. The control means invalidates the contact determination between the pointer and the object in the real space according to the positions of the pointer and the object in the real space, and displays the silhouette in the display area, for the display device according to claim 5.

7. The control means moves the position of the pointer to the position of the virtual object when the silhouette is displayed, for the display device according to claim 5 or 6.

8. The external control device is at least one of a controller held by the operator by hand and a ring-shaped controller that can be worn on the operator's finger, and is characterized by the display device according to any one of claims 1 to 7.

9. The external control device has a direction indicating unit composed of at least one of an analog stick, an OTP, and a touch panel. The display device according to any one of claims 1 to 8, wherein the direction indicating unit receives a two-dimensional direction indication by the operator.

10. The display device according to any one of claims 1 to 9, wherein the display device is a head-mounted display device.

11. The display device according to any one of claims 5 to 7, wherein the control means displays the silhouette on an object in the real space.

12. The display device according to any one of claims 5 to 7, wherein the control means displays the silhouette when the pointer is located behind an object in the real space.

13. The display device according to any one of claims 1 to 12, wherein the control means switches to the second operating state when there is a direction indication in the first operating state, and changes at least one of the shape and color of the pointer.

14. The display device according to any one of claims 1 to 13, wherein after the control means switches from the first operating state to the second operating state and changes at least one of the shape and color of the pointer, two coordinate axes used for moving the display position of the pointer are made different according to the inclination of the external control device.

15. A control method for a display device that is communicably connected to an external control device that receives an operation by an operator and displays a pointer in a three-dimensional display area. An acquisition step of acquiring, as detection information, at least one of the inclination / displacement of the external control device and the two-dimensional direction instruction by the operator to the external control device from the external control device; When the direction instruction is not given to the external control device, a first operating state is set in which the display position of the pointer in the display area is moved according to the acquired inclination / displacement. When the direction instruction is given to the external control device, two coordinate axes used for moving the display position of the pointer are made different according to the inclination of the external control device, and the display position of the pointer in the display area is moved based on the two coordinate axes according to the acquired direction instruction. A control step of controlling to set a second operating state; In the control step, after shifting to the second operating state, when the direction instruction is not acquired as the detection information in the acquisition step for a certain period of time or more, the control step shifts to the first operating state. A method for controlling a display device, characterized in that

16. A method for controlling a display device that is communicably connected to an external control device that receives an operation by an operator and displays a pointer in a three-dimensional display area, An acquisition step of acquiring, as detection information, at least one of the inclination / displacement of the external control device and the two-dimensional direction instruction by the operator to the external control device from the external control device; When the direction instruction is not given to the external control device, a first operating state is set in which the display position of the pointer in the display area is moved according to the acquired inclination / displacement. When the direction instruction is given to the external control device, two coordinate axes used for moving the display position of the pointer are made different according to the inclination of the external control device, and the display position of the pointer in the display area is moved based on the two coordinate axes according to the acquired direction instruction. A control step of controlling to set a second operating state; In the control step, an image of the real space captured in the display area is displayed, A control method for a display device, characterized in that when a virtual object hidden by an object in the real space in the second operating state exists in the display area, a silhouette of the virtual object is displayed in the display area.

17. A computer-executable program that causes a computer to function as each means of the display device according to any one of claims 1 to 14.

18. A computer-readable recording medium storing a program for causing a computer to function as each means of the display device according to any one of claims 1 to 14.

Images