Display control device and display control method

Abstract

A display control apparatus according to one embodiment of the present disclosure includes: an acquisition unit that acquires position / posture information in a three-dimensional space from a first controller; and a display control unit that displays, in the three-dimensional space, a virtual coordinate plane on which two-dimensional coordinates can be specified, on the basis of the position / posture information in the three-dimensional space. Further, the display control apparatus may include a first detection unit that detects a movement of the virtual coordinate plane in a three-dimensional space based on an input through the first controller.

Description

Technical Field

[0001] The present disclosure relates to a display control device and a display control method for performing operation control on a virtual object. Background Art

[0002] With the development of video display technology, it is possible to construct a virtual space, superimpose content in the virtual space on the real space, and view virtual objects stereoscopically like real objects by using a device for virtual reality (VR) or augmented reality (AR).

[0003] Regarding this technology, in order to allow users to create and edit 3D models corresponding to virtual objects while viewing the 3D models stereoscopically, many user interfaces (UIs) have been proposed that enable operations of holding a controller and moving the model in the air and placing or moving points or lines in the air (6 degrees of freedom (6DoF) operations). As an example, a technology for improving user operability by designing a mechanism for assigning a physical controller to a virtual controller has been proposed (for example, Patent Document 1).

[0004] Reference List

[0005] Patent Literature

[0006] Patent Document 1: JP 6859422 B Summary of the invention

[0007] Technical issues

[0008] However, there is still room for improvement in operations for selecting or editing virtual objects in three-dimensional space.

[0009] For example, in the case where a user raises his hand in the air to operate a controller, the physical load is higher than that of operating with a controller (mouse, etc.) on a conventional desktop. Therefore, there is a problem of reduced input accuracy due to hand shaking, etc. For this reason, it is currently difficult to create or edit 3D models using a controller that is operated in the air, which is mainly used in VR or AR, in applications that require high input accuracy (CAD, etc.).

[0010] Therefore, the present disclosure proposes a display control device and a display control method capable of improving input accuracy.

[0011] Solution to the problem

[0012] In order to solve the above-mentioned problems, a display control device according to aspects of the present disclosure includes: an acquisition unit, which acquires position and posture information in a three-dimensional space from a first controller; and a display control unit, which displays a virtual coordinate plane in the three-dimensional space based on the position and posture information in the three-dimensional space, on which two-dimensional coordinates can be specified. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a diagram showing a processing overview of a display control process according to an embodiment.

[0014] Figure 2 is a diagram (1) showing an example of a display control process according to an embodiment.

[0015] Figure 3 is a diagram (2) showing an example of a display control process according to an embodiment.

[0016] Figure 4 is a diagram showing a configuration example of a display control device according to an embodiment.

[0017] Figure 5 is a sequence diagram showing the flow of display control processing according to the embodiment.

[0018] Fig. 6A is a diagram for describing the operation of the device according to the first modification.

[0019] Figure 6B is a flowchart showing the flow of the display control process according to the first modification.

[0020] Figure 7 is a diagram (1) for describing the display control process according to the second modification.

[0021] Figure 8 is a diagram (2) for describing the display control process according to the second modification.

[0022] Fig. 9 is a diagram (3) for describing the display control process according to the second modification.

[0023] Fig.10 is a diagram (1) for describing the display control process according to the third modification.

[0024] Fig.11 is a diagram (2) for describing the display control process according to the third modification.

[0025] Fig.12 is a diagram (3) for describing the display control process according to the third modification.

[0026] Fig.13is a diagram (4) for describing the display control process according to the third modification.

[0027] Fig.14 is a diagram (5) for describing the display control process according to the third modification.

[0028] Fig.15 is a diagram (6) for describing the display control process according to the third modification.

[0029] Fig.16 is a diagram (7) for describing the display control process according to the third modification.

[0030] Fig.17 is a diagram (8) for describing the display control process according to the third modification.

[0031] Fig.18 is a diagram for describing a display control process according to a fourth modification.

[0032] Fig.19 is a hardware configuration diagram showing an example of a computer that realizes the function of the display control device. DETAILED DESCRIPTION

[0033] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. In the following embodiments, the same parts are denoted by the same reference numerals, and repeated description will be omitted.

[0034] The present disclosure will be described in the order of the following items.

[0035] 1. Implementation Method

[0036] 1-1. Overview of Display Control Processing According to Embodiment

[0037] 1-2. Configuration of Display Control Device According to Embodiment

[0038] 1-3. Processing Procedure According to Embodiment

[0039] 1-4. Modification

[0040] 1-4-1. First modification: Operation of only one device

[0041] 1-4-2. Second modification: Adjustment of operating feel

[0042] 1-4-3. Third modification: Processing of superposition in real space

[0043] 1-4-4. Fourth modification: deformation of the surface based on the pointer position

[0044] 2. Other Implementation Methods

[0045] 3. Effects of the display control device according to the present disclosure

[0046] 4. Hardware Configuration

[0047] (1. Implementation Method)

[0048] (1-1. Overview of Display Control Processing According to Embodiment)

[0049] Will refer to Figure 1 An example of a display control process according to the embodiment is described. Figure 1 is a diagram showing an overview of a display control process according to an embodiment. Figure 1 The components of the display control system 1 that executes the display control process according to the embodiment are shown.

[0050] like Figure 1 As shown, the display control system 1 includes a display control device 100 , a stereoscopic device 20 and a flat device 30 .

[0051] The display control device 100 is an information processing terminal for implementing VR or AR technology. In the present embodiment, the display control device 100 is a wearable display used by being worn on the head of the user 10. Specifically, the display control device 100 is, for example, a head-mounted display (HMD), AR glasses, etc.

[0052] The display control device 100 includes a sealed or transmissive display unit (display). For example, the display control device 100 displays a virtual space expressed by computer graphics (CG) or the like. Alternatively, the display control device 100 superimposes a virtual object expressed by CG or the like on a real space and displays the virtual object on the display unit. Figure 1 In the example of , the display control device 100 displays the object 70 as an example of a virtual object. That is, in Figure 1 In the example of , the object 70 and the space visually recognized by the user 10 are assumed to be a virtual space composed of CG or the like. Note that, as will be described later, the space visually recognized by the user 10 may be a so-called AR space in which virtual objects are superimposed on a real space. In addition, in addition to the display unit, the display control device 100 may also have a configuration for outputting a predetermined output signal. For example, the display control device 100 may include a speaker for outputting sound, etc.

[0053] The stereoscopic device 20 is an example of an input device according to an embodiment. In an embodiment, the stereoscopic device 20 is operated by the user 10 and is used to input various types of information to the display control device 100. The stereoscopic device 20 is a VR controller capable of inputting so-called 6DoF information. Specifically, the stereoscopic device 20 includes a sensor such as an inertial sensor, an acceleration sensor, or a gravity sensor, and detects position and posture information of the stereoscopic device. Then, the stereoscopic device 20 transmits the detected position and posture information of the stereoscopic device to the display control device 100.

[0054] The stereoscopic device 20 may be Figure 1 The controller shown is held by the user 10 with one hand, or may be a pen-type controller. Note that the stereoscopic device 20 is not limited to this example, and may be any device as long as the device can obtain position information in the real space. For example, the stereoscopic device 20 may be an air mouse, a digital camera, a smart phone, etc. In addition, in the case where the display control device 100 can capture the position posture information of the stereoscopic device 20, the stereoscopic device 20 may not include a sensor. For example, the stereoscopic device 20 may be a predetermined object, including a mark, a human face, a finger, etc. that can be recognized by the display control device 100 or a predetermined external device (a video camera installed in the real space, etc.).

[0055] The plane device 30 is an example of an input device according to the present embodiment. In the embodiment, the plane device 30 is operated by the user 10 and is used to input various types of information to the display control device 100. The plane device 30 is, for example, a mouse controller or a trackball mouse capable of receiving two-dimensional coordinate information. Specifically, the plane device 30 includes a sensor such as an optical sensor, and detects coordinate information (two-dimensional information) on a plane based on the operation of the plane device. Then, the plane device 30 transmits the detected coordinate information to the display control device 100.

[0056] Here, the processing when the user 10 specifies an arbitrary position of an object in the VR space or the AR space will be described. In the related art, a controller such as the stereoscopic device 20 has been used to select a position in the VR space or the AR space. However, in the case of operating the controller in the air, the physical load is higher than that of operating with the controller on a conventional table, so that there is a problem of reduced input accuracy due to hand shaking, etc.

[0057] As a means to solve such problems, a technology capable of using a high-precision device such as a mouse in three-dimensional space has also been proposed. However, since a planar device such as a mouse is used to receive coordinate information on a specific plane in principle, it is difficult to point to an arbitrary spatial position relative to the three-dimensional space. In this regard, processing using a combination of a controller (such as a VR controller) capable of receiving 6DoF information and a mouse is also being sought. However, when two different pointing devices are operated at the same time, the UI operated by a traditional pointer will collapse, and therefore, multiple devices are typically installed to operate exclusively. For this reason, a technology for accurately and further improving the operability relative to pointing in three-dimensional space has been desired.

[0058] Therefore, the display control device 100 according to the contents of the present disclosure solves such a problem using the following configuration. That is, the display control device 100 obtains position and posture information in the three-dimensional space from the first controller, and displays a virtual coordinate plane in the three-dimensional space based on the position and posture information in the three-dimensional space, on which two-dimensional coordinates can be specified. In addition, the display control device 100 detects the coordinates on the virtual coordinate plane from the second controller, so that the user 10 can accurately select the desired position in the space. In this case, the first controller is a stereoscopic device 20 held by the user 10 with one hand. The second controller is a planar device 30 held by the user 10 with a hand different from the hand holding the first controller (for example, the dominant hand of the user 10).

[0059] That is, the display control device 100 enables the user 10 to select an arbitrary position by combining two different devices. Specifically, the display control device 100 displays a virtual coordinate plane ( Figure 1 ), and uses the stereoscopic device 20 as an input means for moving the surface 50. In addition, the display control device 100 displays the pointer 60 on the surface 50, and uses the planar device 30 as an input means for two-dimensionally moving the pointer 60. Therefore, after displaying the surface 50 at an arbitrary location in the space (for example, near the object 70), the user 10 performs input using the planar device 30 capable of specifying a fine position of the pointer 60, thereby enabling a pointing operation to be performed with high precision in the virtual space.

[0060] The following will refer to Figure 1 The above display control process is described. Figure 1 In the example shown, the display control device 100 displays the surface 50 in the virtual space. A virtual handle is provided on the surface 50, and the user 10 can obtain feedback as if the user holds the handle with his hand and moves the surface 50 by operating the stereoscopic device 20 holding the handle.

[0061] The user 10 places the surface 50 near the object 70. Thereafter, the user 10 operates the plane device 30 placed on the table, and specifies the position of the pointer 60 on the surface 50. The coordinates on the surface 50 and the spatial coordinates on the virtual space are associated with each other. Therefore, the display control device 100 can obtain, for example, the intersection of the extension line of the position of the pointer 60 and the outer shape (3D model) of the object 70 based on the position posture (angle) of the surface 50. Figure 1 In the example of , the display control device 100 obtains the spatial coordinates 65 of an arbitrary point of the object 70 as the position corresponding to the pointer 60. That is, the user 10 can point to a specific position of the object 70 by specifying the position of the pointer 60 on the surface 50. Therefore, the user 10 can select a point, edge, surface, etc. to be edited in the object 70 with high accuracy.

[0062] Next, we will refer to Figure 2 Another display example is described. Figure 2 is a diagram (1) showing an example of the procedure of a display control process according to an embodiment. Figure 2 The example shows that the user 10 operates the stereoscopic device 20 (step S1) and moves along the Figure 1 The same state of moving surface 50 in the direction of object 70.

[0063] In this way, the user 10 can more accurately specify a specific position of the object 70 by moving the surface 50 to a position that is almost in contact with the object 70 . Figure 2 The example of exemplifies a case where the user 10 moves the surface 50 and then operates the planar device 30 to select a specific side 66 of the object 70 .

[0064] Next, we will refer to Figure 3 Another display example is described. Figure 3 FIG. 2 is a diagram showing an example of the procedure of the display control process according to the embodiment. Figure 3 In the example shown in the upper part of Figure 2 , a state is shown in which the user 10 operates the stereoscopic device 20 to move the surface 50 in a direction toward the object 70 .

[0065] On the other hand, Figure 3 In the example shown in the lower part of , the position of the surface 50 is fixed by the operation of the user 10. In this case, the user 10 can switch the operation target to the object 70 and perform an operation of bringing the object 70 close to the surface 50. Also in this way, since the user 10 can bring the surface 50 and the object 70 relatively close to each other, accurate input can be performed.

[0066] As mentioned above Figures 1 to 3As described above, based on the display control process according to the embodiment, by displaying the surface 50 in the space, position specification can be performed in the space using the stereoscopic device 20 and the planar device 30 in combination, and thereby operability and input accuracy can be improved.

[0067] Notice, Figure 1 Each device in conceptually illustrates a function in the display control system 1, and may take various modes depending on the implementation. For example, the display control device 100 may include two or more devices that are different for each function to be described later. Specifically, the display control device 100 may be a device in which a display unit and an information processing unit are configured separately. In this case, the information processing unit of the display control device 100 may be any information processing device such as a server or a personal computer (PC).

[0068] (1-2. Configuration of Display Control Device According to Embodiment)

[0069] Next, the configuration of the display control device 100 will be described. Figure 4 is a diagram showing a configuration example of the display control device 100 according to the embodiment.

[0070] like Figure 4 As shown, the display control device 100 includes a communication unit 110, a storage unit 120, a control unit 130, a sensor unit 140, and a display unit 150. Note that the display control device 100 may include an input unit (operation button, touch panel, etc.) that receives various operations from the user 10 who operates the display control device 100.

[0071] The communication unit 110 is implemented by, for example, a network interface card (NIC), a network interface controller, etc. The communication unit 110 is connected to the network N in a wired or wireless manner to transmit and receive information to and from the stereoscopic device 20, the planar device 30, etc. via the network N. The network N is implemented by, for example, a wireless communication standard or a wireless communication system such as Bluetooth (registered trademark), the Internet, Wi-Fi (registered trademark), ultra-wideband (UWB), or low power wide area (LPWA).

[0072] The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) and a flash memory, or a storage device such as a hard disk and an optical disk.

[0073] The storage unit 120 stores various types of information related to the display control process according to the embodiment. For example, the storage unit 120 stores a 3D model of a virtual object displayed on the display unit 150, etc.

[0074] The sensor unit 140 is a sensor that detects various environmental information. For example, the sensor unit 140 includes an external camera that captures an image of the outside of the display control device 100 and an internal camera that captures an image on the user 10 side.

[0075] For example, the sensor unit 140 has a function as an identification camera device for identifying the space in front of the user. For example, the sensor unit 140 identifies the position of the stereoscopic device 20 in space by imaging the stereoscopic device 20 whose spatial position has been pre-calibrated, and obtains the position posture information of the stereoscopic device 20.

[0076] In addition, the sensor unit 140 identifies a subject located in front of the display control device 100 (for example, a real object located in the real space). In this case, the sensor unit 140 may acquire an image of the subject located in front of the user, and calculate the distance from the display control device 100 (in other words, the user's viewpoint position) to the subject based on the parallax between the images captured by the stereo camera. Alternatively, the sensor unit 140 may detect the distance in the real space using a depth sensor capable of detecting the distance of a real object.

[0077] In addition, in addition to the function of being an identification camera device, the sensor unit 140 may also have a function of detecting various types of information about the user's movement, such as the orientation, inclination, movement, and movement speed of the user's body. Specifically, the sensor unit 140 detects information about the user's head and posture, the movement of the user's head and body (acceleration and angular velocity), the direction of the field of view, the speed of the viewpoint movement, etc. as information about the user's movement. For example, the sensor unit 140 is used as various movement sensors such as a three-axis acceleration sensor, a gyro sensor, and a speed sensor, and detects information about the user's movement. More specifically, the sensor unit 140 detects corresponding components in the yaw direction, the pitch direction, and the roll direction as the movement of the user's head, thereby detecting a change in at least any one of the position and posture of the user's head. Note that the sensor unit 140 is not necessarily provided in the display control device 100, and may be, for example, an external sensor connected to the display control device 100 in a wired manner or wirelessly.

[0078] The display unit 150 displays various types of information output from the control unit 130. For example, the display unit 150 is a display that outputs a video to the user 10. Note that the display unit 150 may include a sound output unit (speaker, etc.) that outputs sound.

[0079] The control unit 130 is implemented by, for example, a central processing unit (CPU), a microprocessing unit (MPU), a GPU, etc., using a RAM, etc. as a work area to execute a program stored inside the display control device 100. The control unit 130 may be a controller and may be implemented by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0080] like Figure 4 As shown, the control unit 130 includes an acquisition unit 131 , a first detection unit 132 , a second detection unit 133 and a display control unit 134 .

[0081] The acquisition unit 131 acquires various types of information. For example, the acquisition unit 131 acquires position and posture information in a three-dimensional space from the stereoscopic device 20, which is the first controller. In addition, the acquisition unit 131 acquires position information from the planar device 30, which is the second controller, and the position information is information of the position on the indication surface 50.

[0082] The first detection unit 132 detects the movement of the surface 50 in the three-dimensional space based on the input of the stereoscopic device 20. Specifically, the first detection unit 132 combines the image of the stereoscopic device 20 captured by the sensor unit 140 with the position and posture information transmitted from the stereoscopic device 20 to detect (calculate) the position and posture information of the stereoscopic device 20 in the space. That is, the position and posture information of the stereoscopic device 20 is obtained by combining the image information obtained by imaging the stereoscopic device 20 using the external camera device of the sensor unit 140 with the inertial measurement unit (IMU) information included in the stereoscopic device 20. Note that the input of the stereoscopic device 20 means that the position and posture information (or its change value) measured by the IMU or the like through the operation of the stereoscopic device 20 in the space is input to the display control device 100 via the communication unit 110.

[0083] For example, the first detection unit 132 detects movement of the surface 50 and the position and posture information by the user 10 operating the stereoscopic device 20 while holding a handle virtually provided on the surface 50 and detecting a change value of the position and posture information obtained based on the operation result.

[0084] In addition to the above-mentioned processing, various known technologies may be used to detect the position of the stereoscopic device 20. For example, in the case where an infrared light emitting diode (LED) is embedded in the stereoscopic device 20, the first detection unit 132 may detect the position and posture of the stereoscopic device 20 by observing the state in which the infrared LED is emitted by using the external camera of the sensor unit 140 and combining the IMU information. In addition, the first detection unit 132 may also detect the position and posture of the stereoscopic device 20 by imaging the hand of the user 10 using the camera of the sensor unit 140, the camera installed in the room, etc. using the hand tracking technology to estimate the position and posture of the hand.

[0085] The first detection unit 132 may detect the relative movement of the surface 50, rather than the movement of the surface 50 itself. That is, the first detection unit 132 may detect the movement of an object, whose position can be specified using the surface 50, based on the input of the stereoscopic device 20. In this case, the first detection unit 132 receives the designation of the object to be moved by the user 10, and performs a process of moving the object closer to or farther from the surface 50 according to the operation of the user 10.

[0086] The second detection unit 133 detects the position specified by the plane 50 (for example, the position of the pointer indicating the specified position) based on the input of the plane device 30 as the second controller. Since the position and posture information (the tilt of the plane, etc.) of the plane 50 in space is given based on the operation of the stereoscopic device 20, the pointer located on the plane 50 includes not only the two-dimensional position (the coordinates of the plane 50) but also the position and posture information corresponding to the plane 50.

[0087] In addition, the second detection unit 133 detects a selected portion of the object based on the designated position on the surface 50. For example, the second detection unit 133 detects a portion such as a point, edge, or surface of the object corresponding to the position indicated by the pointer based on designation by the user 10 or initial setting.

[0088] At this time, the second detection unit 133 can detect the selected part of the object based on the relationship between the angle of the surface 50 and the position of the pointer and the position of the object. Specifically, the second detection unit 133 detects a point intersecting the object on the extension line of the pointer obtained from the angle of the surface 50 and the position of the pointer as the part designated by the user 10.

[0089] For example, the second detection unit 133 may detect a point intersecting the object on an extension line with respect to the normal direction of the surface 50 having the pointer position as a starting point, as the portion designated by the user 10. Alternatively, the second detection unit 133 may detect a point intersecting the object on an extension line connecting the viewpoint position of the user 10 and the pointer position, as the portion designated by the user 10. Alternatively, regardless of the angle of the surface 50, the second detection unit 133 may detect a selected portion of the object based on a position intersecting the object on a line extending horizontally from the pointer position designated on the surface 50.

[0090] The display control unit 134 causes the display unit 150 to display the information output from the control unit 130. That is, the display control unit 134 outputs the virtual space video rendered as the video content to the output destination device. Note that the output destination device is not limited to a head-mounted display such as the display unit 150, and may be a video output device such as an external display, a smartphone, or a television.

[0091] For example, the display control unit 134 displays a surface 50 , which is a virtual coordinate plane on which two-dimensional coordinates can be specified, in the three-dimensional space based on the position and posture information of the stereoscopic device 20 in the three-dimensional space.

[0092] Furthermore, the display control unit 134 performs control to display the pointer on the surface 50 based on the position information on the surface 50 input from the plane device 30. Furthermore, in the case where any part in the object is selected based on the pointer position, the display control unit 134 clearly indicates the selected position.

[0093] (1-3. Processing procedure according to embodiment)

[0094] Will refer to Figure 5 A processing procedure according to an embodiment is described. Figure 5 is a sequence diagram showing the flow of display control processing according to the embodiment.

[0095] like Figure 5 As shown, the stereoscopic device 20 acquires the position and posture information of the stereoscopic device using an IMU or the like (step S11 ). Then, the stereoscopic device 20 transmits the acquired position and posture information to the display control device 100 (step S12 ).

[0096] The display control device 100 uses an external camera or the like to image the stereoscopic device 20 (step S21 ). Then, the display control device 100 estimates the position and posture of the stereoscopic device 20 (step S22 ).

[0097] Thereafter, the display control device 100 recognizes the position and posture of the stereoscopic device 20 by combining the position and posture information acquired from the stereoscopic device 20 with the position and posture information obtained by imaging (step S23). The display control device 100 detects the position and posture of the surface 50 in the virtual space based on the position and posture information of the stereoscopic device 20 (step S24). The display control device 100 displays the surface 50 on the display unit 150 based on the detected position and posture of the surface 50.

[0098] When the position of the surface 50 is determined, the plane device 30 acquires plane coordinate information for identifying the position on the surface 50 (step S31). Then, the plane device 30 transmits the acquired plane coordinate information to the display control device 100 (step S32).

[0099] The display control device 100 detects the position of the pointer on the surface 50 based on the information acquired from the plane device 30 (step S25). Then, the display control device 100 detects the position designated by the user 10 in the object based on the position posture of the pointer (step S26). Note that in this case, the posture information of the pointer is obtained based on the angle of the surface 50, for example.

[0100] (1-4. Modification)

[0101] (1-4-1. First modification: operation with only one device)

[0102] The processing according to the above-described embodiment may be accompanied by various modifications. For example, in the above-described embodiment, an example is described in which the user 10 uses both the stereoscopic device 20 and the planar device 30 to operate the surface 50 and the pointer. However, the user 10 may also use any one of the devices to perform information processing according to the embodiment.

[0103] That is, even if the first controller (stereoscopic device 20) and the second controller (planar device 30) are the same device, the display control device 100 can perform display control processing according to the contents of the present disclosure. In this case, when the same device is located at a predetermined distance from the plane in the real space, the first detection unit 132 detects the same device as the first controller. In addition, when the same device is located within a predetermined distance from the plane in the real space, the second detection unit 133 detects the same device as the second controller.

[0104] Alternatively, in the case where a designation to use the same device as a first controller is received from the user 10 of the same device, the first detection unit 132 may detect the same device as a first controller. In addition, in the case where a designation to use the same device as a second controller is received from the user 10 of the same device, the second detection unit 133 may detect the same device as a second controller.

[0105] Will refer to Fig. 6A and Figure 6B The above-mentioned processing is described. Fig. 6A is a diagram for describing the operation of the device according to the first modification.

[0106] exist Fig. 6A In the left part of FIG. 1 , the user 10 operates the stereoscopic device 20 and the planar device 30 as in the embodiment.

[0107] On the other hand, Fig. 6A In the right part of the image, the user 10 operates only the plane device 30. In this case, when the plane device 30 is lifted from the table, the display control device 100 recognizes the operation. Then, when the plane device 30 is in the air, the display control device 100 recognizes the plane device 30 as a device that performs the same operation as the stereoscopic device 20. Specifically, when the plane device 30 is in the air, the display control device 100 recognizes the movement of such a device as a device for moving the surface 50.

[0108] Then, when the user 10 places the plane device 30 on the desk, the display control device 100 recognizes the plane device 30 as a device for operating the pointer on the surface 50. Through such processing, the user 10 can realize the processing according to the embodiment using only one device.

[0109] Note that the display control device 100 can use the stereoscopic device 20 in the same manner as the planar device 30. For example, in the case where the stereoscopic device 20 is moved near the desktop, the display control device 100 recognizes the movement and uses the stereoscopic device 20 as the planar device 30. More specifically, in the case where the stereoscopic device 20 is located near the desktop, the display control device 100 invalidates the input of the position gesture from the stereoscopic device 20, and can use the stereoscopic device as the planar device 30 that touches the desktop like a stylus pen to receive position information.

[0110] Note that, in the case where the processing according to the embodiment is performed by only one device, the display control device 100 may explicitly receive a switching operation from the user 10 instead of the above processing. For example, in the case where the stereoscopic device 20 has a function (trackball, etc.) capable of recognizing a two-dimensional position, the user 10 can flexibly perform the above switching.

[0111] Furthermore, when the display unit 150 of the display control device 100 can be temporarily moved to the upper part (referred to as flip-up, etc.), the display control device 100 can use the stereoscopic device 20 as the flat device 30 during flip-up. This is because the user 10 cannot perform stereoscopic viewing during flip-up.

[0112] Will refer to Figure 6B The flow of the above-mentioned processing is described. Figure 6B is a flowchart showing the flow of the display control process according to the first modification.

[0113] The display control device 100 determines whether the two devices, the device of the operation surface 50 and the device performing the pointer operation, are different devices (step S41). In the case of different devices (step S41; Yes), as in the embodiment, the display control device 100 sets the movement according to the original setting in each device.

[0114] In the case where the two devices, the device of the operation surface 50 and the device for performing the pointer operation, are not different devices (step S41; No), the display control device 100 determines whether the device is recognized as being on the desktop (on an arbitrary plane in the three-dimensional space) (step S42). When the device is on the desktop (step S42; Yes), the display control device 100 uses the device as the plane device 30.

[0115] On the other hand, if the device is not on the desk (step S42; No), the display control device 100 determines whether the user 10 designates the device as a flat device 30 (step S43). If the user 10 designates the device as a flat device 30 (step S43; Yes), the display control device 100 uses the device as a flat device 30.

[0116] On the other hand, when the device is not specified as the flat device 30 (step S43 ; No), the display control device 100 determines whether the HMD is turned upside down (step S44 ).

[0117] When the HMD is not turned up (step S44; No), the display control device 100 uses the device as the stereoscopic device 20. On the other hand, when the HMD is turned up (step S44; Yes), the display control device 100 uses the device as the flat device 30.

[0118] (1-4-2. Second modification: Adjustment of operational feel)

[0119] In the above embodiment, an example is described in which the user 10 moves the surface 50 by operating the stereoscopic device 20. For example, a virtual handle is provided on the surface 50, and the user 10 can hold the handle by operating the stereoscopic device 20, thereby moving the surface 50 as if the surface 50 is a real object. However, in the virtual space, it is assumed that it is desired to move the surface 50 farther than the distance actually reached by the user 10. Alternatively, in the virtual space, it is assumed that the surface 50 is installed farther than the distance that the user 10 extends the hand and the hand cannot be held.

[0120] Will refer to Figure 7Describe the process in such a case. Figure 7 is a diagram (1) for describing the display control process according to the second modification. Figure 7 In the left part of , as in the embodiment, the user 10 holds the handle of the surface 50 with a hand and moves the hand a distance 200, thereby moving the surface 50.

[0121] On the other hand, Figure 7 , the user 10 desires to move the surface 50 further than the distance 205 that the hand actually moves. In this case, the display control device 100 provides guidance to the user 10 by, for example, displaying a ray in a direction from the hand toward the surface 50 on the virtual space. For example, the user 10 can cause the guidance to be displayed by explicitly selecting a display mode such as a remote control mode that is pre-set by the display control device 100. In this case, the display control device 100 can move the surface 50 a distance longer than the actual moving distance of the hand only by the movement of the hand of the user 10, without the user 10 holding the hand of the surface 50.

[0122] Furthermore, in the case where the surface 50 is installed far away, the display control device 100 can operate as if the user 10 grasps the handle based on the guide extending from the hand, and move the surface 50 .

[0123] Furthermore, in the case where the user 10 holds a region of the handle in the space that is not the surface 50, the display control apparatus 100 may perform display control to move an object other than the surface 50, such as Figure 3 shown.

[0124] In addition, the display control device 100 can perform control so that not only the surface 50 but also the object can be remotely operated. Figure 8 Give a description. Figure 8 is a diagram (2) for describing the display control process according to the second modification.

[0125] exist Figure 8 , the display control device 100 displays the guide 210 for penetrating the surface 50 and grasping the object 70. The user 10 can grasp and select the object 70 via the guide 210. Therefore, the user 10 can perform operations such as grasping or moving even the object 70 installed at a position that cannot be reached by hand in the virtual space.

[0126] Note that in a case where the user 10 desires to select a portion of the object 70, when the object 70 and the surface 50 are in contact with each other, a situation may occur where the portion of the object 70 that is in contact can be selected, but the portion that is not in contact cannot be selected. In this case, the display control device 100 can output a ray (guide) in a specific direction from the pointer on the surface by switching the operation mode of the pointer, so that the portion of the object 70 can be selected. The specific direction is, for example, a direction that can be determined based on the viewpoint position of the user 10 and the pointer position, such as a normal direction relative to the surface 50 or a direction connecting the viewpoint position of the user 10 and the pointer. In addition, in order to make it easy for the user 10 to select a portion of the object 70, the display control device 100 can move the pointer on the surface to the contact portion when the object 70 and the surface 50 change from a state of not contacting each other to a state of contacting each other.

[0127] In addition, the display control device 100 can fix the position of the surface 50 so as not to move to a position where the surface 50 cannot be held in the virtual space. That is, regardless of the position posture information of the stereoscopic device 20 in the three-dimensional space, the display control unit 134 can display the surface 50 in the three-dimensional space based on the viewpoint position of the user 10. This point will be referred to Fig. 9 Give a description. Fig. 9 is a diagram (3) for describing the display control process according to the second modification.

[0128] exist Fig. 9 In the example of , the display control device 100 performs control to display the surface 50 at a specific distance 215 from the display control device 100. In this case, the surface 50 moves with the movement of the display control device 100. Therefore, the user 10 can move the surface 50 to select an arbitrary position of the object 70 by directing the line of sight to the object 70 without using the stereoscopic device 20. Note that when the surface 50 is constantly moving with the movement of the display control device 100, it is difficult to keep the surface 50 at a constant position, and therefore the display control device 100 can turn on or off the automatic movement of the surface 50 in response to a request from the user 10.

[0129] Note that the display control device 100 may change the position and posture of the surface 50 while changing the size of the surface 50. For example, the display control device 100 changes the size of the surface 50 in conjunction with the user 10 rotating the mouse wheel. In addition, the display control device 100 may change the color of the surface 50.

[0130] In addition, the display control device 100 can perform a process of cutting out a cross section of an object in a virtual space, a process of hiding an object even slightly in an area in front of the surface 50, etc. according to the position where the surface 50 is installed. That is, the display control device 100 can change the display content of the object according to the position of the surface 50. Therefore, the display control device 100 can improve the visibility of the object.

[0131] (1-4-3. Third modification: Processing superimposed on real space)

[0132] In addition, the display control device 100 can superimpose the surface 50 on the real space as in the AR display. In this case, the display control device 100 can adjust the movement by adsorbing the surface 50 to the face or side of the object in the real world, so that the user can easily adjust the position. That is, when the display control unit 134 displays the surface 50 in the three-dimensional space of the real space, when the surface 50 is close to a predetermined distance from any plane in the real space, the first detection unit 132 can detect the surface 50 at the following position based on the input of the stereoscopic device 20, which is attracted (adsorbed) to the plane so that the surface 50 is displayed as superimposed on the plane.

[0133] This will refer to Fig.10 and subsequent figures for description. Fig.10 is a diagram (1) for describing the display control process according to the third modification.

[0134] Fig.10 The state where the surface 50 is superimposed on the real space is shown. Specifically, Fig.10 The example shows a state where the surface 50 provided with the handle 220 is displayed so as to be adhered to the screen of the real display 225. Note that, in the case where the plane to which the surface 50 is adsorbed is recognized, the display control device 100 displays the guide 230 along the plane (the outer frame of the display 225 in this example).

[0135] For example, when the user 10 holds the handle 220 and brings the surface 50 close to the display 225 , the display control device 100 recognizes that the display 225 and the surface 50 have approached within a predetermined distance and causes the surface 50 to be adsorbed to the display 225 along the guide 230 .

[0136] Therefore, the user 10 can use the closely attached surface 50 when selecting an object to be projected on the display 225, etc., so that the selection operation, etc. can be accurately performed even in the AR space. Note that the display control device 100 can attach the surface 50 to the side adjacent to the screen instead of the front of the display 225. Therefore, the user 10 can use the display 225 as a normal monitor and perform operations, etc. on the surface 50 virtually displayed next to the display.

[0137] Another example will refer to Fig.11 Give a description. Fig.11 is a diagram (2) for describing the display control process according to the third modification.

[0138] exist Fig.11 In the example of , the display control device 100 recognizes the table 240 as a plane to which the surface 50 is attached. That is, when the user 10 holds the handle 220 and brings the surface 50 close to the table 240, the display control device 100 recognizes that the top surface of the table 240 and the surface 50 have approached within a predetermined distance, and causes the surface 50 to be adsorbed onto the top surface of the table 240.

[0139] Therefore, the user 10 can use the display 225 as a normal monitor and use the table 240 on which the display 225 is placed as another area for performing other work.

[0140] In addition, the information processing on the surface 50 set on the desktop will refer to Fig.12 Give a description. Fig.12 is a diagram (3) for describing the display control process according to the third modification.

[0141] Fig.12 The example shows a state in which the user 10 selects a virtual object 305 displayed behind the surface 50 using the touch pen 300 for a virtual pointer for coordinate designation on the surface 50. Note that the touch pen 300 is displayed, for example, when the user 10 brings the stereoscopic viewing device 20 close to the table, and is operated based on IMU information of the stereoscopic viewing device 20, etc.

[0142] It should be noted that in Fig.12 In the example of , since the virtual object 305 is located behind the surface 50, that is, below the desktop, the desktop interferes with the real movement, and the virtual object 305 cannot be selected using the touch pen 300 using the stereoscopic vision device 20. Therefore, the display control device 100 performs adjustments related to the selection operation.

[0143] Specifically, when the display control unit 134 virtually displays an object and a pointer (in this example, the touch pen 300) for selecting the object in the three-dimensional space of the real space, the second detection unit 133 can control the size or contact point of the pointer so that the user 10 can virtually touch the object using the pointer. This point will be referred to Fig.13 Give a description. Fig.13 is a diagram (4) for describing the display control process according to the third modification.

[0144] Fig.13The left part of shows an example in which the display control device 100 moves the touch pen 300 by the same movement amount as the actual movement of the stereoscopic device 20. In this case, the touch pen 300 collides with the desktop, and the feeling that the user 10 touches the virtual object 305 cannot be obtained.

[0145] Therefore, when the virtual object 305 is recognized to be behind the surface 50, the display control device 100 dynamically changes the length of the touch pen 300, such as Fig.13 Specifically, when the touch pen 300 comes into contact with the virtual object 305, the display control device 100 adjusts the stereoscopic device 20 so that the stereoscopic device actually collides with the desktop. Therefore, the display control device 100 can provide accurate tactile feedback to the user 10 and can perform accurate display control processing without passing through the virtual object 305.

[0146] Fig.14 A display example of the touch pen 300 is shown. Fig.14 is a diagram (5) for describing the display control process according to the third modification.

[0147] For example, the display control device 100 determines the length of the touch pen 300 based on the distance to the virtual object 305 that the user 10 wants to select. Specifically, the display control device 100 sets the distance 310 between the desktop and the stereoscopic device 20 to be the same as the distance 315 between the tip of the touch pen 300 and the virtual object 305.

[0148] In addition, the user 10 can change the size of the surface 50 according to the situation at the work place. That is, the first detection unit 132 can control the position and size of the surface 50 virtually displayed in the real space based on the operation of the stereoscopic device 20 by the user 10. This will be referred to Fig.15 Give a description. Fig.15 is a diagram (6) for describing the display control process according to the third modification.

[0149] exist Fig.15 In the example shown, the user 10 performs work using the surface 50 pasted on the table 240. At this time, the user 10 can perform an operation of enlarging the surface 50 by holding the handle 220, thereby enlarging the surface 50 so as to cover the surface of the table 240. That is, the display control device 100 enlarges or reduces the size of the surface 50 according to the operation of the user 10. Therefore, the user 10 can effectively use the plane in the real space to which the surface 50 is pasted, and thus better operability can be obtained.

[0150] In addition, when the user 10 selects a virtual object, the display control device 100 can perform control so that an appropriate selection can be performed by expanding the designated position, etc. That is, the second detection unit 133 can expand the selection position made by the pointer and control the operation of the pointer so that the user 10 can virtually operate the object. This point will be referred to Fig.16 Give a description. Fig.16 is a diagram (7) for describing the display control process according to the third modification.

[0151] exist Fig.16 In the example of , the surface 50 is displayed at an angle of 45 degrees relative to the display 225. In addition, the camera 320 as a virtual object is displayed behind the surface 50. In this case, since the camera 320 is not on the surface 50, the user 10 cannot directly operate the camera 320.

[0152] At this time, the display control device 100 extends a ray from the pointer 330 displayed by the user 10, and displays a guide 335 indicating the extended line and a pointer 340 which is an intersection of the guide 335 and the camera 320. Therefore, even if the pointer 340 cannot be moved to a position where the user actually touches the camera 320, the user 10 can operate the camera 320 using the extended pointer 330.

[0153] The display control device 100 can display the surface 50 at any angle in the AR display. Fig.17 Give a description. Fig.17 is a diagram (8) for describing the display control process according to the third modification.

[0154] exist Fig.17 In the example of , the user 10 holds the handle 220 and moves the surface 50. In this case, relative to the plane in the space (in this example, the screen of the display 225), the display control device 100 can display the surface 50 at a right angle (90 degrees) relative to the screen of the display 225, or can display the surface 50 at an angle of 45 degrees or 0 degrees (i.e., the desktop). That is, since the display control device 100 can receive any angle from the user 10, the work area can be constructed according to the user 10's expectations.

[0155] Note that, in the case where the user 10 desires to design or edit a virtual object, the display control device 100 may display the surface 50 at a position offset by a predetermined distance relative to the target object. In this case, since the user 10 can perform a selection operation on the surface 50 offset from the object, or perform work such as drawing an object, a work environment with good visibility can be obtained.

[0156] (1-4-4. Fourth modification: deformation of the surface based on the pointer position)

[0157] The display control device 100 can change the size of the surface 50 itself according to the operation of the user 10. That is, the second detection unit 133 can perform control so that the position or size of the surface 50 is changed based on the position (for example, the display position of the pointer) specified by the user 10 on the surface 50. This point will be referred to Fig.18 Give a description. Fig.18 is a diagram for describing a display control process according to a fourth modification.

[0158] exist Fig.18 In the example above, the pointer 350 is displayed at the right end position relative to the surface 50. In this case, the display control device 100 can be as follows Fig.18 The display surface 50 and the pointer 350 are fixedly displayed as shown in the lower left part of the display. In addition, the display control device 100 can move the surface 50 to the right when the pointer 350 reaches the right end, as shown in FIG. Fig.18 In addition, the display control device 100 can enlarge and display the surface 50 at the moment when the pointer 350 reaches the right end, as shown in the lower middle part of FIG. Fig.18 The display control apparatus 100 can switch the display control process by the selection of the user 10. Therefore, the user 10 can automatically change the work area according to the movement of the pointer 350, so that better operability can be obtained.

[0159] (2. Other Implementation Methods)

[0160] The process according to each embodiment described above may be performed in various different forms other than the above-described embodiment.

[0161] In addition, all or part of the processing described in the above-described embodiments as being automatically performed may also be performed manually, or alternatively, all or part of the processing described as being manually performed by a known method may also be performed automatically. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above-mentioned documents and drawings are allowed to be changed arbitrarily unless otherwise specified. For example, the various types of information shown in each figure are not limited to the information shown.

[0162] In addition, each component of each of the devices shown is a functional concept and does not necessarily have to be physically configured as shown in the figure. That is, the specific distribution / integration form of each device is not limited to the one shown in the figure, and it is allowed to functionally or physically distribute / integrate all or part of the device into any unit according to various loads and usage conditions.

[0163] Furthermore, it is allowed to appropriately combine the above-described embodiments and modifications within a range in which processing details do not contradict each other.

[0164] Furthermore, the effects described in this specification are merely examples and are not restrictive, and other effects may exist.

[0165] (3. Effects of the display control device according to the present disclosure)

[0166] As described above, the display control device (display control device 100 in the embodiment) according to the present disclosure includes an acquisition unit (acquisition unit 131 in the embodiment) and a display control unit (display control unit 134 in the embodiment). The acquisition unit acquires position and posture information in the three-dimensional space from the first controller (stereoscopic device 20 in the embodiment). The display control unit displays a virtual coordinate plane (surface 50 in the embodiment) in the three-dimensional space based on the position and posture information in the three-dimensional space, on which two-dimensional coordinates can be specified.

[0167] As described above, the display control device according to the present disclosure can also display a plane capable of specifying two-dimensional coordinates in a three-dimensional space, thereby improving input accuracy, such as pointing to a specific location of an object in space.

[0168] In addition, the display control device includes a first detection unit (the first detection unit 132 in the embodiment) that detects the movement of the virtual coordinate plane in the three-dimensional space based on the input of the first controller. For example, the first detection unit detects the movement of an object based on the input of the first controller, wherein the position of the object can be specified using the virtual coordinate plane.

[0169] As described above, the display control device performs control so that the user can freely move the surface using a VR controller etc. Therefore, the user can perform input operations with a high degree of freedom.

[0170] In addition, the display control device detects the position specified on the virtual coordinate plane based on the input of the second controller (the plane device 30 in the embodiment). For example, the second detection unit detects the selected part of the object based on the position specified on the virtual coordinate plane. Specifically, the second detection unit detects the selected part of the object based on the angle of the virtual coordinate plane and the position specified by the user on the virtual coordinate plane.

[0171] As described above, since the display control device acquires input from the user using a flat device such as a mouse, it is possible to receive accurate input compared to a VR controller or the like.

[0172] In addition, the second detection unit can detect a point that intersects the object on an extension line of the normal direction of the virtual coordinate plane with the specified position as the starting point, or a point that intersects the object on an extension line of the line connecting the user's viewpoint position and the specified position, as a selected part of the object.

[0173] In this way, the display control device can selectively switch the behavior of determining the part selected by the pointer on the surface. The user can avoid behaviors that are stressful to the user, such as selecting an erroneous position, depending on the pointer position, and can achieve the operability desired by the user.

[0174] In the display control process performed by the display control device, the first controller and the second controller may be the same device. In this case, when the same device is located at a predetermined distance from the plane in the real space, the first detection unit may detect the same device as the first controller. In addition, when the same device is located within a predetermined distance from the plane in the real space, the second detection unit may detect the same device as the second controller. Alternatively, in the case of receiving a designation to use the same device as the first controller from a user of the same device, the first detection unit may detect the same device as the first controller. In the case of receiving a designation to use the same device as the second controller from a user of the same device, the second detection unit may detect the same device as the second controller.

[0175] As described above, since the display control device can realize the display control process according to the present disclosure with one device, it is possible to provide the user with information processing for realizing high input accuracy even in various facility environments.

[0176] Furthermore, regardless of the position posture information of the first controller in the three-dimensional space, the display control unit displays the virtual coordinate plane in the three-dimensional space based on the viewpoint position of the user.

[0177] As described above, the display control device can reduce the user's operation process by fixing and displaying the surface, so that information processing with less burden on the user can be achieved.

[0178] In addition, the display control unit displays the virtual coordinate plane in the three-dimensional space of the real space. When the virtual coordinate plane approaches an arbitrary plane in the real space by a predetermined distance, the first detection unit detects the virtual coordinate plane at a position that is attracted to the plane so that the virtual coordinate plane is superimposed and displayed on the plane based on an input from the first controller.

[0179] As described above, the display control device can realize the behavior of adsorbing the surface to the plane of the real space. Therefore, the user can easily set the position where the plane device on the desktop can be easily used as a surface, so that work efficiency can be improved.

[0180] In addition, the display control unit virtually displays the object and the pointer for selecting the object in the three-dimensional space of the real space. The second detection unit controls the size or contact point of the pointer so that the user can virtually contact the object using the pointer.

[0181] As described above, the display control apparatus adjusts the contact point (ie, feedback feeling) and the size of the pointer according to the real space, thereby providing the user with an operation feeling according to the feeling of the real space.

[0182] In addition, the first detection unit controls the position and size of the virtual coordinate plane virtually displayed in the real space based on the user's operation of the first controller.

[0183] As described above, the display control device enables the user to freely adjust the work area by freely changing the size of the face according to the user's desire, so that the operability can be improved.

[0184] Furthermore, the second detection unit expands the selection position by the pointer and controls the operation of the pointer so that the user can virtually operate the object.

[0185] The display control apparatus can improve the operability for the user by performing a process of eliminating the inconvenience of the virtual space such as an object being located far away.

[0186] Furthermore, the second detection unit performs control to change the position or size of the virtual coordinate plane based on the position designated by the user on the virtual coordinate plane.

[0187] As described above, the display control device can provide a comfortable working environment for the user by freely changing the size of the face according to the position of the pointer.

[0188] (4. Hardware Configuration)

[0189] According to each of the above-described embodiments, an information device such as the display control device 100 is composed of, for example, a display device having Fig.19 The computer 1000 of the illustrated configuration implements. Hereinafter, the display control device 100 will be described as an example. Fig.19 1 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the display control device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. The units of the computer 1000 are connected via a bus 1050.

[0190] The CPU 1100 operates based on the program stored in the ROM 1300 or the HDD 1400 and controls each unit. For example, the CPU 1100 develops the program stored in the ROM 1300 or the HDD 1400 in the RAM 1200 and executes processing corresponding to various programs.

[0191] The ROM 1300 stores a boot program such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, a program depending on the hardware of the computer 1000 , and the like.

[0192] The HDD 1400 is a computer-readable recording medium that non-temporarily records a program executed by the CPU 1100 , data used by the program, and the like. Specifically, the HDD 1400 is a recording medium that records a display control program according to the present disclosure, which is an example of the program data 1450 .

[0193] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from another device via the communication interface 1500 or transmits data generated by the CPU 1100 to another device.

[0194] The input / output interface 1600 is an interface that connects the input / output device 1650 to the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard and a mouse via the input / output interface 1600. In addition, the CPU 1100 transmits data to an output device such as a display, an edge, or a printer via the input / output interface 1600. In addition, the input / output interface 1600 can be used as a media interface for reading a program recorded in a predetermined recording medium (medium), etc. The medium is, for example, an optical recording medium such as a digital versatile disc (DVD) or a phase change rewritable disc (PD), a magneto-optical recording medium such as a magneto-optical disc (MO), a magnetic tape medium, a magnetic recording medium, a semiconductor memory, etc.

[0195] For example, in the case where the computer 1000 is used as the display control device 100 according to the embodiment, the CPU 1100 of the computer 1000 implements the functions of the control unit 130 and the like by executing the display control program loaded on the RAM 1200. In addition, the HDD 1400 stores the display control program according to the present disclosure as well as the data in the storage unit 120. The CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program data, but as another example, the program may be acquired from another device via the external network 1550.

[0196] The present technology may also be configured as follows.

[0197] (1) A display control device comprising:

[0198] an acquisition unit, which acquires position and posture information in a three-dimensional space from the first controller; and

[0199] A display control unit displays a virtual coordinate plane in the three-dimensional space based on the position and posture information in the three-dimensional space, on which two-dimensional coordinates can be specified.

[0200] (2) The display control device according to (1), further comprising:

[0201] A first detection unit detects movement of the virtual coordinate plane in the three-dimensional space based on an input of the first controller.

[0202] (3) The display control device according to (2), wherein:

[0203] The first detection unit detects movement of an object whose position can be specified using the virtual coordinate plane based on an input of the first controller.

[0204] (4) The display control device according to (2) or (3), further comprising:

[0205] A second detection unit detects a position designated on the virtual coordinate plane based on an input from a second controller.

[0206] (5) The display control device according to (4), wherein:

[0207] The second detecting unit detects a selected portion of the object based on the position specified on the virtual coordinate plane.

[0208] (6) The display control device according to (5), wherein:

[0209] The second detection unit detects a selected portion of the object based on an angle of the virtual coordinate plane and a position designated by a user on the virtual coordinate plane.

[0210] (7) The display control device according to (6), wherein:

[0211] The second detection unit detects a point that intersects the object on an extension line of the normal direction of the virtual coordinate plane with the specified position as the starting point, or a point that intersects the object on an extension line of the line connecting the user's viewpoint position and the specified position, as a selected part of the object.

[0212] (8) The display control device according to any one of (4) to (7), wherein:

[0213] The first controller and the second controller are the same device.

[0214] (9) The display control device according to (8), wherein:

[0215] The first detection unit detects the same device as the first controller in a case where the same device is located at a predetermined distance from a plane in the real space, and

[0216] The second detection unit detects the same device as the second controller in a case where the same device is located within a predetermined distance from a plane in the real space.

[0217] (10) The display control device according to (8), wherein:

[0218] The first detection unit detects the same device as the first controller when a designation to use the same device as the first controller is received from a user of the same device, and

[0219] The second detection unit detects the same device as the second controller when receiving a designation to use the same device as the second controller from a user of the same device.

[0220] (11) The display control device according to any one of (2) to (10), wherein:

[0221] The display control unit displays the virtual coordinate plane in the three-dimensional space based on the viewpoint position of the user regardless of the position posture information of the first controller in the three-dimensional space.

[0222] (12) The display control device according to any one of (5) to (7), wherein:

[0223] The display control unit displays the virtual coordinate plane in the three-dimensional space of the real space, and

[0224] When the virtual coordinate plane approaches an arbitrary plane in the real space by a predetermined distance, the first detection unit detects the virtual coordinate plane at a position attracted to the plane so that the virtual coordinate plane is superimposed and displayed on the plane based on an input of the first controller.

[0225] (13) The display control device according to (12), wherein:

[0226] The display control unit virtually displays the object and a pointer for selecting the object in the three-dimensional space of the real space, and

[0227] The second detection unit controls a size or a contact point of the pointer so that a user can virtually contact the object using the pointer.

[0228] (14) The display control device according to (12) or (13), wherein:

[0229] The first detection unit controls a position and a size at which the virtual coordinate plane is virtually displayed in a real space based on a user's operation of the first controller.

[0230] (15) The display control device according to (13), wherein:

[0231] The second detection unit expands a selection position of the pointer and controls an operation of the pointer so that a user can virtually operate the object.

[0232] (16) The display control device according to any one of (4) to (7), wherein:

[0233] The second detection unit performs control based on a position designated by a user on the virtual coordinate plane so as to change a position or a size of the virtual coordinate plane.

[0234] (17) A display control method executed by a computer, the method comprising:

[0235] Acquiring position and posture information in three-dimensional space from the first controller; and

[0236] A virtual coordinate plane is displayed in the three-dimensional space based on the position and posture information in the three-dimensional space, and two-dimensional coordinates can be specified on the virtual coordinate plane.

[0237] Reference numerals list

[0238] 1Display control system

[0239] 10 users

[0240] 20 Stereoscopic Installation

[0241] 30 Plane Installation

[0242] 50 faces

[0243] 100 display control device

[0244] 110 Communication Unit

[0245] 120 storage units

[0246] 130 control unit

[0247] 131 Acquisition Unit

[0248] 132 First Detection Unit

[0249] 133 Second detection unit

[0250] 134 Display control unit

[0251] 140 sensor units

[0252] 150 display units

Claims

1. A display control device, comprising: an acquisition unit, which acquires position and posture information in a three-dimensional space from the first controller; as well as A display control unit displays a virtual coordinate plane in the three-dimensional space based on the position and posture information in the three-dimensional space, on which two-dimensional coordinates can be specified.

2. The display control device according to claim 1, further comprising: A first detection unit detects movement of the virtual coordinate plane in the three-dimensional space based on an input of the first controller.

3. The display control device according to claim 2, wherein: The first detection unit detects movement of an object whose position can be specified using the virtual coordinate plane based on an input of the first controller.

4. The display control device according to claim 2, further comprising: A second detection unit detects a position designated on the virtual coordinate plane based on an input from a second controller.

5. The display control device according to claim 4, wherein: The second detecting unit detects a selected portion of the object based on the position specified on the virtual coordinate plane.

6. The display control device according to claim 5, wherein: The second detection unit detects a selected portion of the object based on an angle of the virtual coordinate plane and a position designated by a user on the virtual coordinate plane.

7. The display control device according to claim 6, wherein: The second detection unit detects a point that intersects the object on an extension line of the normal direction of the virtual coordinate plane with the specified position as the starting point, or a point that intersects the object on an extension line of the line connecting the user's viewpoint position and the specified position, as a selected part of the object.

8. The display control device according to claim 4, wherein: The first controller and the second controller are the same device.

9. The display control device according to claim 8, wherein: The first detection unit detects the same device as the first controller in a case where the same device is located at a predetermined distance from a plane in the real space, and The second detection unit detects the same device as the second controller in a case where the same device is located within a predetermined distance from a plane in the real space.

10. The display control device according to claim 8, wherein: The first detection unit detects the same device as the first controller when a designation to use the same device as the first controller is received from a user of the same device, and The second detection unit detects the same device as the second controller when receiving a designation to use the same device as the second controller from a user of the same device.

11. The display control device according to claim 2, wherein: The display control unit displays the virtual coordinate plane in the three-dimensional space based on the viewpoint position of the user regardless of the position posture information of the first controller in the three-dimensional space.

12. The display control device according to claim 5, wherein: The display control unit displays the virtual coordinate plane in the three-dimensional space of the real space, and When the virtual coordinate plane approaches an arbitrary plane in the real space by a predetermined distance, the first detection unit detects the virtual coordinate plane at a position attracted to the plane so that the virtual coordinate plane is superimposed and displayed on the plane based on an input of the first controller.

13. The display control device according to claim 12, wherein: The display control unit virtually displays the object and a pointer for selecting the object in the three-dimensional space of the real space, and The second detection unit controls a size or a contact point of the pointer so that a user can virtually contact the object using the pointer.

14. The display control device according to claim 12, wherein: The first detection unit controls a position and a size at which the virtual coordinate plane is virtually displayed in a real space based on a user's operation of the first controller.

15. The display control device according to claim 13, wherein: The second detection unit expands a selection position of the pointer and controls an operation of the pointer so that a user can virtually operate the object.

16. The display control device according to claim 5, wherein: The second detection unit performs control based on a position designated by a user on the virtual coordinate plane so as to change a position or a size of the virtual coordinate plane.

17. A display control method executed by a computer, the method comprising: Acquire position and posture information in three-dimensional space from the first controller; as well as A virtual coordinate plane is displayed in the three-dimensional space based on the position and posture information in the three-dimensional space, and two-dimensional coordinates can be specified on the virtual coordinate plane.

Images