Method executed by computer, computer, and program

Abstract

To make it possible to handle a line object in a more intelligible form.SOLUTION: A method is executed by a computer configured to be communicable with a digitizer having a drawing surface for detecting a position of an electronic pen on the drawing surface. With the method, a display surface object, which is a 3D object, is rendered in a virtual reality space. A line object, which is a 3D object generated based on a position of the electronic pen on the drawing surface detected by the digitizer, is rendered in the virtual reality space along the display surface object as a 3D object independent of the display surface object. The line object is configured to be movable from on a display surface of the display surface object.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a rendering method, a rendering device, and a program for rendering a 3D object in a virtual reality (including VR: Virtual Reality, AR: Augmented Reality, and MR: Mixed Reality) space. [Background technology]

[0002] In recent years, there has been an increasing need to design various products while viewing 3D objects in a virtual reality space. Patent Document 1 discloses a technology that enables a user to switch the display method (3D display or 2D display) of 3D objects in a virtual reality space according to their selection. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 102825 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, a new need has emerged for modifying 3D objects displayed in a virtual reality space within the virtual reality space. However, while the technology described in Patent Document 1 can display 3D objects in 3D, it is necessary to display them in 2D to enable 2D input, making it difficult to easily modify 3D objects displayed in a virtual reality space.

[0005] Therefore, one object of the present invention is to provide a computer-implemented method, rendering method, computer, rendering device, and program that allows for easy modification of 3D objects displayed in a virtual reality space. [Means for solving the problem]

[0006] A method according to a first aspect of the present invention is a method executed by a computer configured to communicate with a position detection device having a drawing surface and detecting the position of an electronic pen on the drawing surface, the method comprising: rendering a first object, which is a 3D object, in a virtual reality space; rendering a display surface, which is a 3D object, in the vicinity of the first object; rendering a 3D line, which is a 3D object generated based on the position of the electronic pen on the drawing surface detected by the position detection device, on the display surface; and outputting the first object, which is a 3D object, the display surface, and the 3D line to a display.

[0007] A rendering method according to a second aspect of the present invention is a rendering method executed by a computer configured to communicate with a position detection device that detects the position of an electronic pen on a drawing surface, and that renders 3D objects stored in memory into a virtual reality space, the rendering method including the steps of adding 3D objects that constitute a display surface into the memory, rendering one or more 3D objects stored in the memory into the virtual reality space, converting the pointing position of the electronic pen detected by the position detection device into a position in the virtual reality space coordinate system, and arranging characters or figures drawn on the drawing surface on the display surface based on the pointing position of the electronic pen after conversion by the converting step.

[0008] A computer according to a first aspect of the present invention is a computer configured to communicate with a position detection device having a drawing surface that detects the position of an electronic pen on the drawing surface, and has a control unit that renders a first object, which is a 3D object, in a virtual reality space, renders a display surface, which is a 3D object, in the vicinity of the first object, renders 3D lines, which are 3D objects generated based on the position of the electronic pen on the drawing surface detected by the position detection device, on the display surface, and outputs the first object, which is a 3D object, the display surface, and the 3D lines to a display.

[0009] A rendering device according to a second aspect of the present invention is configured to be able to communicate with a position detection device that detects the position of an electronic pen on a drawing surface, and is a rendering device that renders 3D objects stored in a memory into a virtual reality space, adding 3D objects that constitute a display surface to the memory, rendering one or more 3D objects stored in the memory into the virtual reality space, converting the pointing position of the electronic pen detected by the position detection device to a position in the virtual reality space coordinate system, and placing characters or figures drawn on the drawing surface on the display surface based on the converted pointing position of the electronic pen. is.

[0010] A program according to a first aspect of the present invention is a program for causing a computer configured to communicate with a position detection device having a drawing surface and detecting the position of an electronic pen on the drawing surface to execute processing to render a first object, which is a 3D object, in a virtual reality space, render a display surface, which is a 3D object, in the vicinity of the first object, render 3D lines, which are 3D objects generated based on the position of the electronic pen on the drawing surface detected by the position detection device, on the display surface, and output the first object, which is a 3D object, the display surface, and the 3D lines to a display.

[0011] A program according to a second aspect of the present invention is a program for causing a computer to function as a rendering device that is configured to communicate with a position detection device that detects the position of an electronic pen on a drawing surface and that renders 3D objects stored in memory into a virtual reality space, and causes the computer to execute the following steps: adding 3D objects that constitute a display surface into the memory; rendering one or more 3D objects stored in the memory into the virtual reality space; converting the indication position of the electronic pen detected by the position detection device into a position in the virtual reality space coordinate system; and arranging characters or figures drawn on the drawing surface on the display surface based on the indication position of the electronic pen after conversion by the conversion step. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide a rendering method, a rendering device, and a program that allow a 3D object displayed in a virtual reality space to be easily modified. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a diagram showing a configuration of a 3D object rendering system 1 according to an embodiment of the present invention. [Figure 2] 1A and 1B are diagrams illustrating examples of a drawing surface coordinate system, a display surface coordinate system, and a virtual reality space coordinate system. [Figure 3] 10A and 10B are diagrams illustrating other examples of a drawing surface coordinate system, a display surface coordinate system, and a virtual reality space coordinate system. [Figure 4] 2 is a flowchart showing processing performed by a control unit 2a shown in FIG. [Figure 5] 5 is a flowchart showing details of the location information etc. acquisition process shown in FIG. 4. [Figure 6] 10A and 10B are diagrams illustrating an example of a case where a correction instruction is given using a highly accurate 2D input while a 3D object to be corrected is displayed in 3D. [Figure 7] 10A and 10B are diagrams illustrating an example of a case where a correction instruction is given using a highly accurate 2D input while a 3D object to be corrected is displayed in 3D. [Figure 8] 10A and 10B are diagrams illustrating an example of a case where a correction instruction is given using a highly accurate 2D input while a 3D object to be corrected is displayed in 3D. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0015] 1 is a diagram showing the configuration of a 3D object rendering system 1 according to an embodiment of the present invention. As shown in the figure, the 3D object rendering system 1 according to this embodiment includes a computer 2, a virtual reality display 3, a position detection device 4 having a drawing surface 4a, a pen-shaped electronic pen 5, a glove unit 6 that functions as a glove worn on a user's hand, light emitting devices 7a and 7b, and position sensors 8a to 8d. Position sensors 8a, 8b, 8c, and 8d are attached to or built into the drawing surface 4a, the virtual reality display 3, the electronic pen 5, and the glove unit 6, respectively.

[0016] 1 are arranged in a single room, for example. In the 3D object rendering system 1, the entirety or a part of this room can be used as a virtual reality space.

[0017] The computer 2 includes a control unit 2a and a memory 2b. Each process performed by the computer 2, which will be described later, is realized by the control unit 2a reading and executing a program stored in the memory 2b.

[0018] The computer 2 is connected to each of the virtual reality display 3, the position detection device 4, and the light emitting devices 7a and 7b via a wired communication standard such as USB or LAN, or a wireless communication standard such as wireless LAN or a short-range wireless communication standard. Figure 1 shows an example in which the computer 2 is connected to each of the virtual reality display 3, the position detection device 4, and the light emitting devices 7a and 7b via a wired connection. Note that if the position detection device 4 or the virtual reality display 3 has a built-in computer function, the computer 2 may be configured by that computer.

[0019] The control unit 2a is configured to have the function of displaying a virtual reality space on the virtual reality display 3. More specifically, the control unit 2a is configured to function as a rendering device that sets a virtual reality space based on the positions of the light emitting devices 7a and 7b, renders various 3D objects in the set virtual reality space, and updates the display on the virtual reality display 3 with the rendering results.

[0020] The virtual reality space set by the control unit 2a may be a VR (Virtual Reality) space, an AR (Augmented Reality) space, or an MR (Mixed Reality) space. When a VR space is displayed, a user wearing the virtual reality display 3 perceives virtual reality and is separated from the real world. On the other hand, when an AR space or an MR space is displayed, a user wearing the virtual reality display 3 perceives a space in which virtual reality and the real world are mixed. In the following, the description will be continued assuming that the virtual reality space set by the control unit 2a is a VR space.

[0021] Rendering by the control unit 2a is performed based on one or more 3D objects stored in the memory 2b. The 3D object is information indicating the shape, position, and orientation of the 3D object in a virtual reality space coordinate system that indicates the virtual reality space set by the control unit 2a, and is stored in the memory 2b for each 3D object to be rendered. There are no particular limitations on the specific data format of the 3D object, but it is preferable to use, for example, the VRML format or the X3D format.

[0022] Before performing rendering, the control unit 2a is configured to detect the position and orientation of the position sensor 8b in the virtual reality space coordinate system and acquire viewpoint information indicating the user's viewpoint based on the detected position and orientation of the position sensor 8b. The rendering by the control unit 2a is performed based on the viewpoint information thus acquired.

[0023] When the virtual reality space set by the control unit 2a is a VR space, the 3D objects stored in the memory 2b include 3D objects representing the position detection device 4, the electronic pen 5, and the glove unit 6 shown in FIG. 1. Hereinafter, the 3D object representing the position detection device 4 may be referred to as a "position detection device object," the 3D object representing the electronic pen 5 may be referred to as an "electronic pen object," and the 3D object representing the glove unit 6 may be referred to as a "glove unit object." To render these 3D objects, the control unit 2a first detects the positions and orientations of the position sensors 8a, 8c, and 8d in the virtual reality space coordinate system. The control unit 2a then updates the position detection device object based on the detected position and orientation of the position sensor 8a, the electronic pen object based on the detected position and orientation of the position sensor 8c, and the glove unit object based on the detected position and orientation of the position sensor 8d.

[0024] After updating the position detection device object, electronic pen object, and glove unit object, the control unit 2a performs a process of rendering each updated object in the virtual reality space based on the viewpoint information described above. As a result, the position detection device object, electronic pen object, and glove unit object are displayed at positions in the virtual reality space that correspond to the real positions of the position detection device 4, electronic pen 5, and glove unit 6, respectively.

[0025] The control unit 2a also has a function to cause the electronic pen 5 and the glove unit 6 to function as a 3D controller. Specifically, the control unit 2a first detects the positions and orientations of the position sensors 8c and 8d to detect the positions and orientations of the electronic pen 5 and the glove unit 6. The electronic pen 5 and the glove unit 6 each have one or more operation units operable by a user, and the control unit 2a receives operation information indicating the operation status of the operation units from the electronic pen 5 and the glove unit 6. The operation units are typically switches that can be turned on and off, and the following description will be given assuming that they are switches. The control unit 2a is configured to detect operations performed by the user in the virtual reality space based on the position, orientation, and operation information thus acquired. Hereinafter, the operations thus detected may be referred to as "3D operations" to distinguish them from user operations detected by the position detection device 4 (operations indicated by the pointing position of the electronic pen 5 and transmission data, which will be described later).

[0026] The 3D operations performed by the user in the virtual reality space include operations for creating new 3D objects and operations for updating 3D objects. When the control unit 2a detects these operations, it creates a new 3D object and adds it to the memory 2b, or updates a 3D object stored in the memory 2b, depending on the content of the operation.

[0027] The virtual reality display 3 is a VR display (head-mounted display) that is worn on a person's head. There are various types of virtual reality displays generally available on the market, such as "transparent" or "non-transparent," "glasses-type" or "hat-type," and any of these can be used as the virtual reality display 3.

[0028] The virtual reality display 3 is connected to the position sensor 8a, the electronic pen 5 (including the position sensor 8c), and the glove unit 6 (including the position sensor 8d) by wire or wirelessly. The position sensors 8a, 8c, and 8d are configured to notify the virtual reality display 3 of light reception level information (described later) through this connection. The electronic pen 5 and the glove unit 6 are also configured to notify the virtual reality display 3 of the above-mentioned operation information through this connection. The virtual reality display 3 is configured to notify the control unit 2a of the notified light reception level information and operation information along with the light reception level information of its built-in position sensor 8b. The control unit 2a detects the position and orientation of each of the position sensors 8a to 8d in the virtual reality space coordinate system based on the notified light reception level information, and detects the operation status of each switch provided on the electronic pen 5 and the glove unit 6 based on the notified operation information.

[0029] The position detection device 4 is a device that detects the position of the electronic pen 5 on the drawing surface 4a and has the function of receiving data transmitted from the electronic pen 5. The drawing surface 4a is preferably a flat surface and is made of a material suitable for sliding the pen tip of the electronic pen 5. Typically, the position detection device 4 is a so-called digitizer, and is configured with a touch sensor that detects the position indicated by the electronic pen 5 on the drawing surface 4a and a communication function that notifies the control unit 2a of the detected indicated position. In this case, the drawing surface 4a is configured by the drawing surface of the digitizer. The position detection device 4 may also be a so-called tablet computer, and in this case, the drawing surface 4a is configured by the display surface of the display.

[0030] The position sensor 8a is fixedly installed on the surface of the position detection device 4. Therefore, the position and orientation of the position sensor 8a detected by the control unit 2a indicates the position and orientation of the drawing surface 4a in the virtual reality space coordinate system.

[0031] A two-dimensional drawing surface coordinate system different from the virtual reality space coordinate system is defined on drawing surface 4a. The position indicated by electronic pen 5 detected by the touch sensor of position detection device 4 is a position in this drawing surface coordinate system, not a position in the virtual reality space coordinate system.

[0032] The touch sensor may detect the position of the electronic pen 5 using an electromagnetic induction method or an active electrostatic method. When using the active electrostatic method, the touch sensor is configured to transmit a beacon signal at predetermined time intervals from a sensor electrode disposed within the drawing surface 4a. The beacon signal includes a command for controlling the electronic pen 5 from the touch sensor. The control content by the command includes, for example, transmitting writing pressure data (detected by a capacitance sensor) indicating the pressure applied to the pen tip of the electronic pen 5, transmitting the operation status of various operation units provided on the electronic pen 5, and transmitting a unique ID previously stored in the electronic pen 5. Note that the operation units provided on the electronic pen 5 are also typically switches configured to be able to be turned on and off, and the following description will be continued assuming that they are switches.

[0033] When the active electrostatic electronic pen 5 detects the beacon signal, it transmits a pen signal as a response signal. The pen signal includes a burst signal, which is an unmodulated carrier wave, and a data signal obtained by modulating the carrier wave with data corresponding to the command. The touch sensor attempts to detect the burst signal using the sensor electrodes and detects the position of the electronic pen 5 based on the detection result. The sensor electrodes also detect and demodulate the data signal, thereby receiving data transmitted by the electronic pen 5 in response to the command. The position detection device 4 is configured to transmit the thus acquired position of the electronic pen 5 and the transmitted data of the electronic pen 5 to the control unit 2a each time. The control unit 2a is configured to acquire the trajectory of the electronic pen 5 on the drawing surface 4a based on the series of positions thus notified, and to acquire the notified transmitted data as operation information of the electronic pen 5.

[0034] On the other hand, when using the electromagnetic induction method, the touch sensor is composed of multiple loop coils arranged within the drawing surface 4a. The touch sensor generates a magnetic field by passing a current signal through each loop coil, and detects the position of the electronic pen 5 within the drawing surface 4a by using each loop coil to detect a reflected signal transmitted by the electronic pen 5 that enters the magnetic field. As with the active electrostatic method, the reflected signal includes a portion modulated by the data transmitted by the electronic pen 5 (data indicating writing pressure, the operation status of various switches, a unique ID, etc.). The touch sensor demodulates the data signal detected by one or more loop coils to receive the data transmitted by the electronic pen 5 (data indicating writing pressure, the operation status of various switches, a unique ID, etc.). The subsequent processing by the position detection device 4 and the control unit 2a is the same as in the active electrostatic method.

[0035] Here, the 3D objects created by the computer 2 in response to 3D operations by the user include a 3D object that constitutes a display surface for displaying characters and figures drawn by the electronic pen 5 on the drawing surface 4a of the position detection device 4. Hereinafter, this 3D object may be referred to as a "display surface object." A two-dimensional display surface coordinate system, which is different from both the virtual reality space coordinate system and the drawing surface coordinate system, is defined on the display surface of the display surface object.

[0036] When a new display surface object is created, the control unit 2a acquires first correspondence information that associates the display surface coordinate system of the display surface object with the drawing surface coordinate system based on the respective sizes of the drawing surface 4a and the display surface. The control unit 2a also acquires second correspondence information that associates the display surface coordinate system with the virtual reality space coordinate system based on the position, orientation, and size of the display surface object in the virtual reality space. When the position detection device 4 supplies the control unit 2a with the designated position (position in the drawing surface coordinate system) of the electronic pen 5, the computer 2 first converts the designated position into a position in the display surface coordinate system using the first correspondence information. The control unit 2a then further converts the converted position into a position in the virtual reality space coordinate system using the second correspondence information. The control unit 2a is configured to generate a 3D object representing the character or figure drawn on the drawing surface 4a based on the position in the virtual reality space coordinate system thus acquired. Hereinafter, this 3D object may be referred to as a "2D-derived 3D object."

[0037] From the user's perspective, a 2D-derived 3D object appears as a 2D character or figure drawn on the display surface of a display surface object. However, since it is actually a 3D object independent of the display surface object, it can be operated in 3D separately from the display surface object using the electronic pen 5 and glove unit 6 as a 3D controller.

[0038] 2 is a diagram showing examples of a drawing surface coordinate system, a display surface coordinate system, and a virtual reality space coordinate system. The diagram shows a position detection device object 11, which is a 3D object representing the position detection device 4, an electronic pen object 12, which is a 3D object representing the electronic pen 5, and a display surface object 13, which is an example of a 3D object constituting the display surface, rendered in a virtual reality space 10. As shown in FIG. 2, the display surface object 13 is configured to have a flag-like shape. The surface 11a of the position detection device object 11 corresponds to the drawing surface 4a, and the surface 13a of the display surface object 13 corresponds to the display surface.

[0039] 3 is a diagram showing other examples of the drawing surface coordinate system, the display surface coordinate system, and the virtual reality space coordinate system. In addition to the position detection device object 11 and the electronic pen object 12 also shown in FIG. 2, the figure shows a state in which a display surface object 14, which is another example of a 3D object constituting the display surface, is rendered in the virtual reality space 10. As shown in FIG. 3, the display surface object 14 has a simple rectangular shape (a rectangular parallelepiped with a slight thickness). The surface 14a of the display surface object 14 corresponds to the display surface.

[0040] 2 and 3, the virtual reality space coordinate system is defined by three axes VRX, VRY, and VRZ, the drawing surface coordinate system is defined by two axes TRX and TRY, and the display surface coordinate system is represented by two axes DRX and DRY. When a user moves an electronic pen 5 on the drawing surface 4a, the position detection device 4 detects a series of positions indicating the trajectory and transmits them to the computer 2. Note that when the virtual reality display 3 displays a VR space, the user cannot see the position detection device 4 and the electronic pen 5. However, as described above, a position detection device object 11 is displayed at a position in the virtual reality space corresponding to the real position of the position detection device 4, and an electronic pen object 12 is displayed at a position in the virtual reality space corresponding to the real position of the electronic pen 5.

[0041] The computer 2 converts each position (position in the drawing surface coordinate system) received from the position detection device 4 into a position in the virtual reality space coordinate system using the first and second correspondence information as described above. Then, the computer 2 generates a 3D object based on the converted position and the operation information of the electronic pen 5 received from the position detection device 4, thereby generating a 2D-derived 3D object representing the characters or figures drawn by the user on the drawing surface 4a. The operation information of the electronic pen 5 is used to control, for example, the line width, color, transparency, etc. of the characters or figures. The 3D object 20 representing the character string "ABC" shown in FIG. 2 represents the 2D-derived 3D object thus generated. The user can operate the 3D object 20 using the electronic pen 5 or the glove unit 6 as a 3D controller, independently of the display surface object 13.

[0042] Returning to FIG. 1, the light emitting devices 7a and 7b are signal transmitting devices for position detection used in the 3D object rendering system 1, and are each configured to be able to emit a predetermined signal (laser light) while changing direction under control of the computer 2. The position sensors 8a to 8d are each configured to receive a signal (laser light) emitted by each of the light emitting devices 7a and 7b, and acquire light reception level information including the respective light reception levels. As described above, the acquired light reception level information is notified to the computer 2 from each of the position sensors 8a to 8d, and is used to detect their positions and orientations.

[0043] The above has described an overall overview of the 3D object rendering system 1. Next, the processing performed by the control unit 2a of the computer 2 to input 2D input while the 3D object to be modified is displayed in 3D will be described in detail with reference to a processing flow diagram of the control unit 2a.

[0044] 4 is a flow diagram showing the processing performed by the control unit 2a of the computer 2. As shown in the figure, the control unit 2a first executes a process for obtaining position information, etc. (step S1).

[0045] 5 is a flow diagram showing details of the process of acquiring location information, etc., executed in step S1. Note that the processes of steps S20 to S22 described below are in no particular order, and may be executed in an order different from that shown in FIG.

[0046] 5, the control unit 2a first detects the position and orientation of each of the position sensors 8a to 8d in the virtual reality space coordinate system (step S20) based on the light reception level information of each of the position sensors 8a to 8d notified by the virtual reality display 3. As a result, the control unit 2a acquires the positions and orientations of each of the virtual reality display 3, the drawing surface 4a, the electronic pen 5, and the glove unit 6 in the virtual reality space coordinate system.

[0047] Next, the control unit 2a detects the operation state of each switch provided on one or more 3D controllers (specifically, the electronic pen 5 and the glove unit 6; the same applies below) based on the operation information of each of the 3D controllers notified from the virtual reality display 3 (step S21).

[0048] Furthermore, the control unit 2a acquires electronic pen information indicating the position indicated by the electronic pen 5 in the drawing surface coordinate system and operation information of the electronic pen 5 based on the position of the electronic pen 5 and the transmission data of the electronic pen 5 supplied from the position detection device 4 (step S22). After completing steps S20 to S22, the control unit 2a terminates the process of acquiring position information, etc., and proceeds to step S2 in FIG.

[0049] Returning to Figure 4, after completing the process of acquiring position information and the like, the control unit 2a adds a 3D object to the memory 2b or updates a 3D object existing in the memory 2b based on the position and orientation detected in step S20 and the operation states of each switch detected in step S21 (step S2).

[0050] To explain the process of step S2 more specifically, the control unit 2a performs a process of adding a 3D object to the memory 2b or updating a 3D object existing in the memory 2b based on the detected position and orientation of the 3D controller and the content of the 3D operation indicated by the operation states of each switch provided on the 3D controller. For example, if the content of the 3D operation indicates the addition of a display surface object, the control unit 2a adds a new display surface object to the memory 2b based on the content of the 3D operation. Also, if the content of the 3D operation indicates the update of a 3D object (including a display surface object) already stored in the memory 2b, the control unit 2a updates the 3D object based on the content of the 3D operation.

[0051] In addition, in step S2, the control unit 2a also performs a process of updating the position detection device object 11 existing in memory 2b based on the detected position and orientation of the drawing surface 4a, a process of updating the electronic pen object 12 existing in memory 2b based on the detected position and orientation of the electronic pen 5, and a process of updating the glove unit object existing in memory 2b based on the detected position and orientation of the glove unit 6.

[0052] Next, the control unit 2a updates viewpoint information indicating the user's viewpoint in the virtual reality space based on the detected position and orientation of the virtual reality display 3 (step S3). Specific viewpoint information is configured, for example, by vector information starting from one three-dimensional coordinate, and is held as one of the variables by the control unit 2a.

[0053] Next, the control unit 2a performs rendering of each 3D object in the memory 2b based on the updated viewpoint information (step S4), and updates the output to the virtual reality display 3 based on the result (step S5). This allows the user to view the latest 3D objects in the memory 2b in the virtual reality space.

[0054] Next, the control unit 2a determines whether a new display surface has been used (step S6). The result of this determination is positive if a new display surface object has been added in step S2, and negative otherwise. If the result of step S6 is positive, the control unit 2a sets a display surface coordinate system on the newly used display surface and acquires first correspondence information indicating the correspondence between a pre-stored drawing surface coordinate system and the set display surface coordinate system (step S7). This first correspondence information is used to convert two-dimensional coordinates on the drawing surface 4a into two-dimensional coordinates on the display surface. The control unit 2a also acquires second correspondence information indicating the correspondence between a pre-stored virtual reality space coordinate system and the newly set display surface coordinate system (step S8). This second correspondence information is used to convert two-dimensional coordinates on the display surface into three-dimensional coordinates in the virtual reality space. Finally, the control unit 2a sets a display surface image in use flag to True (step S9) and proceeds to step S10. Here, the display surface image in use flag is a Boolean variable whose initial value is False. If the control unit 2a obtains a negative result in step S6, it moves the process to step S10 without performing steps S7 to S9.

[0055] Next, the control unit 2a determines the value of the display surface image in use flag (step S10). If the control unit 2a determines that the value of the display surface image in use flag is False in step S10, the control unit 2a returns to step S1 and repeats the process. In this case, the addition and update of the 2D-derived 3D object are not executed. On the other hand, if the control unit 2a determines that the value of the display surface image in use flag is True, the control unit 2a performs a process of converting the pointing position in the electronic pen information acquired in step S22 into a position in the virtual reality space coordinate system based on the acquired first and second correspondence information (step S11). Specifically, the control unit 2a first converts the pointing position of the electronic pen 5 provided by the position detection device 4 into a position in the display surface coordinate system using the first correspondence information. Then, the control unit 2a further converts the converted position into a position in the virtual reality space coordinate system using the second correspondence information, thereby converting the pointing position in the electronic pen information into a position in the virtual reality space coordinate system.

[0056] Next, the control unit 2a adds or updates the 2D-derived 3D object in the memory 2b based on the electronic pen information including the converted position (step S12). As described above, the 2D-derived 3D object thus added or updated appears as a two-dimensional character or figure. However, its actual form is a 3D object independent of the display surface, and it has a slight thickness. Therefore, the 2D-derived 3D object can be manipulated in 3D separately from the 3D objects constituting the display surface using the electronic pen 5 and glove unit 6 as a 3D controller. When the user performs this 3D manipulation, the control unit 2a acquires the content of the 3D manipulation in the above-described steps S20 and S21, and updates the 2D-derived 3D object in the memory 2b in step S2. After completing step S12, the control unit 2a returns to step S1 and repeats the process.

[0057] 6 to 8 are diagrams showing an example of a case where 2D input is performed while a 3D object to be corrected is displayed in 3D. In the virtual reality space shown in these figures, a position detection device object 11, an electronic pen object 12, and a display surface object 13, which are also shown in FIG. 2, as well as a display surface object 14 different from the display surface object 13 and a 3D object 15 to be corrected are displayed.

[0058] The two display surface objects 13 and 14 were added to the memory 2b in this order. Therefore, the current first and second correspondence information indicate a correspondence relationship with the display surface coordinate system of the display surface object 14, and when the user draws characters or figures on the drawing surface 4a using the electronic pen 5, the control unit 2a generates a 2D-derived 3D object on the display surface of the display surface object 14. The illustrated 3D object 21 represents the 2D-derived 3D object thus generated.

[0059] In this example, the 3D object 15 to be modified is a 3D object having the shape of a bicycle. In this example, the user viewing the virtual reality space is, for example, the boss of another user (subordinate) who placed the 3D object 15 in the virtual reality space, and wants to instruct the subordinate to modify the portion of the 3D object 15 corresponding to the handlebars 15a. In this case, the user first operates the electronic pen 5 or glove unit 6 as a 3D controller to place the display surface object 13 having a flag-like shape in the portion of the handlebars 15a. Then, by using the electronic pen 5 to write a message (e.g., "Please modify as instructed") to instruct the modification on the drawing surface 4a, a 2D-derived 3D object representing the message is placed on the display surface of the display surface object 13.

[0060] Next, the user operates the electronic pen 5 or glove unit 6 as a 3D controller to place a display surface object 14 having a larger display surface near the 3D object 15. Then, when the user uses the electronic pen 5 to draw a figure indicating the correction content on the drawing surface 4a, a 3D object 21 indicating the figure is placed on the display surface of the display surface object 14, as shown in FIG.

[0061] The 3D object 21 thus placed is a different 3D object from the display surface object 14, and can therefore be operated independently of the display surface object 14. The user then grabs the 3D object 21 with the electronic pen 5 or glove unit 6 as a 3D controller, removes it from the display surface of the display surface object 14 as shown in Fig. 7, and moves it to a position where it overlaps with the handle 15a as shown in Fig. 8. This allows a subordinate user who later views this virtual reality space to reliably understand the content of the correction instructions given by the superior.

[0062] As described above, according to the 3D object rendering system 1 of this embodiment, while the display surface object is displayed in 3D together with other 3D objects, characters and figures can be drawn on the display surface by operating the electronic pen 5 on the drawing surface 4a. Therefore, while the 3D object to be corrected is displayed in 3D, it is possible to input corrections with high precision by using 2D input.

[0063] Furthermore, since the characters and figures written on the display surface are themselves 2D-derived 3D objects separate from the display surface object, the user can move the written characters and figures to any position in the virtual reality space independently of the display surface object, making it possible to instruct modifications to the 3D object in a more user-friendly way.

[0064] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and it goes without saying that the present invention can be embodied in various forms without departing from the spirit of the present invention.

[0065] For example, in the above embodiment, an example has been described in which the present invention is applied to a 3D object rendering system 1 of a type that detects the position and orientation of an object in real space using light emitting devices 7a, 7b and position sensors 8a to 8d. However, the present invention can also be applied to a 3D object rendering system of a type in which cameras are placed in place of each of the light emitting devices 7a, 7b, and markers are placed in place of each of the position sensors 8a to 8d, and the position and orientation of an object in real space is detected by detecting the markers from the imaging results obtained by the cameras. [Explanation of symbols]

[0066] 1. 3D Object Rendering System 2. Computer 2a Control section 2b Memory 3 Virtual Reality Displays 4 Position detection device 4a drawing surface 5 Electronic pen 6 Globe Unit 7a,7b Light emission device 8a~8d Position sensors 10 Virtual Reality Space 11 Location Device Object 11a surface 12 Electronic Pen Objects 13,14 Display Surface Objects 13a,14a surface 15 3D object to be modified 15a handle 20,21 2D-derived 3D objects

Claims

1. 1. A computer-implemented method configured to communicate with a digitizer having a drawing surface to detect a position of an electronic pen on the drawing surface, comprising: Rendering the display surface object, which is a 3D object, in a virtual reality space; a line object, which is a 3D object generated based on the position of the electronic pen on the drawing surface detected by the digitizer, is rendered along the display surface object in the virtual reality space as a 3D object independent of the display surface object; the line object is configured to be movable on the display surface of the display surface object; A computer-implemented method.

2. moving the line object based on the position of the 3D controller detected by a tracking system that tracks a tracker included in the 3D controller; 10. The computer-implemented method of claim 1.

3. Detecting the operation state of an operation unit of the 3D controller; moving the line object based on the detected operation state of the operation unit; 10. The computer-implemented method of claim 1.

4. Detecting the operation state of an operation unit of the 3D controller; When the line object is moved, a color, a thickness, or a transparency of the line object is changed according to the detected operation state of the operation unit.

10. The computer-implemented method of claim 1.

5. moving the line object based on the orientation of the 3D controller; 10. The computer-implemented method of claim 1.

6. moving the line object from the display surface of the display surface object and placing it on top of another 3D object in the virtual reality space; 10. The computer-implemented method of claim 1.

7. The virtual reality space is a VR (Virtual Reality) space, an AR (Augmented Reality) space, or an MR (Mixed Reality) space.

10. The computer-implemented method of claim 1.

8. A computer configured to be able to communicate with a digitizer having a drawing surface and detecting a position of an electronic pen on the drawing surface, A control unit is provided. The control unit Rendering the display surface object, which is a 3D object, in a virtual reality space; a line object, which is a 3D object generated based on the position of the electronic pen on the drawing surface detected by the digitizer, is rendered along the display surface object in the virtual reality space as a 3D object independent of the display surface object; The line object is configured to be movable from the display surface of the display surface object. computer.

9. a computer configured to be able to communicate with a digitizer having a drawing surface and detecting a position of an electronic pen on the drawing surface; Rendering the display surface object, which is a 3D object, in a virtual reality space; a line object, which is a 3D object generated based on the position of the electronic pen on the drawing surface detected by the digitizer, is rendered along the display surface object in the virtual reality space as a 3D object independent of the display surface object; The line object is configured to be movable from the display surface of the display surface object. A program for executing a process.

Images