Method executed by computer, computer and program

Abstract

The present invention is to be able to simply correct a 3D object displayed in a virtual reality space. The solution is a method executed by a computer configured to be able to communicate with a position detection device having a drawing surface and detecting a position of an electronic pen on the drawing surface, wherein the method includes: rendering a first object as a 3D object in a virtual reality space, rendering a display surface as a 3D object in the vicinity of the first object, rendering a 3D line as a 3D object on the display surface, the 3D line being generated based on the position of the electronic pen on the drawing surface detected by the position detection device, and outputting the first object as the 3D object, the display surface, and the 3D line to a display.

Description

Technical Field

[0001] This invention relates to a rendering method, rendering apparatus, and program for rendering 3D objects in virtual reality (including VR: Virtual Reality, AR: Augmented Reality, MR: Mixed Reality) space. Background Technology

[0002] In recent years, there has been a growing demand for designing various products while viewing 3D objects in virtual reality space. Patent document 1 discloses a technology that allows switching the display method (3D display or 2D display) of 3D objects in virtual reality space based on the user's selection.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: International Publication No. 2019 / 102825 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] However, in recent years, a new demand has emerged for correcting 3D objects displayed within a virtual reality space. However, while the technology described in Patent Document 1 can display 3D objects in 3D, it requires setting the display to 2D to make 2D input effective, making it difficult to easily correct 3D objects displayed within a virtual reality space.

[0008] Therefore, one of the objectives of this invention is to provide a computer-executed method, rendering method, computer, rendering apparatus, and program capable of easily modifying 3D objects displayed in virtual reality space.

[0009] Technical solutions for solving the problem

[0010] The first aspect of the invention involves a method executed by a computer configured to communicate with a position detection device having a drawing surface and detecting the position of an electronic pen on the drawing surface. The method includes: rendering a first object as a 3D object in a virtual reality space; rendering a display surface as a 3D object near the first object; rendering 3D lines as 3D objects on the display surface, the 3D lines being generated based on the position of the electronic pen on the drawing surface detected by the position detection device; and outputting the first object as a 3D object, the display surface, and the 3D lines to a display.

[0011] The rendering method of the second aspect of the present invention is executed by a computer configured to communicate with a position detection device that detects the position of an electronic pen on a drawing surface. The rendering method renders 3D objects stored in a memory within a virtual reality space. The rendering method includes: adding 3D objects constituting a display surface to the memory; rendering one or more 3D objects stored in the memory within the virtual reality space; converting the pointer position of the electronic pen detected by the position detection device into a position in the virtual reality space coordinate system; and configuring text or graphics drawn on the drawing surface on the display surface based on the pointer position of the electronic pen converted by the conversion step.

[0012] The computer of the first aspect of the present invention is configured to communicate with a position detection device having a drawing surface and detecting the position of an electronic pen on the drawing surface. The computer has a control unit that renders a first object as a 3D object in a virtual reality space, renders a display surface as a 3D object near the first object, and renders 3D lines as 3D objects on the display surface. The 3D lines are generated based on the position of the electronic pen on the drawing surface detected by the position detection device. The computer outputs the first object as the 3D object, the display surface, and the 3D lines to a display.

[0013] The rendering apparatus of the second aspect of the present invention is configured to communicate with a position detection device that detects the position of an electronic pen on a drawing surface. The rendering apparatus renders 3D objects stored in a memory within a virtual reality space. The rendering apparatus adds 3D objects constituting a display surface to the memory and renders one or more 3D objects stored in the memory within the virtual reality space. It also converts the pointer position of the electronic pen detected by the position detection device into a position in the coordinate system of the virtual reality space and arranges the text or graphics drawn on the drawing surface on the display surface based on the converted pointer position of the electronic pen.

[0014] The first aspect of the present invention provides a program for causing a computer to perform the following processes, the computer being configured to communicate with a position detection device having a drawing surface and detecting the position of an electronic pen on the drawing surface: rendering a first object as a 3D object in a virtual reality space; rendering a display surface as a 3D object near the first object; rendering 3D lines as 3D objects on the display surface, the 3D lines being generated based on the position of the electronic pen on the drawing surface detected by the position detection device; and outputting the first object as a 3D object, the display surface, and the 3D lines to a display.

[0015] A second aspect of the invention provides a program for enabling a computer to function as a rendering device, the rendering device being configured to communicate with a position detection device that detects the position of an electronic pen on a drawing surface, the rendering device rendering 3D objects stored in a memory within a virtual reality space, wherein the program causes the computer to perform the following steps: adding 3D objects constituting the display surface to the memory; rendering one or more 3D objects stored in the memory within the virtual reality space; converting the pointer position of the electronic pen detected by the position detection device into a position in the virtual reality space coordinate system; and arranging text or graphics drawn on the drawing surface on the display surface based on the pointer position of the electronic pen converted by the conversion step.

[0016] Invention Effects

[0017] According to the present invention, a rendering method, rendering apparatus and program are provided that can easily modify 3D objects displayed in virtual reality space. Attached Figure Description

[0018] Figure 1 This is a diagram showing the structure of a 3D object rendering system 1 according to an embodiment of the present invention.

[0019] Figure 2 This is a diagram illustrating an example of a depiction plane coordinate system, a display plane coordinate system, and a virtual reality space coordinate system.

[0020] Figure 3 This is another example of a diagram representing a depiction plane coordinate system, a display plane coordinate system, and a virtual reality space coordinate system.

[0021] Figure 4 It means Figure 1 The flowchart shows the processing performed by the control unit 2a.

[0022] Figure 5 It means Figure 4 The flowchart shows the detailed process of obtaining and processing location information, etc.

[0023] Figure 6 This diagram illustrates an example of using high-precision 2D input to provide correction instructions while displaying a 3D object as the object of correction in 3D.

[0024] Figure 7 This diagram illustrates an example of using high-precision 2D input to provide correction instructions while displaying a 3D object as the object of correction in 3D.

[0025] Figure 8 This diagram illustrates an example of using high-precision 2D input to provide correction instructions while displaying a 3D object as the object of correction in 3D. Detailed Implementation

[0026] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0027] Figure 1 This diagram illustrates the structure of a 3D object rendering system 1 according to an embodiment of the present invention. As shown in the diagram, the 3D object rendering system 1 of this embodiment comprises: a computer 2; a virtual reality display 3; a position detection device 4 having a drawing surface 4a; an electronic pen 5 having a pen-like shape; a glove unit 6 that functions as a glove worn on a user's hand; light emitting devices 7a and 7b; and position sensors 8a to 8d. The position sensors 8a, 8b, 8c, and 8d are respectively mounted or built into the drawing surface 4a, the virtual reality display 3, the electronic pen 5, and the glove unit 6.

[0028] As an example, Figure 1 The devices shown are arranged in a room. In the 3D object rendering system 1, the entire room or a part of it can be used as a virtual reality space.

[0029] The computer 2 is configured to include a control unit 2a and a memory 2b. The various processes performed by the computer 2, as described later, are implemented by reading and executing programs stored in the memory 2b from the control unit 2a.

[0030] Computer 2 is connected to virtual reality display 3, position detection device 4, and light emitting devices 7a and 7b respectively via wired communication standards such as USB and LAN, or wireless communication standards such as wireless LAN and short-range wireless communication standards. Figure 1 The diagram shows an example where computer 2, virtual reality display 3, position detection device 4, and light emitting devices 7a and 7b are connected by wires. Furthermore, if the position detection device 4 and virtual reality display 3 have built-in computer functions, computer 2 can also be constructed using this computer.

[0031] The control unit 2a is configured to display a virtual reality space on the virtual reality display 3. More specifically, the control unit 2a is configured to function as a rendering device that sets up the virtual reality space based on the positions of the light emitting devices 7a and 7b, renders various 3D objects within the set virtual reality space, and updates the display of the virtual reality display 3 based on the rendering results.

[0032] The virtual reality space set by control unit 2a can be a VR (Virtual Reality) space, an AR (Augmented Reality) space, or a MR (Mixed Reality) space. When displaying a VR space, the user wearing the virtual reality display 3 perceives the virtual reality and separates it from the real world. Conversely, when displaying an AR or MR space, the user wearing the virtual reality display 3 perceives a space where virtual reality and the real world are mixed. Hereinafter, we will continue the explanation assuming that the virtual reality space set by control unit 2a is a VR space.

[0033] Rendering performed by the control unit 2a is based on one or more 3D objects stored in the memory 2b. Each 3D object represents information about its shape, position, and orientation within a virtual reality spatial coordinate system representing the virtual reality space set by the control unit 2a, and this information is stored in the memory 2b for each 3D object that is the subject of rendering. The specific data format of the 3D objects is not particularly limited; for example, VRML or X3D formats are preferred.

[0034] The control unit 2a is configured to detect the position and orientation of the position sensor 8b in the virtual reality spatial coordinate system before rendering, and obtain viewpoint information representing the user's viewpoint based on the detected position and orientation of the position sensor 8b. Rendering performed by the control unit 2a is executed based on the viewpoint information obtained in this way.

[0035] When the virtual reality space set by the control unit 2a is a VR space, the 3D objects stored in the memory 2b include those representing... Figure 1The 3D objects representing the position detection device 4, electronic pen 5, and glove unit 6 are shown below. Hereinafter, the 3D object representing the position detection device 4 will be referred to as the "position detection device object," the 3D object representing the electronic pen 5 as the "electronic pen object," and the 3D object representing the glove unit 6 as the "glove unit object." When rendering these 3D objects, the control unit 2a first detects the position and orientation of each of the position sensors 8a, 8c, and 8d in the virtual reality spatial coordinate system. Then, the control unit 2a updates the position detection device object based on the detected position and orientation of the position sensor 8a, updates the electronic pen object based on the detected position and orientation of the position sensor 8c, and updates the glove unit object based on the detected position and orientation of the position sensor 8d.

[0036] The control unit 2a, which has updated the position detection device object, the electronic pen object, and the glove unit object respectively, processes the updated rendering of each object in the virtual reality space based on the aforementioned viewpoint information. Thus, the position detection device object, the electronic pen object, and the glove unit object are displayed at their respective positions in the virtual reality space corresponding to the real positions of the position detection device 4, the electronic pen 5, and the glove unit 6.

[0037] The control unit 2a also functions to enable the electronic pen 5 and glove unit 6 to function as a 3D controller. Specifically, the control unit 2a first detects the position and orientation of the electronic pen 5 and glove unit 6 by detecting the position and orientation of the position sensors 8c and 8d. Furthermore, the electronic pen 5 and glove unit 6 each have one or more operable operation units, and the control unit 2a receives operation information from the electronic pen 5 and glove unit 6 indicating the operation status of these operation units. Moreover, the operation units are typically configured as on / off switches, and will be described as switches in the following description. The control unit 2a is configured to detect user operations within the virtual reality space based on the obtained position, orientation, and operation information. Hereinafter, to distinguish this from user operations detected by the position detection device 4 (operations indicated by the indicated position and transmitted data of the electronic pen 5, as described later), such detected operations are sometimes referred to as "3D operations."

[0038] The 3D operations performed by the user in the virtual reality space include operations for creating new 3D objects or operations for updating 3D objects. The control unit 2a, which detects these operations, creates new 3D objects and adds them to the memory 2b or updates the 3D objects stored in the memory 2b according to the content of the operation.

[0039] Virtual reality display 3 is a VR display (head-mounted display) worn on a person's head. Among the commercially available virtual reality displays, there are various types such as "through-view" or "non-through-view", "glasses" or "hat" types. As a virtual reality display 3, any of these types can be used.

[0040] The virtual reality display 3 is connected via wired or wireless means to position sensor 8a, electronic pen 5 (including position sensor 8c), and glove unit 6 (including position sensor 8d). Position sensors 8a, 8c, and 8d are configured to transmit light level information (described later) to the virtual reality display 3 via this connection. Furthermore, electronic pen 5 and glove unit 6 are configured to transmit the aforementioned operation information to the virtual reality display 3 via this connection. The virtual reality display 3 is configured to transmit the transmitted light level information and operation information, along with the light level information from its built-in position sensor 8b, to the control unit 2a. The control unit 2a detects the position and orientation of each of the position sensors 8a to 8d in the virtual reality spatial coordinate system based on the transmitted light level information, and detects the operation status of each switch installed on electronic pen 5 and glove unit 6 based on the transmitted operation information.

[0041] The position detection device 4 is a device that has the function of detecting the position of the electronic pen 5 on the drawing surface 4a and receiving data sent from the electronic pen 5. The drawing surface 4a is preferably a flat surface made of a material suitable for allowing the tip of the electronic pen 5 to slide. Typically, the position detection device 4 is a so-called digitizer, configured to have a touch sensor that detects the indicated position of the electronic pen 5 within the drawing surface 4a and a communication function that notifies the control unit 2a of the detected indicated position. In this case, the drawing surface 4a is formed by the drawing surface of the digitizer. The position detection device 4 can also be a so-called tablet computer, in which case the drawing surface 4a is formed by the display surface of a monitor.

[0042] The position sensor 8a is fixedly mounted on the surface of the position detection device 4. Therefore, the position and orientation of the position sensor 8a detected by the control unit 2a represent the position and orientation of the drawing surface 4a in the virtual reality spatial coordinate system.

[0043] A two-dimensional drawing surface coordinate system, different from the virtual reality spatial coordinate system, is defined on the drawing surface 4a. The position indicated by the electronic pen 5, detected by the touch sensor of the position detection device 4, is not the position in the virtual reality spatial coordinate system, but the position in the drawing surface coordinate system.

[0044] The position detection of the electronic pen 5 by the touch sensor can be performed using either electromagnetic induction or active electrostatic induction. In the case of active electrostatic induction, the touch sensor is configured to send beacon signals from sensor electrodes disposed within the drawing surface 4a at predetermined time intervals. The beacon signals include instructions for controlling the electronic pen 5 from the touch sensor. The control based on these instructions includes, for example, sending pen pressure data (detected by a capacitive sensor) indicating the pressure applied to the tip of the electronic pen 5, sending the operating status of various operating parts provided on the electronic pen 5, and sending a pre-stored unique ID of the electronic pen 5. Furthermore, the operating parts provided on the electronic pen 5 are typically configured as switches that can be turned on and off; these will be further described below as switches.

[0045] When the aforementioned beacon signal is detected, the electronic pen 5, corresponding to the active electrostatic method, sends out a pen signal as a response signal. The pen signal is a signal comprising a burst signal as an unmodulated carrier wave and a data signal obtained by modulating the carrier wave using data corresponding to the aforementioned command. The touch sensor attempts to detect the burst signal through the aforementioned sensor electrodes and detects the position of the electronic pen 5 based on the detection result. In addition, by detecting and demodulating the data signal through the aforementioned sensor electrodes, the electronic pen 5 receives the data sent by the command. The position detection device 4 is configured to send the position of the electronic pen 5 and the data sent by the electronic pen 5 to the control unit 2a each time. The control unit 2a is configured to obtain the trajectory of the electronic pen 5 on the drawing surface 4a based on the series of positions that are notified, and to obtain the notified data sent as operation information of the electronic pen 5.

[0046] On the other hand, when using electromagnetic induction, the touch sensor consists of multiple loop coils arranged within the drawing surface 4a. The touch sensor generates a magnetic field by flowing a current signal through each loop coil, and detects the position of the electronic pen 5 within the drawing surface 4a by detecting the reflected signal sent by the electronic pen 5 that has entered the loop coil. Furthermore, similar to the active electrostatic method, the reflected signal includes a portion modulated by data sent by the electronic pen 5 (data representing pen pressure data, the operating state of various switches, inherent ID, etc.). The touch sensor receives the data sent by the electronic pen 5 (data representing pen pressure data, the operating state of various switches, inherent ID, etc.) by demodulating the data signal detected by any one or more loop coils. The subsequent processing by the position detection device 4 and the control unit 2a is the same as in the active electrostatic method.

[0047] Here, the 3D objects created by the computer 2 upon receiving 3D operations from the user include 3D objects that constitute a display surface for displaying text and graphics drawn by the electronic pen 5 on the drawing surface 4a of the position detection device 4. Hereinafter, this 3D object will sometimes be referred to as a "display surface object". A two-dimensional display surface coordinate system, different from both the virtual reality space coordinate system and the drawing surface coordinate system, is defined on the display surface of the display surface object.

[0048] When a new display surface object is created, the control unit 2a obtains first correspondence information that associates the display surface coordinate system with the drawing surface coordinate system based on the dimensions of the drawing surface 4a and the display surface itself. Furthermore, the control unit 2a obtains second correspondence information that associates the display surface coordinate system with the virtual reality space coordinate system based on the position, orientation, and size of the display surface object within the virtual reality space. Then, when the control unit 2a receives the indicated position (position in the drawing surface coordinate system) from the position detection device 4 via the electronic pen 5, the computer 2 first uses the first correspondence information to convert it to the position in the display surface coordinate system. Next, the control unit 2a uses the second correspondence information to further convert the converted position to the position in the virtual reality space coordinate system. The control unit 2a is configured to generate a 3D object representing the text or graphics drawn on the drawing surface 4a based on the position obtained in the virtual reality space coordinate system. Hereinafter, this 3D object will sometimes be referred to as a "3D object from 2D".

[0049] From the user's perspective, a 3D object from 2D is two-dimensional text and graphics depicted on the display surface of a display object. However, since its entity is a 3D object independent of the display surface object, 3D operations can be performed separately from the display surface object using the electronic pen 5, which acts as a 3D controller, and the glove unit 6.

[0050] Figure 2 This is a diagram illustrating an example of a rendering plane coordinate system, a display plane coordinate system, and a virtual reality space coordinate system. The diagram shows the state in virtual reality space 10 where a position detection device object 11 is being rendered as a 3D object representing a position detection device 4, an electronic pen object 12 is being rendered as a 3D object representing an electronic pen 5, and a display plane object 13 is being rendered as an example of a 3D object constituting a display plane. Figure 2 As shown, the display surface object 13 is configured with a flag-shaped form. The surface 11a of the position detection device object 11 corresponds to the drawing surface 4a, and the surface 13a of the display surface object 13 corresponds to the display surface.

[0051] in addition, Figure 3 This is another example illustrating the depiction plane coordinate system, the display plane coordinate system, and the virtual reality space coordinate system. In this figure, in addition to... Figure 2In addition to the position detection device object 11 and the electronic pen object 12 shown, the state of another example of a display surface object 14, which is being rendered as a 3D object constituting the display surface, is also being rendered in the virtual reality space 10. For example... Figure 3 As shown, the display surface object 14 is configured with a simple rectangular shape (a cuboid with a very small thickness). The surface 14a of the display surface object 14 corresponds to the display surface.

[0052] like Figure 2 and Figure 3 As shown, the virtual reality space coordinate system is defined by three axes VRX, VRY, and VRZ; the drawing surface coordinate system is defined by two axes TRX and TRY; and the display surface coordinate system is represented by two axes DRX and DRY. When the user moves the electronic pen 5 on the drawing surface 4a, the position detection device 4 detects a series of positions representing the trajectory and sends them to the computer 2. Furthermore, when the virtual reality display 3 displays the VR space, the user cannot see the position detection device 4 and the electronic pen 5. However, as described above, the position detection device object 11 is displayed at the position in the virtual reality space corresponding to the actual position of the position detection device 4, and the electronic pen object 12 is displayed at the position in the virtual reality space corresponding to the actual position of the electronic pen 5.

[0053] As described above, computer 2 uses the first and second corresponding information to convert the positions (positions in the drawing surface coordinate system) received from position detection device 4 into positions in the virtual reality spatial coordinate system. Then, based on the converted positions and the operation information of electronic pen 5 received from position detection device 4, a 3D object is generated, representing the text and graphics drawn by the user on drawing surface 4a, derived from 2D. Furthermore, the operation information of electronic pen 5 is used, for example, to control the line width, color, and transparency of the text and graphics. Figure 2 The 3D object 20 shown, representing the string "ABC", represents a 3D object generated from 2D. The user can independently manipulate the 3D object 20 from the display surface object 13 using the electronic pen 5 or glove unit 6, which acts as a 3D controller.

[0054] return Figure 1 The light emitting devices 7a and 7b are signal transmitting devices used in the 3D object rendering system 1 for position detection. They are configured to emit a predetermined signal (laser) while changing direction according to the control of the computer 2. The position sensors 8a to 8d are each composed of multiple light-receiving sensors, configured such that each light-receiving sensor receives the signal (laser) irradiated by the light emitting devices 7a and 7b respectively, and obtains light-receiving level information including its respective light-receiving level. As described above, the obtained light-receiving level information is transmitted from each position sensor 8a to 8d to the computer 2 for detecting their position and orientation.

[0055] The above provides an overview of the 3D object rendering system 1. Next, referring to the processing flowchart of the control unit 2a, the processing performed by the control unit 2a of the computer 2 for inputting 2D input while displaying the 3D object to be corrected in 3D will be explained in detail.

[0056] Figure 4 This is a flowchart illustrating the processing performed by the control unit 2a of computer 2. As shown in the figure, the control unit 2a first performs processing to obtain position information, etc. (step S1).

[0057] Figure 5 This is a flowchart detailing the process of acquiring location information, etc., performed in step S1. Furthermore, the order of steps S20 to S22 described below may differ and can be used in conjunction with... Figure 5 The descriptions are executed in different orders.

[0058] like Figure 5 As shown, the control unit 2a first detects the position and orientation of each of the position sensors 8a to 8d in the virtual reality spatial coordinate system based on the light level information of each of the position sensors 8a to 8d notified from the virtual reality display 3 (step S20). Thus, the control unit 2a obtains the position and orientation of the virtual reality display 3, the drawing surface 4a, the electronic pen 5, and the glove unit 6 in the virtual reality spatial coordinate system.

[0059] Next, the control unit 2a detects the operating status of each switch set on the 3D controller based on the operation information of one or more 3D controllers (specifically, the electronic pen 5 and the glove unit 6, hereinafter the same) notified from the virtual reality display 3 (step S21).

[0060] Furthermore, the control unit 2a obtains electronic pen information (step S22) based on the position of the electronic pen 5 supplied from the position detection device 4 and the transmission data of the electronic pen 5, which indicates the indicated position of the electronic pen 5 in the drawing surface coordinate system and the operation information of the electronic pen 5. After completing steps S20 to S22, the control unit 2a ends the position information acquisition process and allows the processing to proceed to the next step. Figure 4 Step S2.

[0061] return Figure 4 After completing the acquisition of position information, etc., the control unit 2a adds a 3D object to the memory 2b or updates the 3D object existing in the memory 2b based on the position and direction detected in step S20 and the operation state of each switch detected in step S21 (step S2).

[0062] To explain step S2 in more detail, the control unit 2a, based on the detected position and orientation of the 3D controller and the content of the 3D operation indicated by the operating states of the switches provided on the 3D controller, performs processing to add 3D objects to the memory 2b or update 3D objects existing in the memory 2b. For example, if the content of the 3D operation indicates the addition of a display surface object, the control unit 2a adds the new display surface object to the memory 2b based on the content of the 3D operation. Conversely, if the content of the 3D operation indicates an update of a 3D object (including a display surface object) already stored in the memory 2b, the control unit 2a updates the 3D object based on the content of the 3D operation.

[0063] In addition, in step S2, the control unit 2a also performs the following processing: updating the position detection device object 11 stored in the memory 2b based on the detected position and orientation of the drawing surface 4a; updating the electronic pen object 12 stored in the memory 2b based on the detected position and orientation of the electronic pen 5; and updating the glove unit object stored in the memory 2b based on the detected position and orientation of the glove unit 6.

[0064] Next, the control unit 2a updates the viewpoint information representing the user's viewpoint in the virtual reality space based on the detected position and orientation of the virtual reality display 3 (step S3). The specific viewpoint information is, for example, composed of vector information starting from a three-dimensional coordinate, and is maintained by the control unit 2a as one of the variables.

[0065] Next, the control unit 2a renders each 3D object stored in the memory 2b based on the updated viewpoint information (step S4), and updates the output to the virtual reality display 3 based on the result (step S5). Thus, the user can recognize the latest 3D objects stored in the memory 2b within the virtual reality space.

[0066] Next, the control unit 2a determines whether to start using a new display surface (step S6). This determination is affirmative if a new display surface object was added in step S2, and negative otherwise. Upon receiving an affirmative result in step S6, the control unit 2a sets a display surface coordinate system on the newly used display surface and obtains first correspondence information representing the correspondence between the pre-stored drawing surface coordinate system and the set display surface coordinate system (step S7). This first correspondence information is used to convert the two-dimensional coordinates on the drawing surface 4a into two-dimensional coordinates on the display surface. Additionally, the control unit 2a obtains second correspondence information representing the correspondence between the pre-stored virtual reality space coordinate system and the newly set display surface coordinate system (step S8). This second correspondence information is used to convert the two-dimensional coordinates on the display surface into three-dimensional coordinates within the virtual reality space. Finally, the control unit 2a sets the display surface image usage flag to True (step S9) and transfers the processing to step S10. Here, the display surface image usage flag is a Boolean variable with an initial value of False. The control unit 2a, which receives a negative result in step S6, does not proceed to steps S7 to S9 but transfers the processing to step S10.

[0067] Next, the control unit 2a determines the value of the "Display surface image in use" flag (step S10). If the value of the "Display surface image in use" flag is determined to be False in step S10, the control unit 2a returns to step S1 and repeats the process. In this case, the addition and updating of 3D objects from 2D are not performed. On the other hand, if the value of the "Display surface image in use" flag is determined to be True, the control unit 2a performs the process of converting the indicated position in the electronic pen information obtained in step S22 into a position in the virtual reality spatial coordinate system based on the first and second correspondence information already obtained (step S11). Specifically, the control unit 2a first uses the first correspondence information to convert the indicated position of the electronic pen 5 supplied from the position detection device 4 into a position in the display surface coordinate system. Then, it further converts the converted position into a position in the virtual reality spatial coordinate system by using the second correspondence information, thus converting the indicated position in the electronic pen information into a position in the virtual reality spatial coordinate system.

[0068] Next, based on the electronic pen information including the converted position, the control unit 2a adds a 3D object from 2D to the memory 2b, or updates the 3D object from 2D in the memory 2b (step S12). As described above, the appearance of the added or updated 3D object from 2D is two-dimensional text or graphics. However, its entity is a 3D object independent of the display surface, and although small, it still has thickness. Therefore, the 3D object from 2D can be operated separately from the 3D object constituting the display surface by the electronic pen 5 and glove unit 6, which act as 3D controllers. When the user performs this 3D operation, the control unit 2a obtains the content of the 3D operation in the above-mentioned steps S20 and S21, and updates the 3D object from 2D in the memory 2b in step S2. The control unit 2a, having completed step S12, returns to step S1 and repeats the process.

[0069] Figures 6-8 This diagram illustrates an example of 2D input occurring while a 3D object being modified is being displayed in 3D. In the virtual reality space depicted in these diagrams, and... Figure 2 The position detection device object 11, the electronic pen object 12, and the display surface object 13 shown together display a display surface object 14 that is different from the display surface object 13, and a 3D object 15 that is the object to be corrected.

[0070] Two display surface objects 13 and 14 are sequentially added to the memory 2b. Therefore, the current first and second correspondence information represents the correspondence between the display surface coordinate system of the display surface object 14 and the object itself. When the user uses the electronic pen 5 to draw text or graphics on the drawing surface 4a, the control unit 2a generates a 3D object from 2D on the display surface of the display surface object 14. The illustrated 3D object 21 represents this 3D object generated from 2D.

[0071] In this example, the 3D object 15 that is the object of correction is a 3D object shaped like a bicycle. In this example, consider a user viewing a virtual reality space, for example, a superior of another user (subordinate) whose 3D object 15 is positioned within the virtual reality space, who wants to instruct their subordinate on the correction of a portion of the handlebars 15a corresponding to the 3D object 15. In this case, the user first sets a display surface object 13 with a flag shape on the handlebars 15a by operating an electronic pen 5 or a glove unit 6, which acts as a 3D controller. Then, when the electronic pen 5 is used to write a statement indicating the correction (e.g., "Please correct as instructed") on the drawing surface 4a, a 3D object representing that statement from 2D is positioned on the display surface of the display surface object 13.

[0072] Next, the user, by operating the electronic pen 5 (acting as a 3D controller) or the glove unit 6, positions the display surface object 14, which has a larger display area, near the 3D object 15. Then, when using the electronic pen 5 to draw a graphic representing the correction content within the drawing surface 4a, as... Figure 6 As shown, the 3D object 21 representing the graphic is positioned on the display surface of the display surface object 14.

[0073] The 3D object 21 configured in this way is a different 3D object from the display surface object 14, and therefore can be manipulated independently of the display surface object 14. Thus, the user can grasp the 3D object 21 using the electronic pen 5, which acts as a 3D controller, or the glove unit 6, as... Figure 7 As shown, it is removed from the display surface of the display surface object 14, as indicated. Figure 8 Move it to the position overlapping with the handlebars 15a as shown. This allows the user, who will later view the virtual reality space, to reliably grasp the corrective instructions given by their superior.

[0074] As described above, according to the 3D object rendering system 1 of this embodiment, when the display surface object is displayed in 3D along with other 3D objects, text and graphics can be drawn on the display surface by operating the electronic pen 5 on the drawing surface 4a. Therefore, when a 3D object that is to be corrected is displayed in 3D, high-precision correction content can be input by using 2D input.

[0075] Furthermore, since the text and graphics written onto the display surface are themselves 3D objects derived from 2D, distinct from the display surface objects, users can move the written text and graphics to any location within the virtual reality space, independent of the display surface objects. Therefore, corrections to 3D objects can be indicated in a more easily understandable manner.

[0076] The preferred embodiments of the present invention have been described above, but the present invention is not limited to such embodiments. The present invention can of course be implemented in various ways without departing from its spirit.

[0077] For example, in the above embodiment, an example of applying the present invention to a 3D object rendering system 1 of the type that uses light emitting devices 7a and 7b and position sensors 8a to 8d to detect the position and orientation of objects located in real space has been described. However, the present invention can also be applied to 3D object rendering systems of the type in which a camera is provided instead of each of the light emitting devices 7a and 7b, and a marker is provided instead of each of the position sensors 8a to 8d, and the position and orientation of objects located in real space are detected by detecting the markers based on the shooting results obtained by the camera.

[0078] Label Explanation

[0079] 1 3D Object Rendering System

[0080] 2 Computers

[0081] 2a Control Department

[0082] 2b memory

[0083] 3 Virtual Reality Displays

[0084] 4. Position detection device

[0085] 4a Depicting Surface

[0086] 5 electronic pens

[0087] 6 Glove Units

[0088] 7a, 7b Light emission devices

[0089] 8a~8d position sensors

[0090] 10 Virtual Reality Space

[0091] 11. Position detection device for objects

[0092] 11a Surface

[0093] 12 Electronic pen objects

[0094] 13, 14 Display surface objects

[0095] Surfaces of 13a and 14a

[0096] 15 3D objects used for correction

[0097] 15a handlebars

[0098] 20, 21 3D objects from 2D

Claims

1. A method performed by a computer configured to communicate with a position detection device having a drawing surface and detecting the position of an electronic pen on the drawing surface, wherein, The method includes: Rendering the first object as a 3D object within virtual reality space. Render a surface object that serves as a 3D object in the vicinity of the first object. Along the display surface object, a line object, rendered as a 3D object within the virtual reality space, is a 3D object independent of the display surface object. This line object is generated based on the position of the electronic pen on the drawing surface detected by the position detection device. The line object is configured to move off the display surface of the display surface object. The first object, the display surface object, and the line object are output to the display.

2. The method according to claim 1, wherein, The line object is moved according to the operation of the 3D controller.

3. The method according to claim 2, wherein, The 3D controller is the electronic pen.

4. The method according to claim 3, wherein, The position and orientation of the electronic pen are also detected based on light level information from a position sensor mounted on the electronic pen. Based on the detected position and direction of the electronic pen, the line object is moved.

5. The method according to claim 3 or 4, wherein, The operating status of the operating part of the electronic pen is detected. Based on the detected operating state of the operating unit, the linear object is moved.

6. The method according to any one of claims 1 to 4, wherein, Obtain first correspondence information, which represents the correspondence between a first coordinate system defined on the drawing surface and a second coordinate system defined on the display surface. Obtain second correspondence information, which represents the correspondence between the second coordinate system and the third coordinate system representing the virtual reality space. Based on the first and second correspondence information, the position of the electronic pen is converted into a position in the third coordinate system.

7. The method according to any one of claims 1 to 4, wherein, The method further includes: rendering a second object as a 3D object based on the position and orientation of the depicted surface in a third coordinate system representing the virtual reality space, the second object representing the position detection device.

8. The method according to any one of claims 1 to 4, wherein, Based on the position and orientation of the virtual reality display in the third coordinate system representing the virtual reality space, the viewpoint information representing the user's viewpoint within the virtual reality space is updated. Rendering is performed based on the viewpoint information.

9. A computer configured to communicate with a position detection device having a drawing surface and detecting the position of an electronic pen on the drawing surface, wherein... The computer has a control unit. The control unit renders the first object as a 3D object within the virtual reality space. The control unit renders a display surface object as a 3D object near the first object. The control unit renders a line object as a 3D object within the virtual reality space, along the display surface object, as a 3D object independent of the display surface object. This line object is generated based on the position of the electronic pen on the drawing surface detected by the position detection device. The line object is configured to move off the display surface of the display surface object. The control unit outputs the first object, the display surface object, and the line object to the display.

10. The computer according to claim 9, wherein, The computer moves the line object according to the operation of the 3D controller.

11. The computer according to claim 10, wherein, The 3D controller is the electronic pen.

12. A computer program product comprising a program for causing a computer to perform the following processes, the computer being configured to communicate with a position detection device having a drawing surface and detecting the position of an electronic pen on the drawing surface: Render the first object as a 3D object within virtual reality space; Render a surface object that is a 3D object in the vicinity of the first object; Along the display surface object, a line object, rendered as a 3D object in the virtual reality space, is a 3D object independent of the display surface object. This line object is generated based on the position of the electronic pen on the drawing surface detected by the position detection device. The line object is configured to move from the display surface of the display surface object; and The first object, the display surface object, and the line object are output to the display.

13. The computer program product according to claim 12, wherein, The program is used to cause the computer to perform the following process: moving the line object according to the operation of the 3D controller.

14. The computer program product according to claim 13, wherein, The 3D controller is the electronic pen.

Images