Computer, method and program

Abstract

Without installing a position sensor on the tablet terminal, it is possible to transform the results of pen input on the tablet surface, i.e., tablet input data, into a 3D object in XR space. [Solution] In a computer with a processor, the processor obtains first position and posture information representing the position and posture of the stylus in a first coordinate system with the representative position of the device with the digitizer as the origin, obtains second position and posture information representing the position and posture of the stylus in a second coordinate system defining XR space, and calculates third position and posture information representing the position and posture of the device with the digitizer in the second coordinate system based on the first and second position and posture information.

Description

Technical Field

[0001] This invention relates to computers, methods, and programs, and particularly to computers, methods, and programs for transforming tablet input data obtained as a result of pen input to a tablet terminal into 3D objects in XR (eXtended Reality) space. Background Technology

[0002] It is known that there are technologies that can utilize drawings and text (hereinafter collectively referred to as "tablet input data") input to a tablet terminal via pen input as 3D objects configured in XR space. Such technology is disclosed in Patent Document 1.

[0003] Furthermore, in Patent Documents 2 and 3, regarding the electromagnetic induction (EMR) method as a pen input method, techniques are disclosed for detecting the pen's torsion angle (rotation around the pen axis) and tilt angle (tilt of the pen relative to the flat surface). In Patent Document 4, regarding another active electrostatic method as a pen input method, techniques are disclosed for detecting the pen's torsion angle and tilt angle.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: International Publication No. 2019 / 102825

[0007] Patent Document 2: Japanese Patent No. 3015278

[0008] Patent Document 3: Japanese Patent No. 5358834

[0009] Patent Document 4: Japanese Patent No. 6304814 Summary of the Invention

[0010] The problem that the invention aims to solve

[0011] In the technology described in the aforementioned Patent Document 1, a tracker is installed on the pen and tablet terminal in order to use a tracking system to detect the position and posture of the pen and tablet terminal.

[0012] However, in the technology described in the aforementioned patent document 1, trackers need to be installed on both the pen and the tablet terminal.

[0013] Therefore, one of the objectives of this invention is to provide a computer, method, and program that can transform the results of pen input on the tablet surface, i.e., tablet input data, into 3D objects in XR space without setting a tracker on the tablet terminal.

[0014] Methods for solving problems

[0015] The computer of the present invention is a computer with a processor, wherein the processor acquires first position and posture information representing the position and posture of a stylus in a first coordinate system with the representative position of the device with the digitizer as the origin, acquires second position and posture information representing the position and posture of the stylus in a second coordinate system defining an XR space, and calculates third position and posture information representing the position and posture of the device with the digitizer in the second coordinate system based on the first position and posture information and the second position and posture information.

[0016] The method of the present invention is a method executed by a computer having a processor, comprising: the processor acquiring first position and posture information representing the position and posture of a stylus in a first coordinate system with a representative position of a device having a digitizer as the origin; the processor acquiring second position and posture information representing the position and posture of the stylus in a second coordinate system defining an XR space; and the processor calculating third position and posture information representing the position and posture of the device having a digitizer in the second coordinate system based on the first position and posture information and the second position and posture information.

[0017] The program of the present invention is a program for causing a computer to perform the following steps: obtaining first position and posture information representing the position and posture of a stylus in a first coordinate system with the representative position of the device having a digitizer as the origin; obtaining second position and posture information representing the position and posture of the stylus in a second coordinate system defining an XR space; and calculating third position and posture information representing the position and posture of the device having a digitizer in the second coordinate system based on the first position and posture information and the second position and posture information.

[0018] Invention Effects

[0019] According to the present invention, without the need to install a tracker on the tablet terminal, the result of pen input on the tablet surface, i.e., tablet input data, can be transformed into a 3D object in XR space. Attached Figure Description

[0020] Figure 1 This is a diagram illustrating an XR system 1 according to a first embodiment of the present invention.

[0021] Figure 2 (a) is a diagram representing the plane coordinate system set on the plane surface 5a, and (b) is a diagram representing the world coordinate system set on the plane surface 5a.

[0022] Figure 3 (a) and (b) are diagrams illustrating the tilt information detected by the tablet terminal 5, and (c) is a diagram illustrating the torsion information detected by the tablet terminal 5.

[0023] Figure 4 This is a flowchart of the process executed by the arithmetic unit 2a. Detailed Implementation

[0024] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0025] Figure 1 This diagram illustrates an XR system 1 according to a first embodiment of the present invention. As shown in the diagram, the XR system 1 of this embodiment is configured to include a computer 2, a virtual reality display 3, multiple cameras 4, a tablet terminal 5, and a pen 6. The computer 2 is configured to include an arithmetic unit 2a and an XR tracking system 2b as functional units. Furthermore, the computer 2 can be a single computer or a composite computer that functions as a single computer by combining multiple computers.

[0026] Computer 2 is a device configured to include a processor, memory, and communication devices. The processor executes programs stored in the memory to perform tasks including... Figure 1 The various functions of the computer 2, including the arithmetic unit 2a and the XR tracking system 2b, are illustrated. The communication device is configured to communicate with the virtual reality display 3, multiple cameras 4, and tablet terminal 5 via wired or wireless means, under the control of the processor.

[0027] The arithmetic unit 2a is a functional unit that has the following functions: setting an XR space based on the positions of multiple cameras 4, generating an image representing the set XR space, and providing it to the virtual reality display 3. Figure 1 The x2, y2, and z2 axes shown represent the virtual reality spatial coordinate system of the XR space defined by the processor 2a. The positions and poses of various objects rendered by the processor 2a within the XR space can be represented by translation vectors and rotation matrices of this virtual reality spatial coordinate system. Details of the translation vectors and rotation matrices will be described later.

[0028] The virtual reality display 3 is an XR display (head-mounted display) worn on a person's head. Commercially available virtual reality displays include various types such as "through-view" or "non-through-view," "glasses" or "hat-style," but the virtual reality display 3 can use any of these. When the XR space set by the processor 2a is a VR (Virtual Reality) space, the user wearing the virtual reality display 3 perceives virtual reality as separate from the real world. On the other hand, when the XR space set by the processor 2a is an AR (Augmented Reality) space or a MR (Mixed Reality) space, the user wearing the virtual reality display 3 perceives a space where virtual reality and the real world are mixed.

[0029] The processor 2a performs the rendering of various 3D objects and their placement within the aforementioned image. The 3D objects that become the rendered objects can include, for example... Figure 1 The tablet terminal 5 shown contains both real-world 3D objects and non-real-world 3D objects. As a result of this processing by the arithmetic unit 2a, the user wearing the virtual reality display 3 can visually confirm the 3D objects in XR space.

[0030] The rendering of the processor 2a is performed based on the 3D object information stored in memory. The 3D object information represents the shape, position, and pose of the 3D objects in the XR space set by the processor 2a, and is stored in memory for each rendered object.

[0031] When generating an image representing XR space, the processor 2a first obtains the position and pose of the virtual reality display 3. Specifically, it receives the position and pose of the virtual reality display 3 from the XR tracking system 2b, described later. Based on the obtained position and pose of the virtual reality display 3, the processor 2a determines the user's viewpoint, and based on the determined viewpoint, renders 3D objects and generates an image representing XR space. Thus, a user observing XR space through the virtual reality display 3 can observe various 3D objects from the same position as in reality.

[0032] The XR tracking system 2b is a functional unit that detects objects (real-world objects) represented by images captured by multiple cameras 4 and tracks their position and posture. The multiple cameras 4 are configured to capture images from various angles at different locations within the real-world space corresponding to the XR space set by the arithmetic unit 2a. Figure 1 The image shows 3 cameras 4, but the number of cameras 4 is not limited to 3.

[0033] As an example, the XR tracking system 2b detects objects through either a tracker consisting of multiple markers attached to the object's surface (any type of marker capable of optical detection is acceptable) or through image recognition of the object. In the former case, the XR tracking system 2b detects the position and pose of the pen 6 in the virtual reality spatial coordinate system based on the positions of the multiple markers contained in the images of the pen 6 captured by each camera 4. The tracking results of the XR tracking system 2b are sequentially stored in the memory of the arithmetic unit 2a. The arithmetic unit 2a renders the actual 3D object based on the tracking results stored in the memory.

[0034] The tablet terminal 5 is a device equipped with a digitizer for accepting pen input, and is configured to have a flat tablet surface 5a as the input surface. The tablet terminal 5 may or may not have the function of displaying images on the tablet surface 5a (so-called "liquid crystal digitizer"), or it may not have such a function (so-called "digitizer"). The tablet terminal 5 equipped with the function of displaying images on the tablet surface 5a is configured to display various data including tablet input data as a result of pen input on the tablet surface 5a.

[0035] Figure 2 (a) is a diagram showing the coordinate system of the flat plate set on the flat plate surface 5a. Figure 2 (b) is a diagram representing the world coordinate system set on the flat plate surface 5a. The flat plate coordinate system is... Figure 2 The spatial coordinate system represented by the x3, y3, and z3 axes shown in (a) is used to set one of the four corners of the flat plate 5a as the origin via the flat plate terminal 5. The x3 axis is a coordinate axis set along the long side of the flat plate 5a, the y3 axis is a coordinate axis orthogonal to the x3 axis within the flat plate 5a, and the z3 axis is a coordinate axis orthogonal to both the x3 and y3 axes.

[0036] On the other hand, the world coordinate system is composed of Figure 2 The spatial coordinate system represented by the x1, y1, and z1 axes shown in (b) is used by the arithmetic unit 2a to set the representative position P of the tablet terminal 5 in the XR tracking system 2b (i.e., the reference position of the tablet terminal 5 as a 3D object, usually the center position of the tablet surface 5a) as the origin. The x1 axis is a coordinate axis set along the long side of the tablet surface 5a, the z1 axis is a coordinate axis orthogonal to the x1 axis within the tablet surface 5a, and the y1 axis is a coordinate axis orthogonal to both the x1 and z1 axes. The arithmetic unit 2a pre-stores the relationship between the tablet coordinate system and the world coordinate system (i.e., transformation rules used to transform the position and pose represented by the tablet coordinate system to the position and pose represented by the world coordinate system).

[0037] return Figure 1The pen 6 is a position indicator (stylus) with a pen-like shape. Users input data into the tablet terminal 5 by sliding the tip of the pen 6 across the tablet surface 5a. The specific method of pen input is not particularly limited; for example, electromagnetic induction (EMR) or active electrostatic induction may be preferred. Alternatively, the tablet terminal 5 may also support finger input (touch input), which may be implemented using capacitive electrostatic induction, for example.

[0038] The tablet terminal 5 is configured to sequentially detect the position and posture of the pen 6 in a tablet coordinate system. The position of the pen 6 detected by the tablet terminal 5 is represented by coordinate information indicating the position of the pen 6 in the aforementioned tablet coordinate system. This coordinate information includes planar coordinates (X, Y, X) representing the position within the tablet surface 5a. tab Y tab ) and height information Z representing the distance from the flat surface 5a to the pen 6. tab Furthermore, the posture of the pen 6 detected by the tablet terminal 5 is represented by tilt information indicating the tilt of the pen 6 relative to the tablet surface 5a and torsional information indicating the amount of rotation of the pen 6 about the pen axis. In the following description, this information detected by the tablet terminal 5 will sometimes be collectively referred to as the "0th position posture information" that represents the position and posture of the pen 6 in the tablet coordinate system.

[0039] Figure 3 (a) and (b) are diagrams illustrating the tilt information detected by the tablet terminal 5. Figure 3 (a) shows the tilt angle XTILT representing the tilt of the pen axis toward the x3 axis. tab , Figure 3 (b) shows the tilt angle YTILT, which represents the tilt of the pen axis toward the y3 axis. tab The tilt information is composed of these two tilt angles (XTILTtab, YTILTtab). Additionally, the tilt angle (XTILT...) tab YTILT tab The specific value can be any value that can be converted into a physical angle value (e.g., a value based on degrees or radians).

[0040] Figure 3 Figure (c) is a diagram illustrating the torsion information detected by the tablet terminal 5. In this figure, the torsion angle TWIST, representing the amount of rotation of the pen 6 about its axis, is shown. tab The torsion information is provided by the torsion angle TWIST. tab Composition. Twist angle. tab The specific value can be any value that can be converted into a physical angle value (e.g., a value based on degrees or radians).

[0041] Here, taking the electromagnetic induction (EMR) method as an example, we will briefly explain how the tablet terminal 5 acquires the posture information at position 0. Additionally, the tilt angle (XTILT) mentioned in the following explanation... tab YTILT tab The detection technology is based on patent document 2, Twist angle. tab The detection technology is based on patent document 3; therefore, for more detailed technical information, please refer to these documents.

[0042] The tablet terminal 5, corresponding to the electromagnetic induction (EMR) method, is configured with multiple loop coils extending along the x3 axis and multiple loop coils extending along the y3 axis. The pen 6, also corresponding to the EMR method, is configured with: a cylindrical core formed by dividing a plane along the cylindrical axis into a first core and a second core; first and second LC resonant circuits composed of coils and capacitors connected in series; and a switching circuit for the first and second LC resonant circuits. The coil of the second LC resonant circuit is wound around the second core, and the coil of the first LC resonant circuit is wound around the coil of the second LC resonant circuit onto the cylindrical core.

[0043] The flat panel terminal 5 intermittently outputs an alternating magnetic field from the flat panel surface 5a by supplying alternating current to any one of the loop coils. The switching circuit of the pen 6 is configured to activate the first LC resonant circuit when the duration of the alternating magnetic field is less than a predetermined value, and to activate the second LC resonant circuit when the duration is greater than or equal to the predetermined value.

[0044] At the position (X) of pen 6 within the flat surface 5a. tab Y tab ), height information of pen 6 Z tab and the tilt angle of pen 6 (XTILT) tab YTILT tab In the case of detection, the tablet terminal 5 sends out an alternating magnetic field for a time duration less than a predetermined value. Therefore, the pen 6 becomes active in the first LC resonant circuit. In this state, when the coil of the first LC resonant circuit enters the alternating magnetic field, the capacitor of the first LC resonant circuit is charged. When the sending of the alternating magnetic field by the tablet terminal 5 ends, the alternating magnetic field as a reflected signal is sent out from the coil of the first LC resonant circuit using the power stored in the capacitor. The tablet terminal 5 attempts to detect this alternating magnetic field through the aforementioned loop coils, and based on the distribution of the detected alternating magnetic field intensity, detects the position (X) of the pen 6 within the tablet surface 5a. tab Y tab ).

[0045] In addition, the tablet terminal 5 detects the height information Z of the pen 6 based on the maximum intensity of the detected alternating magnetic field.tab In one example, the tablet terminal 5 pre-stores data with the maximum intensity of the alternating magnetic field as a variable and height information Z as a variable. tab This is a function of the output value. Then, the output value obtained by inputting the maximum intensity of the detected alternating magnetic field into this function is taken as the height information Z of pen 6. tab .

[0046] The intensity distribution of the alternating magnetic field detected by the tablet terminal 5 has secondary peaks on both sides of the main peak. Based on the intensity of these secondary peaks, the tablet terminal 5 detects the tilt angle (XTILT) of the pen 6. tab YTILT tab ).

[0047] Twist of pen 6 tab In the case of detection, the tablet terminal 5 sends out an alternating magnetic field for a time duration exceeding the aforementioned predetermined value. Therefore, the pen 6 becomes a state where the second LC resonant circuit is activated. In this state, when the coil of the second LC resonant circuit enters the alternating magnetic field, the capacitor of the second LC resonant circuit is charged. When the sending of the alternating magnetic field by the tablet terminal 5 ends, the alternating magnetic field as a reflected signal is sent out from the coil of the second LC resonant circuit using the power stored in the capacitor. The tablet terminal 5 attempts to detect this alternating magnetic field through each of the aforementioned loop coils, and detects the position of the second LC resonant circuit within the tablet surface 5a based on the distribution of the detected alternating magnetic field intensity. Then, based on the detected position of the second LC resonant circuit and the detected position of the pen 6 (X... tab Y tab (= Position of the first LC resonant circuit), Twist angle of detection pen 6 tab .

[0048] Furthermore, the method for obtaining the zero-position attitude information in the electromagnetic induction (EMR) method described here is one example; other methods can certainly be used. For instance, a gyroscope, accelerometer, and magnetometer can be incorporated into the pen 6, which then uses them to detect the tilt angle (XTILT). tab YTILT tab and twist angle TWIST tab The tablet terminal 5 receives the tilt angle (XTILT) from the pen 6. tab YTILT tab and twist angle TWIST tab The data is used to detect the tilt angle (XTILT) of the pen 6. tab YTILT tab and twist angle TWIST tabIn this case, it is not necessary to include the second LC resonant circuit and the switching circuit in the pen 6. Data transmission from the pen 6 can be achieved by connecting a variable capacitance capacitor in parallel with the capacitor constituting the first LC resonant circuit, and controlling the capacitance of the variable capacitance capacitor according to the content of the data being transmitted. In this case, the tablet terminal 5 obtains the data transmitted by the pen 6 by demodulating the detected alternating magnetic field. The data transmitted by the pen 6 may also include pen pressure values ​​indicating the pressure applied to the pen tip, on / off information indicating the on / off state of a switch located on the pen 6's housing, and a pen ID serving as identification information for the pen 6.

[0049] The above are examples of tablet terminal 5 and pen 6 corresponding to the electromagnetic induction (EMR) method. However, the same applies to tablet terminal 5 and pen 6 corresponding to the active electrostatic method, except for some aspects. Below, we will explain the cases of tablet terminal 5 and pen 6 corresponding to the active electrostatic method, focusing on the differences from the electromagnetic induction (EMR) method. Additionally, the tilt angle (XTILT) mentioned below... tab YTILT tab and twist angle TWIST tab The detection technology is based on Patent Document 4, therefore, for more detailed technical information, please refer to Patent Document 4.

[0050] First, the tablet terminal 5, corresponding to the active electrostatic method, is configured to have multiple linear electrodes extending along the x3 axis and multiple linear electrodes extending along the y3 axis, replacing the aforementioned annular coil. Furthermore, the pen 6, corresponding to the active electrostatic method, is configured to replace the aforementioned core, first and second LC resonant circuits, and switching circuit, and instead has a pen tip electrode disposed at the pen tip, an annular peripheral electrode disposed at the rear in the pen axis direction when viewed from the pen tip electrode, a processing circuit connected to the pen tip electrode, and a battery supplying power to the processing circuit. The annular peripheral electrode is divided into two parts along the plane of the pen axis: a first peripheral electrode and a second peripheral electrode.

[0051] The tablet terminal 5 transmits an uplink signal from the tablet surface 5a by supplying a signal to any one of the linear electrodes. The uplink signal is a signal modulated by a command. When the processing circuit of the pen 6 receives the uplink signal via the pen tip electrode, it generates a downlink signal corresponding to the command and transmits it from any one or more of the pen tip electrode, the first peripheral electrode, and the second peripheral electrode.

[0052] At the position (X) of pen 6 within the flat surface 5a. tab Y tab ) and the height information Z of pen 6 tabIn the case of detection, the tablet terminal 5 uses an uplink signal to control the pen 6 by sending a downlink signal from the pen tip electrode. The tablet terminal 5 attempts to detect the downlink signal through the aforementioned linear electrodes, and detects the position (X) of the pen 6 within the tablet surface 5a based on the distribution of the intensity of the detected downlink signal. tab Y tab ).

[0053] In addition, the tablet terminal 5 detects the height information Z of the pen 6 based on the maximum strength of the detected downlink signal. tab In one example, tablet terminal 5 pre-stores the maximum strength of the downlink signal as a variable, along with height information Z. tab The function is used to output the value. Then, the output value obtained by inputting the maximum strength of the detected downlink signal into this function is taken as the height information Z of pen 6. tab .

[0054] When performing tilt angle (XTILT) tab YTILT tab In the case of detection of uplink signals, the tablet terminal 5 controls the pen 6 to send downlink signals from the first peripheral electrode and the second peripheral electrode, respectively. The tablet terminal 5 attempts to detect the downlink signal through the aforementioned linear electrodes, and detects the position of the peripheral electrodes within the tablet surface 5a based on the intensity distribution of the detected downlink signal. Then, based on the detected positions of the peripheral electrodes and the detected position of the pen 6 (X... tab Y tab (=position of the pen tip electrode), detect the tilt angle of pen 6 (XTILT) tab YTILT tab ).

[0055] Twist angle tab In the event of detection, the tablet terminal 5 controls the pen 6 to transmit a downlink signal using an uplink signal from the first peripheral electrode, and then a downlink signal from the second peripheral electrode. The tablet terminal 5 attempts to detect the downlink signal each time through the aforementioned linear electrodes, and based on the intensity distribution of the detected downlink signal, detects the positions of the first and second peripheral electrodes within the tablet surface 5a. Then, based on the positions of the first and second peripheral electrodes, it detects the twist angle (TWIST) of the pen 6. tab .

[0056] Furthermore, the method for obtaining the zero-position posture information in the active electrostatic method described here is one example; of course, other methods can also be used. For instance, similar to the electromagnetic induction (EMR) method, a gyroscope, accelerometer, and magnetometer can be incorporated into the pen 6, which the pen 6 uses to detect the tilt angle (XTILT). tab YTILT tab and twist angle TWIST tab The tablet terminal 5 receives the tilt angle (XTILT) from the pen 6. tab YTILT tab and twist angle TWIST tab The data is used to detect the tilt angle (XTILT) of the pen 6. tab YTILT tab and twist angle TWIST tab In this case, the transmission of data from pen 6 can be achieved by configuring a data signal within the downlink signal. In this scenario, tablet terminal 5 obtains the data transmitted by pen 6 by demodulating the data signal contained in the detected downlink signal. The data transmitted by pen 6 may also include pen pressure values ​​indicating the pressure applied to the pen tip, on / off information indicating the on / off state of a switch located on the pen 6's housing, and a pen ID serving as identification information for pen 6.

[0057] The tablet terminal 5 is configured to sequentially supply the zeroth position pose information, as described above, to the processor 2a. The processor 2a uses this supplied zeroth position pose information to process the detection of the tablet terminal 5's position and pose in the virtual reality spatial coordinate system. Details of this process will be discussed later. Figure 4 Please provide a detailed explanation.

[0058] In addition, the tablet terminal 5 is based on the obtained planar coordinates (X... tab Y tab The process involves processing data such as pen pressure to generate stroke data, which serves as the result of pen input, i.e., tablet input data. Stroke data comprises a series of coordinate data representing the trajectory of the pen tip. Among these coordinate data, besides the planar coordinates (X...)... tab Y tab In addition to the above-mentioned pen pressure value, on / off information, and height information Z, it may also include the above-mentioned pen pressure value, on / off information, and height information Z. tab Tilt angle (XTILT) tab YTILT tab Twist angle tab The tablet terminal 5 stores the generated stroke data and processes it by rendering and displaying it sequentially on the tablet surface 5a.

[0059] The tablet terminal 5 is configured to supply the generated stroke data to the processor 2a. Upon receiving the stroke data from the tablet terminal 5, the processor 2a transforms the zeroth position pose information contained therein into position pose information in the virtual reality spatial coordinate system based on the position and pose of the tablet terminal 5 in the virtual reality spatial coordinate system. Details of this transformation will be described later, but the use of the position and pose of the tablet terminal 5 in the virtual reality spatial coordinate system during the transformation is to display the 3D object representing the stroke data at the same position as the actual tablet surface 5a. The processor 2a stores the transformed stroke data in memory as one of the aforementioned 3D object information and performs rendering and configuration processing within the XR space. Thus, the user can visually recognize the tablet input data, which is the result of pen input, as a 3D object within the XR space.

[0060] To perform the aforementioned transformation, the arithmetic unit 2a sequentially detects the position and posture of the tablet terminal 5 in the virtual reality spatial coordinate system. Specifically, based on the 0th position posture information sequentially supplied from the tablet terminal 5, the arithmetic unit 2a acquires first position posture information representing the position and posture of the pen 6 in the world coordinate system each time, and based on the information sequentially supplied from the XR tracking system 2b (information representing the position and posture of the pen 6 stored in memory), it acquires second position posture information representing the position and posture of the pen 6 in the virtual reality spatial coordinate system each time. Furthermore, the arithmetic unit 2a is configured to detect third position posture information representing the position and posture of the tablet terminal 5 in the virtual reality spatial coordinate system by transforming the second position posture information based on the first position posture information.

[0061] Figure 4 This is a flowchart of the process executed by the arithmetic unit 2a. The following refers to this... Figure 4 More specifically, this describes the processing performed by the arithmetic unit 2a to detect the position and posture of the tablet terminal 5 in the virtual reality spatial coordinate system.

[0062] The arithmetic unit 2a first obtains the zero-position pose information (step S1) from the tablet terminal 5, representing the position and pose of the pen 6 in the tablet coordinate system. The zero-position pose information includes the aforementioned planar coordinates (X... tab Y tab ), height information Z tab Tilt angle (XTILT) tab YTILT tab Twist angle tab .

[0063] Next, the arithmetic unit 2a transforms the obtained 0th position pose information based on the pre-stored relationship (transformation rule) between the flat plate coordinate system and the world coordinate system, thereby obtaining the first position pose information representing the position and pose of the pen 6 in the world coordinate system (step S2).

[0064] More specifically, the first position pose information includes a translation vector TV representing the position of pen 6 in the world coordinate system. pen And the rotation matrix R representing the pose of pen 6 in the world coordinate system. pen Among them, for the translation vector TV pen The arithmetic unit 2a obtains the translation vector TV by performing the transformation represented by the following equation (1). pen =(PEN_X world PEN_Y world PEN_Z world ). Among them, XMAX is included in equation (1). tab YMAX tab They are X tab Y tab The maximum values ​​of XREStab and YRES tab ZRES tab They are X tab Y tab Z tab The resolution. Here, we assume XRES tab YRES tab ZRES tab All values ​​are 0.01 mm / point, but of course, other values ​​are also possible. Multiplying the denominator on the right side of equation (1) by 1000 is because XRES... tab YRES tab ZRES tab The unit is set to mm / point.

[0065] [Number 1]

[0066] In addition, the arithmetic unit 2a obtains the rotation matrix R by performing the transformation shown in equation (2). pen The R contained in the middle edge of equation (2) x R y R z It is calculated using equation (3). Additionally, the right-hand side of equation (2) represents the rotation matrix R through variables. pen The elements are used in the following description.

[0067] [Number 2]

[0068] Next, the processor 2a obtains second position and pose information representing the position and pose of the pen 6 in the virtual reality spatial coordinate system based on the information from the XR tracking system 2b (step S3). Specifically, the information from the XR tracking system 2b includes a translation vector TV representing the position of the pen 6 in the virtual reality spatial coordinate system. ext =(X ext Y ext Z ext ), and the rotation matrix R representing the pose of pen 6 in the virtual reality spatial coordinate system. ext The arithmetic unit 2a is based on these translation vectors TV ext and rotation matrix R ext Generate an affine transformation matrix M2 (the second affine transformation matrix) representing the position and pose of pen 6 in the virtual reality spatial coordinate system, and obtain the generated affine transformation matrix M2 as the second position and pose information. If the rotation matrix R... ext If each element is represented by a variable as in equation (4), then the specific content of the affine transformation matrix M2 is represented by the following equation (5).

[0069] [Number 3]

[0070] Finally, the arithmetic unit 2a transforms the second position and pose information based on the first position and pose information to obtain the third position and pose information representing the position and pose of the tablet terminal 5 in the virtual reality spatial coordinate system (step S4). In summary, the transformation of the second position and pose information based on the first position and pose information is the inverse matrix M1 of the affine transformation matrix M1 (first affine transformation matrix) obtained from the first position and pose information. -1 The process involves multiplying by the affine transformation matrix M2. Equation (6) below represents this process. The third position pose information is obtained by multiplying by the affine transformation matrix M2, which is the result of this process. tab Representation. Inverse matrix M1 -1 This can be described as a transformation rule that converts the position and pose of pen 6 into the position and pose of tablet terminal 5. Therefore, by applying the inverse matrix M1 to the position and pose of pen 6 (=M2) in the virtual reality spatial coordinate system... -1 It can determine the position and orientation of the tablet terminal 5 in the virtual reality spatial coordinate system.

[0071] [Number 4]

[0072] Affine transformation matrix M1 and inverse matrix M1 -1These are matrices represented by equations (7) and (8), respectively. The arithmetic unit 2a can first obtain the affine transformation matrix M1 from the first position and pose information, and then obtain the inverse matrix M1 by performing operations on the inverse of the obtained affine transformation matrix M1. -1 However, the translation vector TV shown in equation (9) can also be obtained first. pen inverse vector -TV pen The rotation matrix R shown in equation (10) pen The inverse matrix R pen -1 And obtain the inverse matrix M1 based on them. -1 Therefore, the amount of time spent calculating the inverse matrix M1 can be reduced. -1 The computational complexity. Furthermore, equation (10) holds because the rotation matrix R... pen It is an orthogonal matrix, Rpen -1 =R pen T Established.

[0073] [Number 5]

[0074] The affine transformation matrix M obtained through the operation of equation (6) tab Become the translation vector TV tab and rotation matrix R tab Information, translation vector TV tab The rotation matrix R represents the position of the tablet terminal 5 in the virtual reality spatial coordinate system. tab This represents the pose of the tablet terminal 5 in the virtual reality spatial coordinate system. That is, if the affine transformation matrix M... tab The elements are represented by equation (11), then the translation vector TV tab and rotation matrix R tab These are expressed by equations (12) and (13), respectively. The third position pose information can also be represented by the translation vector TV. tab and rotation matrix R tab Information composed of combinations of these elements.

[0075] [Number 6]

[0076] Through the processing up to step S4, the detection of the position and posture of the tablet terminal 5 in the virtual reality spatial coordinate system is completed.

[0077] As explained above, the computer 2 according to this embodiment can acquire third position and pose information representing the position and pose of the tablet terminal 5 in the virtual reality spatial coordinate system, even without installing a position sensor on the tablet terminal 5. Therefore, without installing a position sensor on the tablet terminal 5, the result of pen input on the tablet surface 5a, i.e., tablet input data, can be transformed into a 3D object in XR space.

[0078] The preferred embodiments of the present invention have been described above, but the present invention is not limited to such embodiments in any way, and the present invention can certainly be implemented in various ways without departing from its spirit.

[0079] For example, the above embodiment illustrates a case where the flat coordinate system and the virtual reality space coordinate system are left-handed, and the world coordinate system is right-handed. However, each coordinate system can of course be either left-handed or right-handed. The mathematical formulas described above can be modified appropriately depending on whether each coordinate system is left-handed or right-handed.

[0080] Explanation of reference numerals in the attached figures

[0081] 1 XR System

[0082] 2 Computers

[0083] 2a Arithmetic Unit

[0084] 2b XR Tracking System

[0085] 3 Virtual Reality Displays

[0086] 4. Camera

[0087] 5. Tablet Terminal

[0088] 5a flat surface

[0089] 6 strokes

[0090] P represents position.

Claims

1. A computer having a processor, wherein, The processor acquires first position and posture information representing the position and posture of the stylus in a first coordinate system with the representative position of the device with the digitizer as the origin. Obtain second position and pose information representing the position and pose of the stylus in a second coordinate system that defines the XR space. Based on the first position and pose information and the second position and pose information, a third position and pose information representing the position and pose of the device with the digitizer in the second coordinate system is calculated.

2. The computer according to claim 1, wherein, The first position and pose information is represented by a first affine transformation matrix that indicates the position and pose of the stylus in the first coordinate system. The second position and pose information is represented by a second affine transformation matrix that indicates the position and pose of the stylus in the second coordinate system. The third positional pose information is represented by a third affine transformation matrix obtained by multiplying the inverse of the first affine transformation matrix by the second affine transformation matrix.

3. The computer according to claim 1, wherein, The processor obtains the first position posture information based on information output from the device with a digitizer. The information includes coordinate information indicating the position of the stylus in a third coordinate system set on the surface of the device with a digitizer, tilt information indicating the tilt of the stylus relative to the surface, and torsion information indicating the amount of rotation of the stylus about the stylus axis.

4. The computer according to claim 3, wherein, The coordinate information includes coordinates representing the position of the stylus in the surface and information representing the distance from the surface to the stylus.

5. The computer according to claim 3 or 4, wherein, The processor transforms the information output from the device with the digitizer based on the relationship between the first coordinate system and the third coordinate system, thereby obtaining the first position and posture information.

6. The computer according to claim 1, wherein, Multiple markings are arranged on the surface of the stylus. The processor obtains the second positional pose information based on the positions of the plurality of markers contained in the image of the stylus captured by the camera.

7. A method executed by a computer having a processor, wherein, The method includes: The step of the processor acquiring first position posture information representing the position and posture of the stylus in a first coordinate system with the representative position of the device with the digitizer as the origin; The step of the processor obtaining second position and posture information representing the position and posture of the stylus in a second coordinate system defining the XR space; and The step of the processor calculating third position and pose information representing the position and pose of the device with the digitizer in the second coordinate system based on the first position and pose information and the second position and pose information.

8. The method according to claim 7, wherein, The first position and pose information is represented by a first affine transformation matrix that indicates the position and pose of the stylus in the first coordinate system. The second position and pose information is represented by a second affine transformation matrix that indicates the position and pose of the stylus in the second coordinate system. The third positional pose information is represented by a third affine transformation matrix obtained by multiplying the inverse of the first affine transformation matrix by the second affine transformation matrix.

9. The method according to claim 7, wherein, The processor obtains the first position posture information based on information output from the device with a digitizer. The information includes coordinate information indicating the position of the stylus in a third coordinate system set on the surface of the device with a digitizer, tilt information indicating the tilt of the stylus relative to the surface, and torsion information indicating the amount of rotation of the stylus about the stylus axis.

10. The method according to claim 9, wherein, The coordinate information includes coordinates representing the position of the stylus in the surface and information representing the distance from the surface to the stylus.

11. The method according to claim 9 or 10, wherein, The processor transforms the information output from the device with the digitizer based on the relationship between the first coordinate system and the third coordinate system, thereby obtaining the first position and posture information.

12. The method according to claim 7, wherein, Multiple markings are configured on the surface of the stylus. The processor obtains the second positional pose information based on the positions of the plurality of markers contained in the image of the stylus captured by the camera.

13. A program for causing a computer to perform the following steps: Obtain first position and posture information representing the position and posture of the stylus in a first coordinate system with the representative position of the device with the digitizer as the origin; Obtain second position and pose information representing the position and pose of the stylus in a second coordinate system defining the XR space; and Based on the first position and pose information and the second position and pose information, a third position and pose information representing the position and pose of the device with the digitizer in the second coordinate system is calculated.

14. The procedure according to claim 13, wherein, The first position and pose information is represented by a first affine transformation matrix that indicates the position and pose of the stylus in the first coordinate system. The second position and pose information is represented by a second affine transformation matrix that indicates the position and pose of the stylus in the second coordinate system. The third positional pose information is represented by a third affine transformation matrix obtained by multiplying the inverse of the first affine transformation matrix by the second affine transformation matrix.

Images