Calibration method and coordinate conversion tool

Abstract

[Problem] To provide a calibration method and a coordinate conversion tool which are capable of making a tablet terminal easily utilizable for XR technology. [Solution] A calibration method according to the present invention is executed by a computer. The computer calculates, on the basis of the position and posture of a coordinate conversion tool included in an image obtained by imaging the coordinate conversion tool located on a surface of a device having a digitizer, a conversion rule for performing coordinate conversion processing that converts, into a second coordinate designating a position in an XR space, a first coordinate designating a position on the surface indicated by a stylus.

Description

Calibration methods and coordinate transformation tools

[0001] The present invention relates to a calibration method and a coordinate transformation tool, and more particularly to a calibration method for handling the results of pen input performed on a tablet terminal as an object in XR space, and a coordinate transformation tool for implementing the calibration method.

[0002] In conventional XR (eXtended Reality) technology, a user operates a 3D controller in mid-air.

[0003] Patent Literature 1 discloses a technology that enables editing of 3D objects using a tablet device. In this technology, a tracker is placed on the tablet device. The position and orientation of the tablet device in real space are then detected based on the position and orientation of the tracker in real space.

[0004] International Publication No. 2019 / 102825

[0005] However, with the technology described in Patent Document 1, it is difficult to easily use a tablet terminal in XR technology.

[0006] Therefore, one object of the present invention is to provide a calibration method and a coordinate conversion tool that enable tablet terminals to be easily used in XR technology.

[0007] A calibration method according to one aspect of the present invention is a calibration method executed by a computer, in which the computer calculates a transformation rule for performing coordinate transformation processing to transform first coordinates specifying a position within the surface indicated by a stylus into second coordinates specifying a position within XR space, based on the position and orientation of a coordinate transformation tool contained in an image obtained by photographing the coordinate transformation tool located on the surface of a device having a digitizer.

[0008] A coordinate transformation tool according to one aspect of the present invention is a coordinate transformation tool used by a computer, which includes a two-dimensional code, and the computer calculates a transformation rule for performing a coordinate transformation process that transforms first coordinates that specify a position on the surface indicated by a stylus into second coordinates that specify a position in XR space, based on the position and shape of the two-dimensional code contained in an image obtained by photographing the coordinate transformation tool fixed to the surface of a device having a digitizer.

[0009] A calibration method according to another aspect of the present invention is a calibration method executed by a computer, in which the computer receives information indicating the position and orientation of a coordinate transformation tool relative to a surface of a device having a digitizer, and calculates a transformation rule for performing a coordinate transformation process that transforms first coordinates that specify a position on the surface indicated by a stylus into second coordinates that specify a position in the XR space, based on the received information indicating the position and orientation of the coordinate transformation tool relative to the surface and the position and orientation of the coordinate transformation tool in XR space.

[0010] A calibration method according to another aspect of the present invention may be a calibration method executed by a computer, in which the computer receives coordinates indicating a display position of a coordinate conversion tool on a surface of a device having a digitizer, and calculates a conversion rule for performing a coordinate conversion process that converts first coordinates specifying a position on the surface indicated by a stylus into second coordinates specifying a position in the XR space, based on the received coordinates indicating the display position of the coordinate conversion tool on the surface and the position and orientation of the coordinate conversion tool in XR space.

[0011] A coordinate transformation tool according to another aspect of the present invention is a coordinate transformation tool used by a computer, which includes a position indicator configured so that a device having a digitizer can detect its position and orientation relative to the surface, and the computer calculates a transformation rule for performing the coordinate transformation processing based on the position and orientation of the position indicator relative to the surface detected by the device having the digitizer.

[0012] A coordinate transformation tool according to another aspect of the present invention may be a coordinate transformation tool used by a computer, which includes an image displayed on a surface of a device having a digitizer, and the computer acquires, based on coordinates indicating a display position of the image displayed on the surface, a transformation rule for performing a coordinate transformation process that transforms first coordinates specifying a position on the surface indicated by a stylus into second coordinates specifying a position in XR space.

[0013] According to the present invention, it becomes possible to easily use a tablet terminal in XR technology.

[0014] [Correction based on Rule 91 22.08.2024] This figure shows an XR system 1 according to a first embodiment of the present invention. It is a top view of a tablet terminal 5. It is a sequence diagram showing a process in which a computer 2 acquires a transformation rule for converting between a tablet surface coordinate system and a virtual reality space coordinate system. It is a sequence diagram showing a process in which a computer 2, having acquired the transformation rule, displays the result of a pen input on a tablet terminal 5 as an object in an XR space. It is a sequence diagram showing a process for constantly updating the transformation rule acquired by the process of FIG. 3. (a) is a top view of a coordinate transformation tool T2 according to a modification of the first embodiment of the present invention, and (b) is a side view of the coordinate transformation tool T2 according to a modification of the first embodiment of the present invention. (a) is a top view of a tablet terminal 5 and a coordinate transformation tool T3 according to a second embodiment of the present invention, and (b) is a side view of the tablet terminal 5 and the coordinate transformation tool T3 according to the second embodiment of the present invention. It is a sequence diagram showing a process in which a computer 2 acquires a transformation rule for converting between a tablet surface coordinate system and a virtual reality space coordinate system. 1A is a top view of a tablet terminal 5 and a coordinate transformation tool T4 according to a first modified example of the second embodiment of the present invention, and FIG. 1B is a side view of the tablet terminal 5 and the coordinate transformation tool T4 according to the first modified example of the second embodiment of the present invention. 1A is a top view of a tablet terminal 5 and a coordinate transformation tool T5 according to a second modified example of the second embodiment of the present invention, and FIG. 1B is a side view of the tablet terminal 5 and the coordinate transformation tool T5 according to the second modified example of the second embodiment of the present invention. 1B is a top view of a tablet terminal 5 and a coordinate transformation tool T6 according to a third embodiment of the present invention. 1C is a sequence diagram showing processing by a computer 2 to acquire a transformation rule for converting between a tablet surface coordinate system and a virtual reality space coordinate system.

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0016] FIG. 1 is a diagram illustrating an XR system 1 according to a first embodiment of the present invention. As shown in the figure, the XR system 1 according to this embodiment includes a computer 2, a virtual reality display 3, multiple cameras 4, a tablet terminal 5, a pen 6, and a coordinate transformation tool T1. The computer 2 includes a computing unit 2a and an XR tracking system 2b as functional units. The computer 2 may be a single computer, or a composite computer that functions as a single computer by combining multiple computers. A two-dimensional code C (described below), shown in a speech bubble in the figure, is placed on the surface of the coordinate transformation tool T1.

[0017] The computer 2 is a device including a processor, a memory, and a communication device. The processor executes programs stored in the memory to realize various functions of the computer 2, including the computing unit 2a and the XR tracking system 2b shown in Fig. 1. The communication device is configured to communicate with the virtual reality display 3, the multiple cameras 4, and the tablet terminal 5 via wired or wireless communication under the control of the processor.

[0018] The calculator 2a is a functional unit that sets an XR space based on the positions of the multiple cameras 4, generates an image representing the set XR space, and supplies it to the virtual reality display 3. The x1, y1, and z1 axes shown in FIG. 1 indicate a virtual reality space coordinate system that defines the XR space set by the calculator 2a. The positions and orientations of various objects displayed in the XR space by the calculator 2a are represented by six-dimensional vectors (x1 coordinate, y1 coordinate, z1 coordinate, rotation amount around the x1 axis, rotation amount around the y1 axis, and rotation amount around the z1 axis) of this virtual reality space coordinate system.

[0019] The virtual reality display 3 is an XR display (head-mounted display) worn on a human head. Virtual reality displays generally available on the market come in a variety of types, including "transparent" and "non-transparent," "glasses-type" and "hat-type," and any of these can be used as the virtual reality display 3. When the XR space set by the calculator 2 a is a virtual reality (VR) space, a user wearing the virtual reality display 3 perceives virtual reality and is separated from the real world. On the other hand, when the XR space set by the calculator 2 a is an augmented reality (AR) space or a mixed reality (MR) space, a user wearing the virtual reality display 3 perceives a space in which virtual reality and the real world are mixed.

[0020] The computing unit 2a also performs a process of rendering various 3D objects (materials) and placing them within the image. The 3D objects to be rendered may include 3D objects that exist in reality, such as the tablet terminal 5 shown in FIG. 1, and 3D objects that do not exist in reality. As a result of this process performed by the computing unit 2a, a user wearing the virtual reality display 3 can view the 3D objects in the XR space.

[0021] The rendering by the computing unit 2a is performed based on 3D object information stored in memory. The 3D object information is information that indicates the shape, position, and orientation of a 3D object in the XR space set by the computing unit 2a, and is stored in memory for each 3D object to be rendered.

[0022] To generate an image representing the XR space, the calculator 2a first acquires the position and orientation of the virtual reality display 3. Specifically, the calculator 2a receives the position and orientation of the virtual reality display 3 from an XR tracking system 2b (described later). The calculator 2a determines the user's viewpoint based on the acquired position and orientation of the virtual reality display 3, and then performs rendering of 3D objects and generation of an image representing the XR space based on the determined viewpoint. This allows the user viewing the XR space through the virtual reality display 3 to see each 3D object in the same position as in real life.

[0023] The XR tracking system 2b is a functional unit that detects objects (real-world objects, including the virtual reality display 3) included in images captured by each of the multiple cameras 4 and tracks their positions and orientations. The multiple cameras 4 are positioned so that they can capture images from various angles of various locations in the real space corresponding to the XR space set by the computing unit 2a. Although three cameras 4 are shown in FIG. 1, more cameras 4 may be positioned in practice.

[0024] The XR tracking system 2b detects an object by detecting an optical marker (any type of optically detectable marker) attached to the object or by image recognition of the object. The tracking results by the XR tracking system 2b are sequentially stored in the memory of the calculator 2a as part of the above-mentioned 3D object information. The calculator 2a renders the 3D object that actually exists based on the tracking results stored in the memory.

[0025] The tablet terminal 5 is a device (computer) having a flat tablet surface 5a and is configured with a digitizer that detects the position of a position indicator on the tablet surface 5a. The tablet surface 5a serves as both an input surface that accepts pen input and a display surface that displays images, and the tablet terminal 5 is configured to be able to display various data on the tablet surface 5a, including stroke data (described below) that is the result of pen input. However, in this embodiment and a second embodiment described below, it is also possible to use a tablet terminal 5 whose tablet surface 5a does not function as a display surface.

[0026] The pen 6 is a pen-shaped electronic pen (stylus) and a position indicator. The user performs input (pen input) to the tablet terminal 5 by sliding the pen tip of the pen 6 on the tablet surface 5a. The specific method of pen input is not particularly limited, but, for example, an electromagnetic induction (EMR) method or an active electrostatic method can be suitably used. The tablet terminal 5 may also support input by a finger (touch input), and a specific method of touch input may be, for example, a capacitive method.

[0027] The tablet terminal 5 is configured with a function to sequentially detect the position of the pen 6 on the tablet surface 5a. The x2 axis and y2 axis shown in Figure 1 indicate a planar coordinate system that defines positions on the tablet surface 5a, and the position of the pen 6 detected by the tablet terminal 5 is represented by coordinates of this planar coordinate system.

[0028] Here, the specific configuration and processing for implementing pen input will be described in detail, taking the electromagnetic induction (EMR) method and the active electrostatic method as examples. First, in the case of using the electromagnetic induction (EMR) method, a tablet terminal 5 compatible with the electromagnetic induction (EMR) method is configured with multiple loop coils each extending in the x2-axis direction and multiple loop coils each extending in the y2-axis direction. Furthermore, a pen 6 compatible with the electromagnetic induction (EMR) method is configured with an LC resonant circuit consisting of a coil and a capacitor connected in series. The tablet terminal 5 supplies an alternating current to one of the loop coils to intermittently emit an alternating magnetic field from the tablet surface 5a. When the coil of the pen 6 enters this alternating magnetic field, the capacitor of the pen 6 is charged. When the tablet terminal 5 stops emitting the alternating magnetic field, the power stored in the capacitor causes the coil of the pen 6 to emit an alternating magnetic field as a reflected signal. The tablet terminal 5 attempts to detect this alternating magnetic field using each of the loop coils and detects the position of the pen 6 on the tablet surface 5a based on the distribution of the strength of the detected alternating magnetic field.

[0029] Next, a description will be given of a case where the active electrostatic method is used. A tablet terminal 5 that supports the active electrostatic method is configured with multiple linear electrodes each extending in the x2-axis direction and multiple linear electrodes each extending in the y2-axis direction. A pen 6 that supports the active electrostatic method is configured with a pen tip electrode provided at the pen tip, a processing circuit connected to this electrode, and a battery that supplies power to the processing circuit. The tablet terminal 5 transmits an uplink signal from the tablet surface 5a by supplying a signal to one of the linear electrodes. When the processing circuit of the pen 6 receives this uplink signal via the pen tip electrode, it generates a downlink signal as a response signal and transmits it from the pen tip electrode toward the tablet surface 5a. The tablet terminal 5 attempts to detect this downlink signal using each of the linear electrodes and detects the position of the pen 6 on the tablet surface 5a based on the distribution of the strength of the detected downlink signal.

[0030] The tablet terminal 5 is also configured with a function to acquire various data from the pen 6. This data may include a writing pressure value indicating the pressure applied to the pen tip, on / off information indicating the on / off state of a switch provided on the housing of the pen 6, and a pen ID, which is identification information for the pen 6. The pen 6 transmits this data by modulating the alternating magnetic field or downlink signal described above. The tablet terminal 5 acquires the data transmitted by the pen 6 by demodulating the received alternating magnetic field or downlink signal.

[0031] The tablet terminal 5 performs processing to generate stroke data indicating the trajectory of the pen tip based on the position of the pen 6 acquired as described above and the various data received from the pen 6. The stroke data is data represented by a series of coordinate data, and each piece of coordinate data may include the above-mentioned pen pressure value and on / off information in addition to planar coordinates indicating a position on the tablet surface. The tablet terminal 5 stores the generated stroke data and performs processing to display it on the tablet surface 5a.

[0032] Furthermore, each time the tablet terminal 5 detects the position of the pen 6, it supplies the calculator 2a with planar coordinates indicating the detected position. The calculator 2a performs coordinate conversion processing to convert the planar coordinates received from the tablet terminal 5 into coordinates in the virtual reality space coordinate system, stores the converted stroke data in memory as one piece of the above-mentioned 3D object information, and also performs processing to render and place the stroke data in the XR space. This allows the user to visually recognize the stroke data generated by pen input as a 3D object in the XR space.

[0033] FIG. 2 is a top view of the tablet terminal 5. As shown in the figure, a user attaches a coordinate transformation tool T1 at a known position and in a known orientation on the surface of the tablet terminal 5. Here, "at a known position and in a known orientation" means that the computer 2 pre-stores the position and orientation of the coordinate transformation tool T1 relative to the tablet surface 5a. The computer 2 stores the position and orientation of the coordinate transformation tool T1 relative to the tablet surface 5a using six-dimensional vectors (x2 coordinate, y2 coordinate, z2 coordinate, amount of rotation around the x2 axis, amount of rotation around the y2 axis, and amount of rotation around the z2 axis) of a three-dimensional coordinate system (tablet surface coordinate system) formed by adding a z2 axis, which is the normal direction of the tablet surface 5a, to the x2 axis and y2 axis within the tablet surface 5a.

[0034] 2, the coordinate transformation tool T1 is configured to have a box-shaped housing D1 and a two-dimensional code C placed on its surface. The two-dimensional code C is a type of optical marker described above, and is generated, for example, based on the identification information of the coordinate transformation tool T1. In a typical example, the two-dimensional code C is printed on the top surface of the housing D1.

[0035] Returning to Figure 1, in order to execute the coordinate conversion process described above, the computer 2 uses the coordinate conversion tool T1 to perform a process (calibration) to acquire a conversion rule for converting between the tablet surface coordinate system and the virtual reality space coordinate system. This process will be described in detail below.

[0036] FIG. 3 is a sequence diagram showing the process by which the computer 2 acquires the conversion rule. As shown in FIG. 3, in the initial state, the computing unit 2a, tablet terminal 5, and XR tracking system 2b are all operating in normal operation mode (step S1). That is, the XR tracking system 2b detects objects included in images captured by each of the multiple cameras 4, tracks their positions and orientations, and sequentially stores the results in the memory of the computer 2 as part of 3D object information. The computing unit 2a also generates an image representing the XR space including rendering results of various 3D objects and supplies the image to the virtual reality display 3. The tablet terminal 5 generates stroke data indicating the trajectory of the pen 6 on the tablet surface 5a, displays the data on the display surface, and sequentially supplies the detected position of the pen 6 to the computer 2.

[0037] The user fixes the coordinate transformation tool T1 at a known position on the tablet terminal 5 in a known orientation (step S2), and then performs a predetermined operation to transition the calculator 2a to a calibration mode (a mode for performing calibration) (step S3). This operation may be performed on the computer 2 or on the tablet terminal 5. Upon receiving the operation in step S3, the calculator 2a enters the calibration mode (step S4) and transmits an instruction to transition to the calibration mode to the XR tracking system 2b (step S5).

[0038] Upon receiving the instruction to transition to the calibration mode from the computing unit 2a, the XR tracking system 2b enters the calibration mode (step S6) and detects the position and orientation of the coordinate transformation tool T1 in the XR space (step S7). This detection is performed based on the position and shape of the two-dimensional code C contained in the images captured by the multiple cameras 4. The XR tracking system 2b transmits a six-dimensional vector in the virtual reality space coordinate system indicating the detected position and orientation to the computing unit 2a (step S8) and returns to the normal operation mode (step S9).

[0039] The user also inputs information about the tablet terminal 5 to be used into the computer 2 (step S10). While step S10 is depicted after step S3 in Fig. 3, step S10 may be performed before steps S2 and S3. Upon receiving the input information, the calculator 2a reads from the memory of the computer 2 a six-dimensional vector in the tablet surface coordinate system that indicates the position and orientation (position and orientation relative to the tablet surface 5a) of the coordinate transformation tool T1 that has been stored in advance for the tablet terminal 5 (step S11).

[0040] Next, the calculator 2a calculates a transformation rule for mutual transformation between the tablet surface coordinate system and the virtual reality space coordinate system based on the position and orientation of the coordinate transformation tool T1 read in step S11 and the position and orientation of the coordinate transformation tool T1 in the XR space received in step S8 (step S12). Specifically, the calculator 2a calculates a rotation matrix for transforming the six-dimensional vector read in step S11 into the six-dimensional vector received in step S8, and acquires the result as the transformation rule.

[0041] The calculator 2a determines the position and orientation of the tablet surface 5a in the XR space based on the position and orientation of the coordinate transformation tool T1 relative to the tablet surface 5a read in step S11 and the position and orientation of the coordinate transformation tool T1 in the XR space received in step S8 (step S13).Then, based on the position and orientation of the tablet surface 5a determined in step S13, an object representing the tablet terminal 5 is displayed in the XR space (step S14).

[0042] Thereafter, the calculator 2a notifies the user of the end of the calibration mode (step S15) and returns to the normal operation mode (step S16). The specific method of notification in step S15 is not particularly limited, and various methods such as display or audio notification can be adopted.

[0043] 4 is a sequence diagram showing the process in which the computer 2, which has acquired the conversion rule, displays the result of pen input on the tablet terminal 5 as an object in XR space. The process shown in the figure is executed each time the tablet terminal 5 detects the position of the pen 6. As shown in the figure, the tablet terminal 5 detects the position of the pen 6 on the tablet surface 5a as a result of pen input by the user (step S20) (step S21), and transmits planar coordinates indicating the detected pen position to the calculator 2a (step S22).

[0044] Upon receiving the planar coordinates indicating the pen position from the tablet terminal 5, the calculator 2a performs a coordinate transformation process to convert the received planar coordinates into coordinates in the virtual reality space coordinate system (a six-dimensional vector in the virtual reality space coordinate system) using the transformation rule calculated in step S12 of FIG. 3 (step S23). The calculator 2a generates a six-dimensional vector in the tablet surface coordinate system by adding a z2 coordinate, a rotation amount around the x2 axis, a rotation amount around the y2 axis, and a rotation amount around the z2 axis, all of which are set to zero, to the planar coordinates (two-dimensional vector) indicating the pen position. The calculator 2a then applies the transformation rule to the generated six-dimensional vector to calculate the six-dimensional vector in the virtual reality space coordinate system. The calculator 2a then displays an object indicating the trajectory of the pen 6 in the XR space using the coordinates obtained by the transformation in step S23 (step S24). This allows the user to visually recognize the results of pen input performed on the tablet terminal 5 as an object in the XR space.

[0045] Fig. 5 is a sequence diagram showing a process for continually updating the conversion rule acquired by the process in Fig. 3. When there is a possibility that the tablet terminal 5 will move after the computer 2 acquires the conversion rule, such as when the user is holding the tablet terminal 5 in his / her hand as shown in Fig. 1, the computer 2 needs to execute this process.

[0046] The XR tracking system 2b operating in the normal operation mode periodically detects the position and orientation of the coordinate transformation tool T1 in the XR space (step S30). The specific detection method may be the same as step S7 in FIG. 3 . The XR tracking system 2b then compares the detected position and orientation with the previously detected position and orientation to determine whether there has been a change (step S31). If there has been no change, no particular processing is performed. If there has been a change, a six-dimensional vector in the virtual reality space coordinate system indicating the detected position and orientation is transmitted to the calculator 2a (step S32). The calculator 2a may also perform the determination in step S31. In this case, the XR tracking system 2b simply transmits a six-dimensional vector in the virtual reality space coordinate system indicating the detected position and orientation to the calculator 2a.

[0047] In step S32, the calculator 2a receives from the XR tracking system 2b the six-dimensional vector in the virtual reality space coordinate system indicating the position and orientation of the coordinate transformation tool T1. Based on the received six-dimensional vector and the six-dimensional vector in the tablet surface coordinate system indicating the position and orientation of the coordinate transformation tool T1 read in step S11 of Fig. 3, the calculator 2a updates the transformation rule for mutual transformation between the tablet surface coordinate system and the virtual reality space coordinate system (step S33). Specifically, the calculator 2a calculates a rotation matrix for transforming the six-dimensional vector read in step S11 into the six-dimensional vector received in step S32, and reacquires the matrix as the transformation rule.

[0048] After updating the conversion rules, computing unit 2a updates the object that represents the trajectory of pen 6 that is being displayed in the XR space (step S34). Specifically, the six-dimensional vector in the virtual reality space coordinate system that corresponds to the representative position of the object (for example, the position of the starting point of the first stroke data) is updated in accordance with the new conversion rules, and the display of the entire object is updated based on the updated representative position.

[0049] 3 and the position and orientation of the coordinate transformation tool T1 in the XR space received in step S32 (step S35).The calculator 2a then updates the object representing the tablet terminal 5 currently displayed in the XR space based on the position and orientation of the tablet surface 5a redetermined in step S35 (step S36).

[0050] When the processing up to this point is completed, the calculator 2a may notify the user that the conversion rule has been updated (step S35). By notifying the user in this way, the user can know that the processing corresponding to the movement of the tablet terminal 5 is being performed appropriately.

[0051] As described above, according to the XR system 1 of this embodiment, the computer 2 can acquire the transformation rule for mutual transformation between the tablet surface coordinate system and the virtual reality space coordinate system, based on the position and orientation of the coordinate transformation tool T1 contained in the images captured by the multiple cameras 4. Therefore, compared to the case in which a light-receiving sensor is used as the coordinate transformation tool as in Patent Document 1, it is possible to achieve calibration using an inexpensive coordinate transformation tool.

[0052] Furthermore, according to the XR system 1 of this embodiment, the coordinate transformation tool T1 is provided with a two-dimensional code C, so that the computer 2 can obtain the position and orientation of the coordinate transformation tool T1 contained in the image based on the position and shape of the two-dimensional code C.

[0053] In the present embodiment, the user is responsible for attaching the coordinate transformation tool T1 to the surface of the tablet terminal 5 at a known position in a known orientation. However, it is often difficult to manually attach the coordinate transformation tool T1 without misalignment. Therefore, a fixture may be used to more easily fix the coordinate transformation tool to the surface of the tablet terminal 5.

[0054] Fig. 6(a) is a top view of a coordinate transformation tool T2 according to a modified example of the present embodiment, and Fig. 6(b) is a side view of the coordinate transformation tool T2 according to the modified example. Fig. 6(a) also shows a tablet terminal 5. The coordinate transformation tool T2 according to the modified example differs from the coordinate transformation tool T1 according to the present embodiment in that, in addition to the same housing D1 and two-dimensional code C as the coordinate transformation tool T1 according to the present embodiment, it also has a fixture E for fixing the coordinate transformation tool T2 to the tablet terminal 5. The housing D1 is fixed to the surface of the fixture E in advance.

[0055] 6(a) and 6(b) is configured to have an insertion cavity Ea into which the short side of the tablet terminal 5 can be fitted. This allows the user to easily fix the coordinate transformation tool T2 at a known position on the surface of the tablet terminal 5 in a known orientation by simply fitting the tablet terminal 5 into the insertion cavity Ea.

[0056] Next, an XR system 1 according to a second embodiment of the present invention will be described. The XR system 1 according to this embodiment differs from the XR system 1 according to the first embodiment in that the coordinate transformation tool includes a position indicator and the tablet terminal 5 detects the position and orientation of the coordinate transformation tool. Note that in this embodiment, the tablet terminal 5 is described as being compatible with the electromagnetic induction (EMR) method. Since the XR system 1 according to the second embodiment is otherwise similar to the XR system 1 according to the first embodiment, the following description will focus on the differences from the XR system 1 according to the first embodiment.

[0057] 7(a) is a top view of the tablet terminal 5 and the coordinate conversion tool T3 according to the present embodiment, and FIG. 7(b) is a side view of the tablet terminal 5 and the coordinate conversion tool T3 according to the present embodiment. As shown in FIG. 7(a), the coordinate conversion tool T3 according to the present embodiment, like the coordinate conversion tool T1 according to the first embodiment, is configured to have a box-shaped housing D3 and a two-dimensional code C disposed on its surface. On the other hand, as shown in FIG. 7(b), the coordinate conversion tool T3 according to the present embodiment is configured to further have an electromagnetic induction (EMR) type position indicator D3a inside the housing D3.

[0058] Specifically, the position indicator D3a is a circuit including an LC resonant circuit made up of a coil and a capacitor connected in series. The tablet terminal 5 detects the position of the position indicator D3a (i.e., the position of the coordinate transformation tool T3) on the tablet surface 5a in the same manner as the process described above for detecting the position of the pen 6.

[0059] The tablet terminal 5 is also configured to detect the attitude (amount of rotation around the z2 axis) of the coordinate transformation tool T3 relative to the tablet surface 5a. To enable this detection, the coil that constitutes the position indicator D3a is arranged in the housing D3 so that it is oblique to the tablet surface 5a when the coordinate transformation tool T3 is fixed to the tablet surface 5a. The tablet terminal 5 is configured to detect the amount of rotation around the z2 axis of the coordinate transformation tool T3 based on the distribution of detected strength within the tablet surface 5a of the alternating magnetic field emitted by the position indicator D3a.

[0060] 8 is a sequence diagram showing the process for the computer 2 to acquire a conversion rule according to this embodiment. The following description will focus on the differences from the process shown in FIG.

[0061] In this embodiment, the user fixes the coordinate transformation tool T3 at any position on the tablet surface 5a in any orientation (with the two-dimensional code C on the top surface) (step S40), and then performs a predetermined operation to transition the calculator 2a to calibration mode (step S3).

[0062] In step S5, the calculator 2a according to this embodiment transmits a calibration mode transition instruction not only to the XR tracking system 2b but also to the tablet terminal 5. The processing performed by the XR tracking system 2b upon receiving the calibration mode transition instruction is the same as that shown in FIG. 3 (steps S6 to S9). Meanwhile, upon receiving the calibration mode transition instruction, the tablet terminal 5 enters the calibration mode (step S41) and detects the position and orientation of the coordinate transformation tool T3 relative to the tablet surface 5a (step S42). The specific method of this detection is as described above. The tablet terminal 5 transmits a six-dimensional vector in the tablet surface coordinate system indicating the detected position and orientation to the calculator 2a (step S43) and returns to the normal operation mode (step S44). In addition, the tablet terminal 5 sets the x2 coordinate and y2 coordinate of the elements of the six-dimensional vector transmitted in step S43 to plane coordinates indicating the position of the detected coordinate conversion tool T3, sets the rotation amount around the z2 axis to the rotation amount around the z2 axis of the detected coordinate conversion tool T3, and sets the z2 coordinate, rotation amount around the x2 axis, and rotation amount around the y2 axis to 0.

[0063] The computing unit 2a receives the six-dimensional vector of the tablet surface coordinate system indicating the position and orientation of the coordinate transformation tool T3 relative to the tablet surface 5a in step S43, and receives the six-dimensional vector of the virtual reality space coordinate system indicating the position and orientation of the coordinate transformation tool T3 in XR space in step S8. The computing unit 2a performs the same processing as steps S12 to S16 shown in Fig. 3. In this case, the computing unit 2a uses the six-dimensional vector of the tablet surface coordinate system received in step S43 instead of the six-dimensional vector of the tablet surface coordinate system read in step S11. As a result, a transformation rule for performing the coordinate transformation processing is calculated, and an object indicating the tablet terminal 5 is displayed in XR space.

[0064] As described above, with the XR system 1 according to this embodiment, the computer 2 can also acquire the transformation rule for performing coordinate transformation processing based on the position and orientation of the coordinate transformation tool T3 contained in the images captured by the multiple cameras 4. Therefore, compared to the case in which a light-receiving sensor is used as the coordinate transformation tool as in Patent Document 1, it is possible to achieve calibration using an inexpensive coordinate transformation tool.

[0065] Furthermore, in the XR system 1 according to this embodiment, the coordinate transformation tool T3 is provided with a two-dimensional code C, so that the computer 2 can obtain the position and orientation of the coordinate transformation tool T3 contained in the image based on the position and shape of the two-dimensional code C.

[0066] Furthermore, according to the XR system 1 of this embodiment, the position indicator D3a is disposed within the coordinate transformation tool T3, so the user can fix the coordinate transformation tool T3 at any position on the tablet surface 5a in any orientation, thereby reducing the burden on the user regarding the coordinate transformation tool.

[0067] In this embodiment, too, it is preferable that the computer 2 perform processing similar to the update processing shown in Fig. 5. In this update processing, the computer 2 may use the six-dimensional vector in the tablet surface coordinate system indicating the position and orientation of the coordinate transformation tool T3 received in step S43 of Fig. 8, instead of the six-dimensional vector in the tablet surface coordinate system indicating the position and orientation of the coordinate transformation tool T1 read in step S11 of Fig. 3.

[0068] In addition, in this embodiment, an example has been described in which the coil constituting the position indicator D3a is arranged within the housing D3 so that it is angled with respect to the tablet surface 5a when the coordinate transformation tool T3 is fixed to the tablet surface 5a, so that the tablet terminal 5 can detect the attitude of the coordinate transformation tool T3 (amount of rotation around the z2 axis).However, it is also possible to have the tablet terminal 5 detect the attitude of the coordinate transformation tool (amount of rotation around the z2 axis) using other methods.

[0069] 9(a) is a top view of a tablet terminal 5 and a coordinate transformation tool T4 according to a first modified example of the present embodiment, and FIG. 9(b) is a side view of the tablet terminal 5 and the coordinate transformation tool T4 according to this modified example. As shown in these figures, the coordinate transformation tool T4 is similar to the coordinate transformation tool T3 according to the present embodiment in that it is configured with a box-shaped housing D4 and a two-dimensional code C placed on its surface, but differs from the coordinate transformation tool T3 according to the present embodiment in that the housing D4 has a shape that is longer and thinner than the housing D3.

[0070] 9(b), two electromagnetic induction (EMR) type position indicators D4a are disposed inside the housing D4. Each position indicator D4a is similar to the position indicator D3a in that it is configured by a circuit including an LC resonant circuit consisting of a coil and a capacitor connected in series. However, each position indicator D4a differs from the position indicator D3a in that each coil is disposed within the housing D4 so as to be horizontal to the tablet surface 5a when the coordinate transformation tool T4 is fixed to the tablet surface 5a. As shown in FIG. 9(b), the coils of each position indicator D4a are disposed at a fixed distance in a direction parallel to the tablet surface 5a.

[0071] The tablet terminal 5 according to this modification is configured to detect the position of the coordinate transformation tool T4 on the tablet surface 5a and detect the orientation (amount of rotation around the z2 axis) of the coordinate transformation tool T4 relative to the tablet surface 5a by detecting the positions of the two position indicators D4a included in the coordinate transformation tool T4. As a result, according to this modification, as in the present embodiment, the computer 2 can obtain the transformation rule for performing coordinate transformation processing.

[0072] Furthermore, in this embodiment, an example has been described in which the tablet terminal 5 supports pen input using an electromagnetic induction (EMR) method, but even if the tablet terminal 5 supports pen input using another method, it is possible to obtain the same effect as in this embodiment.

[0073] [Correction based on Rule 91 22.08.2024] Figure 10(a) is a top view of a tablet terminal 5 and a coordinate transformation tool T5 according to a second modified example of this embodiment, and Figure 10(b) is a side view of the tablet terminal 5 and the coordinate transformation tool T5 according to this modified example. In this modified example, an example will be described in which the tablet terminal 5 is an active electrostatic type.

[0074] The coordinate transformation tool T5 according to this modified example is similar to the coordinate transformation tool T4 according to the first modified example of this embodiment in that it is configured with a long, narrow, box-shaped housing D5 and a two-dimensional code C arranged on its surface. However, it differs from the coordinate transformation tool T4 according to the first modified example of this embodiment in that, instead of two position indicators D4a, two position indicators D5a, each corresponding to an active electrostatic method, are arranged within the housing D5.

[0075] Specifically, each position indicator D5a includes the pen tip electrode, processing circuit, and battery described above. However, the processing circuit and battery may be common to both position indicators D5a. The pen tip electrodes of each position indicator D5a are arranged at a fixed distance in a direction parallel to the tablet surface 5a, as shown in FIG. 10(b).

[0076] The tablet terminal 5 according to this modification is configured to detect the position of the coordinate transformation tool T5 on the tablet surface 5a and detect the orientation (amount of rotation around the z2 axis) of the coordinate transformation tool T5 relative to the tablet surface 5a by detecting the positions of each of the two position indicators D5a included in the coordinate transformation tool T5. As a result, according to this modification, as in the present embodiment and the first modification, the computer 2 can acquire the transformation rule for performing coordinate transformation processing.

[0077] Next, an XR system 1 according to a third embodiment of the present invention will be described. The XR system 1 according to this embodiment differs from the XR system 1 according to the second embodiment in that the coordinate conversion tool is configured by an image displayed on the tablet surface 5a. In other respects, the XR system 1 according to this embodiment is the same as the XR system 1 according to the second embodiment, so the following description will focus on the differences from the XR system 1 according to the second embodiment.

[0078] 11 is a top view of the tablet terminal 5 and the coordinate transformation tool T6 according to this embodiment. As shown in the figure, the coordinate transformation tool T6 according to this embodiment is configured by a two-dimensional code C displayed on the tablet surface 5a. Because the coordinate transformation tool T6 is displayed on the tablet terminal 5, the tablet terminal 5 can obtain the planar coordinates indicating the display position of the coordinate transformation tool T6 on the tablet surface 5a and the amount of rotation of the coordinate transformation tool T6 around the z2 axis.

[0079] 12 is a sequence diagram showing the process for the computer 2 to acquire a conversion rule according to this embodiment. The following description will focus on the differences from the process shown in FIG.

[0080] In this embodiment, the user first performs a predetermined operation to transition the computing unit 2a to the calibration mode (step S3). In response to this operation, the computing unit 2a and the XR tracking system 2b perform the same processing as that described with reference to FIG.

[0081] In this embodiment, when the tablet device 5 receives a calibration mode transition instruction from the computing unit 2a (step S5), it enters the calibration mode (step S50) and displays the coordinate transformation tool T6, which is a two-dimensional code C, at an arbitrary position on the tablet surface 5a (step S51). Then, the tablet device 5 transmits a six-dimensional vector in the tablet surface coordinate system indicating the display position and display orientation of the coordinate transformation tool T6 to the computing unit 2a (step S52) and returns to the normal operation mode (step S53). Note that, among the elements of the six-dimensional vector transmitted in step S52, the tablet device 5 sets the x2 coordinate and y2 coordinate to plane coordinates indicating the display position of the coordinate transformation tool T6, sets the rotation amount around the z2 axis of the coordinate transformation tool T6 to the rotation amount around the z2 axis, and sets the z2 coordinate, the rotation amount around the x2 axis, and the rotation amount around the y2 axis to zero.

[0082] In step S52, the calculator 2a receives the six-dimensional vector in the tablet surface coordinate system indicating the display position and display orientation of the coordinate transformation tool T6, and in step S8, the calculator 2a receives the six-dimensional vector in the virtual reality space coordinate system indicating the position and orientation of the coordinate transformation tool T6 in the XR space. The calculator 2a then performs the same processing as steps S12 to S16 shown in Fig. 3. As a result, a transformation rule for performing the coordinate transformation processing is calculated, and an object indicating the tablet terminal 5 is displayed in the XR space.

[0083] As described above, with the XR system 1 according to this embodiment, the computer 2 can also acquire the transformation rule for performing coordinate transformation processing based on the position and orientation of the coordinate transformation tool T6 contained in the images captured by the multiple cameras 4. Therefore, compared to the case in which a light-receiving sensor is used as the coordinate transformation tool as in Patent Document 1, it is possible to achieve calibration using an inexpensive coordinate transformation tool.

[0084] Furthermore, according to the XR system 1 of this embodiment, the coordinate transformation tool T6 is configured using the two-dimensional code C displayed on the tablet surface 5a, and the computer 2 can acquire the position and orientation of the coordinate transformation tool T6 included in the image based on the position and shape of the two-dimensional code C. Furthermore, since the user does not need to physically handle the coordinate transformation tool T6, the burden on the user regarding the coordinate transformation tool can be reduced.

[0085] In the present embodiment, an example has been described in which a six-dimensional vector in the tablet surface coordinate system is transmitted from the tablet terminal 5 to the computing unit 2a in step S52, but if the display attitude (amount of rotation around the z2 axis) of the coordinate transformation tool T6 is predetermined, the tablet terminal 5 may transmit to the computing unit 2a only the planar coordinates indicating the display position of the coordinate transformation tool T6. In this case, the tablet terminal 5 can acquire the six-dimensional vector in the tablet surface coordinate system indicating the display position and display attitude of the coordinate transformation tool T6 by generating a six-dimensional vector in the tablet surface coordinate system from the received planar coordinates.

[0086] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and it goes without saying that the present invention can be embodied in various forms without departing from the spirit of the present invention.

[0087] 1 XR system 2 Computer 2a Calculator 2b XR tracking system 3 Virtual reality display 4 Camera 5 Tablet terminal 5a Tablet surface 6 Pen C Two-dimensional code D1, D3 to D5 Housing D3a to D5a Position indicator E Fixture Ea Insertion cavity T1 to T6 Coordinate conversion tool

Claims

1. A calibration method executed by a computer, in which the computer calculates a conversion rule for performing coordinate conversion processing to convert first coordinates specifying a position on the surface indicated by a stylus into second coordinates specifying a position in XR space, based on the position and orientation of a coordinate conversion tool contained in an image obtained by photographing a coordinate conversion tool located on the surface of a device having a digitizer.

2. The calibration method according to claim 1, wherein the coordinate transformation tool includes a two-dimensional code, and the position and orientation of the coordinate transformation tool are detected based on the position and shape of the two-dimensional code included in the image.

3. The calibration method of claim 1, wherein the coordinate transformation tool includes a position indicator configured to enable an equipment having the digitizer to detect its position and orientation relative to the surface, and the computer receives information indicating the position and orientation of the position indicator relative to the surface, and calculates a transformation rule for performing the coordinate transformation processing based on the received information indicating the position and orientation of the position indicator relative to the surface.

4. The calibration method according to claim 3, wherein an apparatus having the digitizer obtains information indicating the position and orientation of the position indicator relative to the surface by detecting an alternating magnetic field emitted by the position indicator.

5. The calibration method according to claim 3, wherein the device having the digitizer obtains information indicating the position and orientation of the position indicator relative to the surface by detecting a downlink signal transmitted by the position indicator.

6. The calibration method according to claim 1, wherein the computer receives coordinates indicating a display position of the coordinate transformation tool displayed on the surface, and calculates a transformation rule for performing the coordinate transformation process based on the received coordinates indicating the display position of the coordinate transformation tool.

7. A calibration method executed by a computer, wherein the computer receives information indicating the position and orientation of a coordinate transformation tool relative to a surface of a device having a digitizer, and calculates a transformation rule for performing coordinate transformation processing to transform first coordinates specifying a position on the surface indicated by a stylus into second coordinates specifying a position in the XR space, based on the received information indicating the position and orientation of the coordinate transformation tool relative to the surface and the position and orientation of the coordinate transformation tool in XR space.

8. The calibration method according to claim 7, wherein the coordinate transformation tool includes a position indicator configured to enable an equipment having the digitizer to detect its position and orientation relative to the surface, and information indicating the position and orientation of the coordinate transformation tool relative to the surface is information indicating the position and orientation of the position indicator relative to the surface detected by the equipment having the digitizer.

9. The calibration method according to claim 8, wherein an apparatus having the digitizer obtains information indicating the position and orientation of the position indicator relative to the surface by detecting an alternating magnetic field emitted by the position indicator.

10. The calibration method according to claim 9, wherein the position indicator is configured to have a coil arranged so as to be oblique to an apparatus having the digitizer when the coordinate transformation tool is fixed to the surface.

11. The calibration method according to claim 9, wherein the coordinate conversion tool is configured to have a plurality of the position indicators.

12. The calibration method according to claim 8, wherein an apparatus having the digitizer obtains information indicating the position and orientation of the position indicator relative to the surface by detecting a downlink signal transmitted by the position indicator.

13. The calibration method according to claim 12, wherein the coordinate conversion tool is configured to have a plurality of the position indicators.

14. The calibration method according to claim 7, wherein the coordinate transformation tool includes a two-dimensional code, and the computer photographs the two-dimensional code and detects the position and orientation of the coordinate transformation tool based on the position and shape of the two-dimensional code contained in the image obtained by photographing.

15. A calibration method executed by a computer, comprising: receiving coordinates indicating a display position of a coordinate conversion tool on a surface of a device having a digitizer; and calculating a conversion rule for performing coordinate conversion processing to convert first coordinates specifying a position on the surface indicated by a stylus into second coordinates specifying a position in the XR space, based on the received coordinates indicating the display position of the coordinate conversion tool on the surface and the position and orientation of the coordinate conversion tool in XR space.

16. The calibration method according to claim 15, wherein the coordinate transformation tool includes a two-dimensional code, and the computer photographs the two-dimensional code and detects the position and orientation of the coordinate transformation tool based on the position and shape of the two-dimensional code contained in an image obtained by photographing the code.

17. A coordinate conversion tool used by a computer, the coordinate conversion tool including a two-dimensional code, the computer calculating a conversion rule for performing coordinate conversion processing to convert first coordinates specifying a position on the surface indicated by a stylus into second coordinates specifying a position in XR space, based on the position and shape of the two-dimensional code contained in an image obtained by photographing a coordinate conversion tool fixed to a surface of a device having a digitizer.

18. A coordinate transformation tool used by a computer, comprising a position indicator configured to enable a device having a digitizer to detect its position and orientation relative to the surface, wherein the computer calculates a transformation rule for performing the coordinate transformation process based on the position and orientation of the position indicator relative to the surface detected by the device having the digitizer.

19. The coordinate transformation tool according to claim 18, further comprising a two-dimensional code, wherein the computer calculates a transformation rule for performing the coordinate transformation process based on the position and shape of the two-dimensional code contained in an image obtained by photographing the coordinate transformation tool.

20. A coordinate conversion tool used by a computer, comprising an image displayed on a surface of a device having a digitizer, the computer acquiring a conversion rule for performing a coordinate conversion process for converting a first coordinate designating a position on the surface indicated by a stylus into a second coordinate designating a position in an XR space, based on coordinates indicating a display position of the image displayed on the surface.

21. The coordinate transformation tool according to claim 20, wherein the image includes a two-dimensional code, and the computer acquires a transformation rule for performing the coordinate transformation process based on the position and shape of the two-dimensional code contained in an image obtained by photographing the coordinate transformation tool.

Images