Computer, method, and program

Abstract

[Problem] To provide a computer, method, and program whereby a device corresponding to a second tracking system can easily be used in an XR space related to a first tracking system. [Solution] A computer according to the present invention has a processor. The processor calculates a first coordinate in a first coordinate system of a 3D object to be rendered in an XR space defined by the first coordinate system; calculates, for a device to be tracked by a tracking system related to the establishment of a second coordinate system that differs from the first coordinate system, a second coordinate in the second coordinate system; and, in response to a prescribed operation by a user, calculates a coordinate conversion equation for converting coordinates of the second coordinate system to coordinates of the first coordinate system on the basis of the first coordinate and the second coordinate.

Description

[Technical Field]

[0001] The present invention relates to a computer, method, and program for rendering 3D objects in a space (hereinafter referred to as "XR space") constructed using XR technologies such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and SR (Substitutional Reality). [Background technology]

[0002] XR technology uses a tracking system that detects the position and orientation of various devices such as head-mounted displays, controllers, etc. One example of a tracking system includes multiple cameras, and is configured to determine the position and orientation of each device in the XR space based on images of each device captured by each camera.

[0003] Before starting to use the tracking system, calibration is performed to match the position and orientation of the device displayed in the XR space with the position and orientation of the device in reality. Patent Documents 1 and 2 describe examples of this calibration. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 10,127,732 [Patent Document 2] International Publication No. 2019-175582 Summary of the Invention [Problem to be solved by the invention]

[0005] In recent years, the types of devices compatible with XR technology have become more diverse, creating a need to simultaneously use multiple devices manufactured by different vendors in a single XR space.However, the devices that can be used in an XR space built using a certain vendor's tracking system are limited to those that are compatible with that tracking system, and it has not been easy to use devices that are compatible with another vendor's tracking system in an XR space built using a certain vendor's tracking system.

[0006] Therefore, one object of the present invention is to provide a computer, a method, and a program that enable a device compatible with a second tracking system to be easily used in an XR space related to a first tracking system. [Means for solving the problem]

[0007] A computer according to the present invention is a computer having a processor, wherein the processor calculates first coordinates in a first coordinate system of a 3D object to be rendered in an XR space defined by the first coordinate system, calculates second coordinates in the second coordinate system for a device tracked by a tracking system involving the setting of a second coordinate system different from the first coordinate system, and calculates a coordinate transformation formula for transforming coordinates in the second coordinate system into coordinates in the first coordinate system based on the first coordinates and the second coordinates in response to a predetermined operation by a user.

[0008] A computer according to the present invention is a computer having a processor, and the processor may be configured to: render a virtual device in an XR space defined by a first coordinate system; calculate first coordinates in the first coordinate system of the virtual device rendered in the XR space; calculate second coordinates in the second coordinate system for a tracker provided on a device that is tracked by a tracking system involving the setting of a second coordinate system different from the first coordinate system; and calculate, in response to a predetermined operation by a user, a coordinate transformation formula for transforming coordinates in the second coordinate system into coordinates in the first coordinate system based on the calculated first coordinates and second coordinates.

[0009] A method according to the present invention includes the steps of: calculating first coordinates in a first coordinate system of a 3D object to be rendered in an XR space defined by the first coordinate system; calculating second coordinates in the second coordinate system for a device tracked by a tracking system involving the setting of a second coordinate system different from the first coordinate system; and calculating, in response to a predetermined operation by a user, a coordinate transformation formula for transforming coordinates in the second coordinate system into coordinates in the first coordinate system, based on the first coordinates and the second coordinates.

[0010] A method according to the present invention may also be configured to include the steps of: rendering a virtual device in an XR space defined by a first coordinate system; calculating first coordinates in the first coordinate system of the virtual device rendered in the XR space; and calculating second coordinates in the second coordinate system for a tracker provided on a device that is tracked by a tracking system involving the setting of a second coordinate system different from the first coordinate system; and calculating, in response to a predetermined operation by a user, a coordinate transformation formula for transforming coordinates in the second coordinate system into coordinates in the first coordinate system based on the calculated first coordinates and second coordinates.

[0011] A program according to the present invention causes a computer to execute the steps of: calculating first coordinates in a first coordinate system of a 3D object to be rendered in an XR space defined by the first coordinate system; calculating second coordinates in the second coordinate system for a device tracked by a tracking system involving the setting of a second coordinate system different from the first coordinate system; and calculating, in response to a predetermined operation by a user, a coordinate transformation formula for transforming coordinates in the second coordinate system into coordinates in the first coordinate system, based on the first coordinates and the second coordinates.

[0012] A program according to the present invention may also be configured to cause a computer to execute the steps of: rendering a virtual device in an XR space defined by a first coordinate system; calculating first coordinates in the first coordinate system of the virtual device rendered in the XR space, and calculating second coordinates in the second coordinate system for a tracker provided on a device tracked by a tracking system involving the setting of a second coordinate system different from the first coordinate system; and calculating, in response to a predetermined operation by a user, a coordinate transformation formula for transforming coordinates in the second coordinate system into coordinates in the first coordinate system based on the calculated first coordinates and second coordinates. [Effects of the Invention]

[0013] According to the present invention, it is possible to provide a computer, a method, and a program that enable a device compatible with a second tracking system to be easily used in an XR space related to a first tracking system. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a schematic block diagram showing functional blocks of a 3D object rendering system 1 according to a first embodiment of the present invention. [Figure 2] 1 is a diagram showing the overall configuration of a 3D object rendering system 1. FIG. [Figure 3] FIG. 1 is a diagram illustrating a basic hardware configuration of a computer 10. [Figure 4] 1A is a diagram showing the external appearance of the controller C2, and FIG. 1B is a diagram showing the internal configuration of the controller C2. [Figure 5] 10 is a flow diagram showing the processing flow of calibration processing performed by an application unit 20 and a device information acquisition unit 23. FIG. [Figure 6] FIG. 6 is a diagram showing a virtual device 40 displayed in step S2 of FIG. 5. [Figure 7] FIG. 10 is a diagram showing an internal configuration of a controller C2 according to a second embodiment of the present invention. [Figure 8] 10 is a flow diagram showing the processing flow of calibration processing performed by an application unit 20 and a device information acquisition unit 23. FIG. [Figure 9] 9 is a diagram showing a virtual device 40 and tilt display information 41 displayed in step S11 of FIG. 8. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0016] Fig. 1 is a schematic block diagram showing functional blocks of a 3D object rendering system 1 according to a first embodiment of the present invention, and Fig. 2 is a diagram showing the overall configuration of the 3D object rendering system 1.

[0017] 1, a 3D object rendering system 1 includes a computer 10, cameras 11a and 11b, a head-mounted display 12, cameras 13a to 13c, a coupling device 14, and controllers C1 and C2. Functionally, the computer 10 includes an application unit 20, an XR system unit 21, a tracking processing unit 22, a device information acquisition unit 23, and a coordinate conversion processing unit 24.

[0018] The cameras 11a and 11b, the head-mounted display 12, the coupling device 14, and the controller C1 are configured to be able to communicate directly with the computer 10 via a wired or wireless connection. The cameras 13a to 13c and the controller C2 are configured to be able to communicate directly with the coupling device 14 via a wired or wireless connection, and are configured to be able to communicate with the computer 10 via the coupling device 14. The cameras 11a and 11b and the XR system unit 21 constitute a tracking system T1 for tracking the head-mounted display 12 and the controller C1 (detecting their position and tilt), and the cameras 13a to 13c, the coupling device 14, and the tracking processing unit 22 constitute a tracking system T2 for tracking the controller C2 (detecting its position and tilt).

[0019] Next, referring to FIG. 2, the 3D object rendering system 1 further includes a position detection device 15. The position detection device 15 has a touch surface and is capable of detecting the position of a pointer on the touch surface. In one example, the position detection device 15 is a digitizer connected to the computer 10 by wire or wirelessly, and is configured to supply the detected position of the pointer to the computer 10 each time. Note that while the position detection device 15 and the computer 10 are depicted as separate devices in FIG. 2, the position detection device 15 may also be a device built into the computer 10. In this case, the display of the computer 10 may also serve as the touch surface.

[0020] Cameras 11a, 11b and cameras 13a to 13c are positioned so as to capture images of the space above the desk top where the user is seated. More specifically, cameras 11a, 11b are installed above both ends of one side of the desk that is on the far side as seen from the user. Camera 13a is installed above the center of one side of the desk that is on the far side as seen from the user. Cameras 13b, 13c are installed above a position closer to the user than the center of each of the two sides of the desk as seen from the user. Coupling device 14 is built into camera 13a.

[0021] The controllers C1 and C2 are devices configured to be held by the user and used by the user. The head-mounted display 12 is a display device that is worn on the user's head and is configured to be able to display 3D images by projecting different images to the left and right eyes.

[0022] The positions and tilts of the controller C1 and head-mounted display 12 are detected by the tracking system T1 shown in Fig. 1. Specifically, the XR system unit 21 constituting the tracking system T1 detects the positions and tilts of the controller C1 and head-mounted display 12 based on images captured by the cameras 11a and 11b. The positions detected by the XR system unit 21 are represented, for example, by coordinates in a coordinate system (shown in Fig. 2 with coordinate axes X1, Y1, and Z1; hereinafter referred to as "coordinate system 1") whose origin is a predetermined position on the head-mounted display 12, and the tilt is represented by a quaternion indicating rotation in the coordinate system 1.

[0023] On the other hand, the position and tilt of the controller C2 are detected by the tracking system T2 shown in FIG. 1. As will be described in detail later, a plurality of trackers (for example, a plurality of points) are provided on the surface of the controller C2, and a tracking processing unit 22 constituting the tracking system T2 detects the position and tilt of the controller C2 by identifying the positions of at least three or more of these trackers based on images captured by the cameras 13a to 13c. The position detected by the tracking processing unit 22 is represented, for example, by coordinates in a coordinate system (shown in FIG. 2 with coordinate axes X2, Y2, and Z2; hereinafter referred to as "coordinate system 2") whose origin is a predetermined position of the coupling device 14, and the tilt is represented by a rotation matrix indicating rotation in coordinate system 2. However, similar to the tilt detected by the XR system unit 21, the tilt of the controller C2 may also be represented by a quaternion indicating rotation in coordinate system 2.

[0024] 2, the computer 10 is configured as a notebook personal computer placed in the center of a desk. However, the computer 10 does not need to be placed in the center of the desk, and may be placed in a position where it can communicate with the cameras 11a and 11b, the head-mounted display 12, the coupling device 14, and the controller C1. In addition to a notebook personal computer, the computer 10 may be configured as various types of computers, such as a desktop personal computer, a tablet personal computer, a smartphone, or a server computer.

[0025] 3 is a diagram showing the basic hardware configuration of the computer 10. As shown in the figure, the computer 10 is configured to have a configuration in which a processor 101, a storage device 102, a communication device 103, an input device 104, and an output device 105 are interconnected via a bus 106.

[0026] The processor 101 is a central processing unit that reads and executes programs stored in the storage device 102. The application unit 20, XR system unit 21, tracking processing unit 22, device information acquisition unit 23, and coordinate conversion processing unit 24 shown in FIG. 1 are each realized by the processor 101 reading and executing programs stored in the storage device 102. The processor 101 is configured to be able to communicate with each unit in the server via a bus 106, and controls each unit and processes data stored in the storage device 102 according to the description of the program to be executed.

[0027] The storage device 102 is a device that temporarily or permanently stores various programs and various data. The storage device 102 is typically configured by combining multiple storage devices, such as a main storage device configured with a dynamic random access memory (DRAM) or a static random access memory (SRAM), and an auxiliary storage device configured with a hard disk or a solid state drive (SSD).

[0028] The communication device 103 is a device that executes communication with external communication devices (including the cameras 11a and 11b, the head-mounted display 12, the coupling device 14, and the controller C1 shown in FIG. 1) under the control of the processor 101. The communication method used by the communication device 103 is not particularly limited, but examples include a wired or wireless WAN (Wide Area Network) or LAN (Local Area Network), or short-range wireless communication such as Bluetooth (registered trademark).

[0029] The input device 104 is a device that accepts input from a user, and includes various input means such as a mouse, a keyboard, a touch panel, etc. The content of the user input accepted by the input device 104 is transmitted to the processor 101 via the bus 106. The output device 105 is a device that outputs to the user under the control of the processor 101, and includes various output means such as a display and a speaker.

[0030] Referring again to FIG. 1, the XR system unit 21 is software for realizing an XR space in cooperation with the cameras 11a and 11b, the head-mounted display 12, and the controller C1, and is configured to have the function of detecting the position and tilt of the head-mounted display 12 and the controller C1 as described above, as well as the function of acquiring the operation state of an operation switch provided on the surface of the controller C1, and the function of rendering the XR space on the display surface of the head-mounted display 12 based on instructions from the application unit 20.

[0031] The application unit 20 is, for example, an XR-compatible app and has the function of providing the user with various experiences in an XR space through the XR system unit 21. As an example, the application unit 20 is sold as a set together with the XR system unit 21, cameras 11a and 11b, head-mounted display 12, and controller C1. The application unit 20 constructs an XR space based on a program pre-stored in the storage device 102, and renders a 3D object representing the controller C1 in the XR space based on the position and tilt of the controller C1 detected by the XR system unit 21. The application unit 20 also controls the XR space according to the operation states of the operation switches acquired by the XR system unit 21.

[0032] The application unit 20 supplies information about the XR space after control to the XR system unit 21. The XR system unit 21 sets the user's viewpoint within the XR space based on the detected position and tilt of the head mounted display 12, and supplies an image seen from this viewpoint to the head mounted display 12. The head mounted display 12 displays the image thus supplied, allowing the user to have a variety of experiences within the XR space.

[0033] The tracking processing unit 22 is configured to have the function of detecting the position and tilt of the controller C2 as described above, and the function of acquiring the operation state of the operation switches provided on the surface of the controller C2, etc. The data detected or acquired by the tracking processing unit 22 is supplied to the application unit 20 via the device information acquisition unit 23 and the coordinate conversion processing unit 24.

[0034] The device information acquisition unit 23 is software created by a software development kit supplied by the vendor of the application unit 20, and serves to supply data detected or acquired by the tracking processing unit 22 to the application unit 20. The coordinate conversion processing unit 24 is plug-in software for the application unit 20, and serves to convert the position and tilt supplied from the tracking processing unit 22 (position and tilt in coordinate system 2) into a position and tilt in XR space (position and tilt in coordinate system 1). The specific method of this conversion (coordinate conversion formula) is determined by calibration processing performed by the application unit 20 and the device information acquisition unit 23. Details of this calibration processing will be explained in detail later.

[0035] Fig. 4(a) is a diagram showing the external appearance of the controller C2, and Fig. 4(b) is a diagram showing the internal configuration of the controller C2. First, referring to Fig. 4(a), the controller C2 is a device shaped like a pen C2a with a handle C2b attached to the center. As shown in Fig. 2, a user uses the controller C2 while holding the handle C2b.

[0036] As shown in Figure 4(b), the controller C2 is configured to have a control unit 30, a core body 31, a pen tip electrode 32, pressure sensors 33 and 34, a battery 35, a wireless communication unit 36, an operation switch 37, and multiple light-emitting units 38.

[0037] The control unit 30 is a microprocessor that controls the entire controller C2. The core body 31 is a rod-shaped object arranged along the pen shaft of the pen C2a, and its tip forms the pen tip of the pen C2a. The end of the core body 31 abuts against the pressure sensor 33. The pressure sensor 33 serves to acquire the pressure applied to the pen tip through this abutment. The control unit 30 is configured to acquire the pressure acquired by the pressure sensor 33 as writing pressure. The pressure sensor 34 is provided on the surface of the controller C2 and is a sensor that acquires the force with which the user grips the controller C2. The control unit 30 is configured to acquire the force acquired by the pressure sensor 34 as writing pressure. This enables the control unit 30 to acquire writing pressure even when the controller C2 is used without the pen tip being in contact with a hard surface such as a touch surface.

[0038] The pen tip electrode 32 is a conductor arranged near the tip of the core body 31 and is electrically connected to the control unit 30. The control unit 30 is configured to be able to send and receive signals bidirectionally to and from the position detection device 15, or to be able to send signals to the position detection device 15, via this pen tip electrode 32. The position detection device 15 is configured to obtain the position of the controller C2 on the touch surface using the signal received from the controller C2 in this manner, and to sequentially transmit information indicating the obtained position to the computer 10. The signal transmitted from the controller C2 to the position detection device 15 may include a value indicating the writing pressure obtained by the control unit 30 from the pressure sensor 33.

[0039] The control unit 30 is also configured to transmit a value indicating the writing pressure obtained from the pressure sensor 34 to the coupling device 14 via the wireless communication unit 36, in addition to transmitting a signal to the position detection device 15. The coupling device 14 transmits the value indicating the writing pressure obtained from the pressure sensor 34 to the computer 10. This makes it possible to generate writing pressure even when the controller C2 is used in the air.

[0040] The battery 35 serves to supply operating power (including power necessary to cause the light emitting unit 38 to emit light) to the controller C2. The wireless communication unit 36 ​​is a communication device for communicating with the coupling device 14 shown in Fig. 1 by short-range wireless communication such as Bluetooth (registered trademark). The control unit 30 communicates with the coupling device 14 via this wireless communication unit 36.

[0041] The operation switch 37 is a switch configured to be turned on and off by the user, and is arranged, for example, on the surface of the handle C2b, as shown in FIG. 4(a). However, the arrangement location of the operation switch 37 is not particularly limited as long as it is a place where the user can operate it. The tracking processing unit 22 is configured to acquire the operation state of the operation switch 37.

[0042] The light-emitting units 38 are, for example, LEDs (Light Emitting Diodes) that emit light in the infrared range, and are arranged at various locations on the surface of the controller C2, as shown in FIG. 4(a). The light-emitting units 38 constitute the tracker described above, and the tracking processing unit 22 detects the position and tilt of the controller C2 by identifying the positions of at least three or more light-emitting units 38 based on the images captured by the cameras 13a to 13c. The controller C2 may be provided with at least eight light-emitting units 38 so that the tracking processing unit 22 can identify the positions of at least three or more light-emitting units 38 even when part of the controller C2 is hidden by the user's hand and regardless of the angle of the controller C2.

[0043] The calibration process performed by the application unit 20 and the device information acquisition unit 23 will be described in detail below.

[0044] 5 is a flow diagram showing the processing flow of the calibration process performed by the application unit 20 and the device information acquisition unit 23. In this process, first, the application unit 20 receives a calibration execution instruction (step S1). In one example, this instruction is issued in the XR space by the user using the controller C1. Next, the application unit 20 renders a virtual device 40 representing the controller C2 in the XR space defined by the coordinate system 1 (step S2). The position of the virtual device 40 in this rendering in the coordinate system 1 may be determined in advance.

[0045] FIG. 6 is a diagram showing an example of the display of the virtual device 40 rendered in step S2. As an example, the virtual device 40 is displayed on the head-mounted display 12. As shown in the figure, the virtual device 40 has the same external shape as the controller C2. When the XR space is a VR space, the controller C2 shown in the figure is invisible to the user wearing the head-mounted display 12. The user moves the hand holding the controller C2 and, using the sensation of their hand, aligns the controller C2 with the position of the virtual device 40 rendered in the XR space. Then, in that state, they press the operation switch 37.

[0046] Here, it is preferable to set the orientation of the virtual device 40 rendered in step S2 so that the part corresponding to the pen C2a is horizontal or vertical. This makes it easier for the user to align the position compared to when the part corresponding to the pen C2a is at an angle, and it becomes possible to reduce errors in the rotation matrix A, which will be described later.

[0047] Returning to FIG. 5, the device information acquisition unit 23 waits for a predetermined operation by the user while the virtual device 40 is being displayed (step S3). One example of the predetermined operation is pressing the operation switch 37. Upon detecting the predetermined operation, the device information acquisition unit 23 performs a series of processes, such as calculating a coordinate transformation formula in cooperation with the application unit 20 (steps S4 to S6) and setting the calculated coordinate transformation formula in the coordinate transformation processing unit 24 (step S7).

[0048] Specifically, the device information acquisition unit 23 first identifies coordinates VHP in the coordinate system 2 for at least three of the multiple trackers (light-emitting units 38) provided in the controller C2 (step S4). The device information acquisition unit 23 also causes the application unit 20 to identify coordinates VUP in the coordinate system 1 for each position on the virtual device 40 that is the same as each of the at least three trackers (step S5), and acquires the identified coordinates VUP.

[0049] Thereafter, the device information acquisition unit 23 derives a rotation matrix A and a translation vector B by substituting the acquired three coordinates VUP, VHP into the following equation (1) (step S6). Then, a coordinate transformation equation including the derived rotation matrix A and translation vector B is set in the coordinate transformation processing unit 24 (step S7), and the processing ends. Thereafter, the coordinate transformation processing unit 24 performs processing to convert the position and tilt supplied from the tracking processing unit 22 into a position and tilt in the XR space using the set rotation matrix A and translation vector B. VUP = A × VHP + B (1)

[0050] As described above, according to the computer 10 of this embodiment, a user wearing the head-mounted display 12 moves the hand holding the controller C2, aligns the position of the controller C2 with the virtual device 40 displayed in the XR space, and then presses the operation switch 37. In response to the user's operation, the computer 10 can calculate a coordinate transformation formula (specifically, a rotation matrix A and a translation vector B) for transforming coordinates in the coordinate system 2 into coordinates in the coordinate system 1 and set the formula in the coordinate transformation processing unit 24. Therefore, in the XR space constructed using the tracking system T1, it becomes possible to use the controller C2 that is compatible only with the tracking system T2.

[0051] Next, a 3D object rendering system 1 according to a second embodiment of the present invention will be described. This embodiment differs from the first embodiment in that the controller C2 has an inertial measurement unit (IMU), and the tracking system T2 detects the tilt of the controller C2 from the measurement results of this inertial measurement unit. The following will focus on these differences and provide a detailed description.

[0052] FIG. 7 is a diagram showing the internal configuration of the controller C2 according to this embodiment. As can be seen by comparing FIG. 7 with FIG. 4(b), the controller C2 according to this embodiment differs from the controller C2 according to the first embodiment in that it includes an inertial measurement unit 39. The inertial measurement unit 39 is a device incorporating a three-axis gyro and a three-directional accelerometer, and serves to detect the angle and acceleration of the controller C2 in three axes. The tracking processing unit 22 according to this embodiment is configured to detect the tilt of the controller C2 in the coordinate system 2 based on the angle and acceleration measured by the inertial measurement unit 39. Specifically, this tilt is detected in the form of a 3×3 attitude matrix.

[0053] 8 is a flow diagram showing the processing flow of calibration processing performed by application unit 20 and device information acquisition unit 23 according to the present embodiment. First, as in the first embodiment, application unit 20 accepts a calibration execution instruction (step S10). Upon accepting the calibration execution instruction, application unit 20 renders, in the XR space, a virtual device 40 representing controller C2 as in the first embodiment, and also renders tilt display information 41 indicating the tilt of virtual device 40 in coordinate system 1 (step S11).

[0054] 9 is a diagram showing the virtual device 40 and tilt display information 41 rendered in step S11. The tilt display information 41 is rendered in the XR space in the shape of a cube arranged so as to surround the virtual device 40. The size of the tilt display information 41 is preferably set to a size that is sufficiently larger than the virtual device 40, and that is within the user's field of vision.

[0055] Returning to Fig. 8, the device information acquisition unit 23 then acquires the attitude matrix of the controller C2 based on the measurement results of the inertial measurement unit 39 (step S12). Then, the application unit 20 renders tilt display information 42 in the XR space based on the attitude matrix acquired by the device information acquisition unit 23 (step S13).

[0056] Referring again to FIG. 9, the tilt display information 42 is also shown. In the figure, the tilt display information 42 is depicted with a dashed line for convenience, but the actual tilt display information 42 is displayed so that it can be seen by the user in the XR space. The application unit 20 renders the tilt display information 42 based on the position of the virtual device 40 in the XR space. This allows the user to align the tilt of the controller C2 with the tilt of the virtual device 40 by matching the tilt display information 42 with the tilt display information 41. This allows the positioning of the controller C2 and the virtual device 40 to be performed with higher accuracy than when positioning is performed solely based on the virtual device 40.

[0057] Returning to Fig. 8, the device information acquisition unit 23 waits for a predetermined operation by the user while the virtual device 40 is being displayed (step S14). The predetermined operation may be the pressing of the operation switch 37, as in the first embodiment. When the device information acquisition unit 23 detects the predetermined operation, it performs a series of processes, such as calculating a coordinate transformation formula in cooperation with the application unit 20 (steps S15 to S20) and setting the calculated coordinate transformation formula in the coordinate transformation processing unit 24 (step S21).

[0058] Specifically, the device information acquisition unit 23 first acquires an attitude matrix VHR of the controller C2 based on the measurement results of the inertial measurement unit 39 (step S15), and also acquires an attitude matrix VUR of the virtual device 40 from the application unit 20 (step S16). Then, the device information acquisition unit 23 derives a rotation matrix A by substituting the two acquired attitude matrices into the following equation (2) (step S17). VUR = A × VHR (2)

[0059] Next, the device information acquisition unit 23 identifies coordinates VHP in coordinate system 2 for at least one of the multiple points (light-emitting units 38) provided on the controller C2 (step S18). The device information acquisition unit 23 also causes the application unit 20 to identify coordinates VUP in coordinate system 1 for the same point on the virtual device 40 (step S19), and acquires the identified coordinates VUP. Then, the device information acquisition unit 23 derives a translation vector B by substituting the acquired coordinates VHP, VUP and the rotation matrix A derived in step S17 into the above-mentioned equation (1) (step S20).

[0060] Thereafter, the device information acquisition unit 23 sets the derived rotation matrix A and translation vector B as coordinate transformation formulas in the coordinate transformation processing unit 24 (step S21), and ends the processing. Thereafter, the coordinate transformation processing unit 24 performs processing to convert the position and tilt supplied from the tracking processing unit 22 into a position and tilt in the XR space using the set rotation matrix A and translation vector B.

[0061] As described above, according to the computer 10 of this embodiment, the tilt display information 41, 42 is rendered in the XR space, and therefore the user can align the positions of the controller C2 and the virtual device 40 with higher accuracy than in the first embodiment. Therefore, it becomes possible to calculate the coordinate transformation formula (specifically, the rotation matrix A and the translation vector B) for transforming the coordinates in the coordinate system 2 into the coordinates in the coordinate system 1 with higher accuracy and set it in the coordinate transformation processing unit 24.

[0062] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and it goes without saying that the present invention can be embodied in various forms without departing from the spirit of the present invention.

[0063] For example, in the above embodiment, a 3D object rendering system 1 was described that uses a tracking system that detects 3D objects based on camera images, but the present invention can also be applied to a 3D object rendering system that uses a tracking system that uses a laser, for example. [Explanation of symbols]

[0064] 1. 3D Object Rendering System 10. Computers 11a, 11b Camera 12 Head-mounted display 13a~13c Camera 14 Coupling device 15 Position detection device 20 Application Section 21 XR Systems Department 22 Tracking processing section 23 Device information acquisition unit 24 Coordinate transformation processing section 30 Tracking System 30 Control Unit 31 Core body 32 Pen tip electrode 33,34 Pressure sensors 35 Batteries 36 Radio Communication Department 37 Operation switch 38 Light-emitting part (tracker) 39 Inertial Measurement Unit 40 Virtual Devices 41,42 Tilt display information 101 processors 102 Storage device 103 Communication equipment 104 Input Device 105 Output Device Bus 106 C1, C2 controller C2a Pen C2b Handle T1, T2 tracking system

Claims

1. a computer having a processor, The processor: calculating first coordinates of a 3D object to be rendered in XR space in a first coordinate system that defines coordinates in the XR space of a first device tracked by a first tracking system; calculating second coordinates of a second device in a second coordinate system different from the first coordinate system that defines coordinates in the XR space of a second device different from the first device that is tracked by a second tracking system different from the first tracking system; calculating a coordinate transformation formula for transforming the coordinates of the second coordinate system into the coordinates of the first coordinate system based on the first coordinates and the second coordinates in response to a predetermined operation by a user; A computer characterized by:

2. the 3D object is a virtual device corresponding to the second device and is displayed on a display; The computer of claim 1.

3. the predetermined operation is an operation on an operation switch provided on the second device; The computer of claim 1.

4. The processor: calculates coordinates in the second coordinate system for at least three of a plurality of trackers provided on the second device in response to the predetermined operation by the user, calculates coordinates in the first coordinate system for each position on the 3D object corresponding to each position of the at least three trackers, and calculates the coordinate transformation formula by deriving a translation vector and a rotation matrix based on the calculated coordinates; A computer according to any one of claims 1 to 3.

5. The processor: controlling a display to display, together with the 3D object, first information indicating a tilt of the 3D object in the first coordinate system and second information indicating a tilt of the second device in the second coordinate system, which is indicated by a measurement result of an inertial measurement unit included in the second device, in the XR space; A computer according to any one of claims 1 to 3.

6. The processor: acquiring a first matrix indicating an orientation of the second device in the second coordinate system from an inertial measurement unit included in the second device, and acquiring a second matrix indicating an orientation of the 3D object in the first coordinate system, and deriving a rotation matrix based on the acquired first matrix and second matrix; further specifying coordinates in the second coordinate system for one of a plurality of points provided on the second device, and specifying coordinates in the first coordinate system for a point of the 3D object corresponding to the one of the plurality of points provided on the second device, and calculating the coordinate transformation formula by deriving a translation vector based on the specified coordinates and the rotation matrix. A computer according to any one of claims 1 to 3.

7. calculating first coordinates of a 3D object to be rendered in XR space in a first coordinate system that defines coordinates in the XR space of a first device tracked by a first tracking system; calculating second coordinates of a second device in a second coordinate system different from the first coordinate system that defines coordinates in the XR space of a second device different from the first device that is tracked by a second tracking system different from the first tracking system; calculating a coordinate transformation formula for transforming coordinates in the second coordinate system into coordinates in the first coordinate system based on the first coordinates and the second coordinates in response to a predetermined operation by a user; A method comprising:

8. calculating first coordinates of a 3D object to be rendered in XR space in a first coordinate system that defines coordinates in the XR space of a first device tracked by a first tracking system; calculating second coordinates of a second device in a second coordinate system different from the first coordinate system that defines coordinates in the XR space of a second device different from the first device that is tracked by a second tracking system different from the first tracking system; calculating a coordinate transformation formula for transforming coordinates in the second coordinate system into coordinates in the first coordinate system based on the first coordinates and the second coordinates in response to a predetermined operation by a user; A program that causes a computer to execute the following.

Images