Information processing systems, information processing methods, and computer programs

Abstract

This system provides an information processing system that utilizes a common reference coordinate system across multiple terminals. [Solution] The information processing system includes a communication unit that communicates between a first terminal and a second terminal; a first acquisition unit that acquires first image information from a first imaging device installed at the first terminal; a first reference coordinate system determination unit that determines a first reference coordinate system at the first terminal; a first position and orientation information calculation unit that calculates first position and orientation information representing the position and orientation of the first terminal using the first image information and the first reference coordinate system; a second acquisition unit that acquires second image information from a second imaging device installed at the second terminal; a second position and orientation information calculation unit that calculates second position and orientation information representing the position and orientation of the second terminal using the second image information; a relative position and orientation information calculation unit that calculates relative position and orientation information representing the relative position and orientation of the first terminal and the second terminal; and a second position and orientation information correction unit that corrects the second position and orientation information to the first reference coordinate system.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, a computer program, etc.

Background Art

[0002] There is known a technology of MR (Mixed Reality) that superimposes and displays virtual objects (CG or characters) on an image captured by a camera. In MR, a technology of estimating the position and orientation of a terminal (an information terminal such as an HMD or a smartphone) based on video information and calculating the drawing position of a virtual object according to the movement of the terminal is used.

[0003] In order to draw a virtual object, a coordinate system serving as an origin somewhere in a three-dimensional space (hereinafter referred to as a reference coordinate system) is required. In many cases, in an MR application using a marker, the reference coordinate system is set based on the position of the marker. <​​​​​​​​​​​​​​​​​C. Campos, R. Elvira, JJG Rodríguez, JM Montiel, and JD Tardos, ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM, IEEE Transactions on Robotics 37(6), Dec. 2021. [Non-Patent Document 2] Parallel Tracking and Mapping for Small AR Workspaces, Klein.et.al, ISMAR2007 [Non-Patent Document 3] Tateno, Kotake, Uchiyama, "A High-Precision, Highly Stable Model Fitting Method for Maximum Likelihood Integration of Distance and Grayscale Images for Bin Picking of Real Parts," Proceedings of the 13th Symposium on Image Recognition and Understanding (MIRU2010), OS5-1, 2010. [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] However, with conventional technology, if multiple users want to experience MR without using markers and each user has their own device, setting a separate reference coordinate system for each device will result in virtual objects being displayed at different locations in physical space on each device.

[0008] This invention was made in view of the above-mentioned problems, and one of its objectives is to provide an information processing system that allows multiple terminals to use a common reference coordinate system.

[0009] [Means for solving the problem]

[0010] One embodiment of the present invention is, An information processing system for setting a coordinate system shared by multiple terminals, A communication means for communicating between a first terminal and a second terminal, A first acquisition means for acquiring first image information from a first imaging device installed in the first terminal, A first reference coordinate system determination means for determining a first reference coordinate system in the first terminal based on the first image information, A first position and orientation information calculation means calculates first position and orientation information representing the position and orientation of the first terminal using the first image information and the first reference coordinate system, A second acquisition means for acquiring second image information from a second imaging device provided in the second terminal, A second position and orientation information calculation means that calculates second position and orientation information representing the position and orientation of the second terminal using the second image information, Relative position and orientation information calculation means calculates relative position and orientation information representing the relative position and orientation of the first terminal and the second terminal based on the second image information and the second position and orientation information, A second position and attitude information correction means that corrects the second position and attitude information to the first reference coordinate system using the first position and attitude information, the second position and attitude information, and the relative position and attitude information, It is characterized by having the following features. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide an information processing system that allows multiple terminals to use a common reference coordinate system. [Brief explanation of the drawing]

[0012] [Figure 1] This figure illustrates an example of a situation and coordinate relationship when using the information processing system 1 of Embodiment 1 of the present invention. [Figure 2] This is a block diagram showing an example of the hardware configuration of the information processing system 1 of Embodiment 1. [Figure 3] This is a functional block diagram illustrating an example configuration of the information processing system 1 of Embodiment 1. [Figure 4]It is a flowchart showing a processing example of an information processing method using the information processing system 1 of Embodiment 1. [Figure 5] It is a functional block diagram showing a configuration example of the information processing system of Embodiment 2. [Figure 6] It is a flowchart showing a processing example of an information processing method using the information processing system of Embodiment 2. [Figure 7] It is a functional block diagram showing a configuration example of the information processing system of Embodiment 3. [Figure 8A] It is a flowchart showing a processing example of an information processing method using the information processing system of Embodiment 3. [Figure 8B] It is a flowchart showing the continuation of FIG. 8A. [Figure 8C] It is a flowchart showing the continuation of FIG. 8B [Figure 9] It is a diagram showing an example of a GUI displayed by the information processing system of Embodiment 3.

Mode for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each figure, the same members or elements are given the same reference numerals, and duplicate explanations are omitted or simplified.

[0014] <Embodiment 1> In this embodiment, a use case in which two terminals share one reference coordinate system will be described. FIG. 1 is a diagram for explaining an example of the situation and coordinate relationship when using the information processing system 1 of Embodiment 1 of the present invention, and shows the relationship between two users and two terminals. Note that the information processing system of this embodiment is configured so that a shared coordinate system can be set with a plurality of terminals.

[0015] In Figure 1, U10 and U20 are users who are about to experience MR. T11 and T21 are terminals worn by users U10 and U20, respectively, and in this embodiment, the terminals are HMDs (Head Mount Displays). Terminals T11 and T21 correspond to the first terminal and the second terminal, respectively.

[0016] C12 and C22 are imaging devices, i.e., cameras, mounted on terminals T11 and T21, respectively, which calculate the position and orientation information of each terminal using SLAM technology from the surrounding scenery. D13 and D23 are displays mounted on terminals T11 and T21, respectively, and the user views the virtual objects displayed on the displays. X31 is the reference coordinate system (first reference coordinate system) set on terminal T11, and is the origin for displaying the virtual objects.

[0017] In this embodiment, in order for terminal T21 to draw the same virtual object as terminal T11, the position and orientation information of terminal T21 at time t when camera C22 captured the image (Mw2(t) in the figure) is calculated relative to the reference coordinate system X31.

[0018] In this embodiment, the position and orientation represent six degrees of freedom that indicate the geometric relationship between two objects, and are represented, for example, by a 4x4 matrix.

[0019] Since terminals T11 and T21 are independent devices, terminal T21 can use the reference coordinate system X32 (second reference coordinate system) that terminal T21 uses as its reference, but it cannot use the reference coordinate system X31 of terminal T11.

[0020] In other words, terminal T21 can only calculate position and orientation information (Mv2(t) in the figure) with respect to the reference coordinate system X32. However, in this embodiment, terminal T21 holds relative position and orientation information (Mwv in the figure) between the reference coordinate system X31 and terminal T21's reference coordinate system X32, and calculates Mw2(t) by matrix operation as shown in Equation 1 below.

number

[0021] U20' represents the position of user U20 at a certain time s. Here, Mw1(s) is the position and orientation information of terminal T11 in the first reference coordinate system X31 observed at time s, Mv2(s) is the position and orientation information of terminal T21 in the second reference coordinate system X32, and M12(s) is the relative position and orientation information of terminal T11 and terminal T21. In this case, Mwv can be calculated by the matrix operation shown in Equation 2 below. Note that the subscript -1 in Equation 2 indicates the inverse matrix.

number

[0022] In this embodiment, a common reference coordinate system is used across multiple different terminals by performing calculations as shown in Equations 1 and 2 above. Furthermore, information for commonizing the reference coordinate system is transmitted and received between each terminal via a communication unit.

[0023] Furthermore, as mentioned above, the position and orientation information in this embodiment is information representing the six degrees of freedom that indicate the geometric relationship between two objects. When referring to the position and orientation information of a terminal, it is sufficient to know the terminal's position and orientation in space. In this embodiment, this refers to the position and orientation information of the mounted camera.

[0024] However, it is not limited to that; for example, it could be the position and orientation information of the terminal's center of gravity, or the position and orientation information of the center of the display (these can be converted to each other if the coordinate relationships within the terminal are known). Time s and time t are the times when the terminal's imaging device (camera) captured the image information, and in this embodiment, UNIX time is used.

[0025] Furthermore, unless otherwise specified, time t represents the time when image information was captured, i.e., the time when the position and orientation of the terminal are calculated. Time s represents the time used to align the reference coordinate systems of the first terminal (terminal T11) and the second terminal (terminal T21).

[0026] In other words, this means using image information captured at the same time by the first and second terminals. Note that the time s does not refer to a time that is exactly the same, but allows for a slight margin of error.

[0027] Furthermore, the shape model in this embodiment is described in SLT (Standard Tessellation Language) file format. SLT is a file composed of linked triangles that describes the surface geometry of 3D models and objects. However, the file format is not limited to this.

[0028] Figure 2 is a block diagram showing an example of the hardware configuration of the information processing system 1 of Embodiment 1. 211 is the CPU of the computer, which controls various devices connected to the system bus 219. 212 is ROM, which stores the BIOS program and boot program.

[0029] 213 is RAM, used as the main memory of the CPU 211. 214 is external memory, which stores the computer program that operates the information processing system 1. 215 is an input unit that receives input from a keyboard, mouse, or touch panel, and performs processing related to the input of information, etc.

[0030] 216 is a display unit that outputs video information to the display installed in the terminal. 217 is a communication interface that communicates information with other terminals. 218 is an I / O that acquires video from the camera installed in the terminal. In this embodiment, there are assumed to be two terminals, each holding the hardware shown in Figure 3, and sending and receiving information via the communication interface 217.

[0031] Figure 3 is a functional block diagram illustrating an example configuration of the information processing system 1 of Embodiment 1, which includes a first terminal T11 and a second terminal T21.

[0032] Furthermore, some of the functional blocks shown in Figure 3 are realized by having the CPU 211 and other components of the information processing system execute computer programs stored in memory, which serves as a storage medium.

[0033] However, some or all of these can be implemented in hardware. Hardware options include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs).

[0034] Furthermore, each functional block shown in Figure 3 does not necessarily have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths. Also, at least one part of the functions of the information processing system 1 may be provided on an external server or the like. The above explanation regarding Figure 3 also applies to Figures 5 and 7.

[0035] The first imaging device 110 and the second imaging device 120 are cameras, corresponding to C12 and C22 in Figure 1, respectively. The information processing system 1 uses the first image information and the second image information captured by the first imaging device 110 and the second imaging device 120, respectively, to calculate position and orientation information in a common reference coordinate system for the first terminal T11 and the second terminal T21.

[0036] In this embodiment, we describe a configuration in which the second terminal T21 corrects the second terminal position and orientation information (Mv2(t)) in the second reference coordinate system X32 to the position and orientation information (Mw2(t)) in the first reference coordinate system X31. In this embodiment, the first terminal T11 and the second terminal T21 are assumed to have the same shape, and each terminal holds its own shape model in a holding part not shown.

[0037] The first acquisition unit 111 is mounted on the first terminal T11, receives first image information from the first imaging device 110, and outputs the input first image information to the first reference coordinate system determination unit 112 and the first position and orientation information calculation unit 113. The first acquisition unit 111 functions as a first acquisition means for acquiring first image information from the first imaging device installed on the first terminal.

[0038] The first reference coordinate system determination unit 112 is mounted on the first terminal T11 and determines the first reference coordinate system X31, which represents the origin in the calculation of position and orientation, based on the first image information input from the first acquisition unit 111.

[0039] The first reference coordinate system determination unit 112 outputs the first reference coordinate system X31 to the first position and orientation information calculation unit 113. The first reference coordinate system determination unit 112 functions as a first reference coordinate system determination means that determines the first reference coordinate system at the first terminal based on the first image information.

[0040] The first position and orientation information calculation unit 113 is mounted on the first terminal T11 and uses the first reference coordinate system X31 determined by the first reference coordinate system determination unit 112 to calculate the position and orientation information of the first terminal relative to the first reference coordinate system X31 (first position and orientation information).

[0041] Furthermore, the first position and orientation information calculation unit 113 functions as a first position and orientation information calculation means that calculates first position and orientation information representing the position and orientation of the first terminal using first image information and a first reference coordinate system.

[0042] The first position and attitude information is Mw1(s) as explained in Figure 1. The first position and attitude information calculation unit 113 outputs the calculated first position and attitude information Mw1(s) to the communication unit 131.

[0043] The communication unit 131 is installed in the first terminal T11 and the second terminal T21 respectively, and is a wireless network module that serves as a means of communication between the first terminal T11 and the second terminal T21. The communication unit 131 transmits the first position and attitude information Mw1(s) calculated by the first position and attitude information calculation unit 113 to the second terminal via the wireless network.

[0044] Meanwhile, the second acquisition unit 121 receives image information captured by the second imaging device 120. The second acquisition unit 121 inputs the received image to the second position and orientation information calculation unit 122 and the relative position and orientation information calculation unit 123. The second acquisition unit 121 functions as a second acquisition means for acquiring second image information from the second imaging device installed at the second terminal.

[0045] The second position and orientation information calculation unit 122 calculates the position and orientation information of the second terminal T21 using the second image information input by the second acquisition unit 121. In this embodiment, the position and orientation information of the second terminal (second position and orientation information) is calculated in a reference coordinate system where the coordinates at the time the second terminal was started are the origin, and this reference coordinate system is used as the reference coordinate system for the second terminal T21.

[0046] Furthermore, the second position and orientation information calculation unit 122 functions as a second position and orientation information calculation means that calculates second position and orientation information representing the position and orientation of the second terminal using the second image information.

[0047] As shown in Figure 1, the second position and attitude information calculated at time s is denoted as Mv2(s) (and the second position and attitude information calculated at time t is denoted as Mv2(t)). The second position and attitude information calculation unit 122 outputs the calculated second position and attitude information to the second position and attitude information correction unit 124.

[0048] The relative position and orientation information calculation unit 123 calculates the relative position and orientation information of the first terminal T11 and the second terminal T21 based on the image of the first terminal shown in the second image information input from the second acquisition unit 121.

[0049] Furthermore, the relative position and orientation information calculation unit 123 functions as a relative position and orientation information calculation means that calculates relative position and orientation information representing the relative position and orientation of the first terminal and the second terminal based on the second image information and the second position and orientation information.

[0050] The relative position and attitude information referred to here is M12(s) as shown in Figure 1. The relative position and attitude information calculation unit 123 outputs the calculated relative position and attitude information M12(s) to the second position and attitude information correction unit 124.

[0051] The second position and orientation information correction unit 124 calculates the second position and orientation information in the first reference coordinate system X31 using the first position and orientation information Mw1(s), the second position and orientation information Mv2(s), and the relative position and orientation information M12(s) of the first terminal T11. The second position and orientation information corresponds to Mw2(t) in Figure 1.

[0052] Furthermore, the second position and attitude information correction unit 124 functions as a second position and attitude information correction means that corrects the second position and attitude information to the first reference coordinate system using the first position and attitude information, the second position and attitude information, and the relative position and attitude information.

[0053] Figure 4 is a flowchart showing an example of the information processing method using the information processing system 1 of Embodiment 1. The CPUs and other components of the computers located inside the first and second terminals execute computer programs stored in memory, sequentially performing the operations of each step in the flowchart of Figure 6. Furthermore, this embodiment of the information processing method sets up a coordinate system shared by multiple terminals.

[0054] Step S410 is a process performed at the first terminal T11, and step S420 is a process performed at the second terminal T21. Each process in step S410 starts when the power to the first terminal T11 is turned on. Similarly, step S420 starts when the power to the second terminal T21 is turned on.

[0055] Furthermore, the processing steps shown in the flowchart in the following explanation are not limited to the examples shown in the flowchart. Also, each process in the flowchart can be individually extracted and function as a single functional element, and can be used in combination with processes other than those shown.

[0056] In step S1101, the system initialization of the module related to the first terminal T11 of the information processing system 1 is performed. That is, the CPU 211 reads a program from the external memory 214 and makes terminal T11 operational.

[0057] Additionally, camera parameters and communication profiles are loaded from external memory 214 into RAM 213 as needed. Once the series of initialization processes is complete, the process moves to step S1102.

[0058] In step S1102, the first acquisition unit 111 acquires the first image captured by the first imaging device 110. Here, step S1102 functions as a first acquisition step in which first image information is acquired from the first imaging device installed at the first terminal.

[0059] The acquired first image is then output to the first reference coordinate system determination unit 112 and the first position and orientation information calculation unit 113, and the process moves to step S1103.

[0060] In step S1103, the first reference coordinate system determination unit 112 detects a plane from the first image information and determines the first reference coordinate system X31 with an arbitrary coordinate on the plane as the origin. Here, step S1103 functions as a first reference coordinate system determination step that determines the first reference coordinate system at the first terminal based on the first image information.

[0061] Furthermore, the method for determining the reference coordinates can be the method described in Non-Patent Document 2. Once the first reference coordinate system X31 is set, proceed to step S1104.

[0062] In step S1104, the first position and orientation information calculation unit 113 calculates first position and orientation information Mw1(s) with respect to the first reference coordinate system X31. Here, step S1104 functions as a first position and orientation information calculation step that calculates first position and orientation information representing the position and orientation of the first terminal using the first image information and the reference coordinate system.

[0063] SLAM is used to calculate position and orientation. Specifically, feature points are detected from the input image information, and three-dimensional map information of the feature points relative to the first reference coordinate system X31 is created. For example, the first position and orientation information calculation unit 113 has a map information holding means for holding map information for calculating the second position and orientation information.

[0064] Feature points are singular points, such as corners, in an image, and are indicators that allow us to determine the correspondence between images. Three-dimensional map information refers to map data that stores the three-dimensional locations of feature points.

[0065] The position and orientation in which the appearance of the map information and the feature points detected from the first image information best match, i.e., the position and orientation in which the reprojection error is minimized, is calculated. Such a SLAM configuration is described in Non-Patent Document 2 above, and we will use that as a reference. Once the first position and orientation information Mw1(s) is calculated, we proceed to step S1105.

[0066] In step S1105, the communication unit 131 transmits the first position and attitude information Mw1(s) for example, via a wireless network. That is, the communication means transmits the first position and attitude information from the first terminal to the second terminal. Here, step S1105 functions as a communication step that performs communication between the first terminal and the second terminal.

[0067] In this embodiment, the first position and orientation information Mw1(s) is broadcast to a terminal near the first terminal T11. Once the transmission is complete, the process proceeds to step S1106.

[0068] In step S1106, it is determined whether or not the calculation of the first position and orientation information is completed. That is, a determination unit (not shown) determines whether or not the calculation of the first position and orientation information Mw1(s) at the first terminal T11 is completed.

[0069] If it is determined via an input unit (not shown) that the user has entered an exit flag to end their use of the HMD, the processing flow shown in Figure 4 is terminated; otherwise, the process returns to step S1104 and the calculation of the first position and orientation information Mw1(s) is repeated.

[0070] In step S1201, the system of the module related to the second terminal T21 of the information processing system 1 is initialized. Similar to step S1101, the CPU 211 reads a program from the external memory 214 and makes terminal T21 operational. In addition, camera parameters and communication profiles are read from the external memory 214 into RAM 213 as needed. Once the series of initialization processes is complete, the process moves to step S1202.

[0071] In step S1202, the second acquisition unit 121 acquires the second image information captured by the second imaging device 120. Here, step S1202 functions as a second acquisition step in which the second image information is acquired from the second imaging device installed at the second terminal. The acquired second image information is then input to the second position and orientation information calculation unit 122 and the relative position and orientation information calculation unit 123, and the process moves to step S1203.

[0072] In step S1203, the second position and orientation information calculation unit 122 calculates the second position and orientation information in the second reference coordinate system X32. That is, it calculates Mv2(s) in Figure 1. Here, step S1203 functions as a second position and orientation information calculation step that calculates second position and orientation information representing the position and orientation of the second terminal using the second image information. In this embodiment, the second reference coordinate system X32 is set with the position and orientation of the second terminal T21 at the time it was started as the origin.

[0073] Furthermore, s is the time when the second image information was captured. SLAM is used to calculate the position and orientation information, and the method described in Non-Patent Literature 2 is used. In this embodiment, the map information used in the aforementioned SLAM is generated individually for each terminal, specifically for the first terminal T11 and the second terminal T21. That is, map information created in different reference coordinate systems is used. After calculating the second position and orientation information Mv2(s), the process moves to step S1204.

[0074] In step S1204, it is determined whether the corrected second position and attitude information has been received. That is, the second position and attitude information calculation unit 122 determines whether the second position and attitude information Mv2(s) has been corrected to match the first reference coordinate system X31.

[0075] Specifically, if the relative position and orientation information Mwv between the reference coordinate system X31 of the first terminal T11 and the reference coordinate system X32 of the second terminal T21 has been calculated (as described later), it is determined that the second position and orientation information Mv2(s) has been corrected to match the first reference coordinate system X31. Then the process proceeds to step S1209; otherwise, it proceeds to step S1205.

[0076] In step S1205, the relative position and orientation information calculation unit 123 calculates relative position and orientation information M12(s) for the first terminal T11 and the second terminal T21. Here, step S1205 functions as a relative position and orientation information calculation step that calculates relative position and orientation information representing the relative position and orientation of the first terminal and the second terminal based on the second image information and the second position and orientation information.

[0077] The relative position and orientation information M12(s) is calculated by model fitting the second image information with the shape model (CAD data) of the first terminal T11 held by a holding part (not shown) within the second terminal.

[0078] Furthermore, the method for estimating the position and orientation of an object based on model fitting can be the method described in Non-Patent Document 3.

[0079] After calculating the relative position and attitude information M12(s) between the first terminal T11 and the second terminal T21 in this manner, the process moves to step S1206. In step S1206, the first position and attitude information Mw1(s) is obtained from the first terminal T11 via the communication unit 131. After that, the process moves to step S1207.

[0080] In step S1207, the second position and attitude information is corrected. Specifically, the second position and attitude information correction unit 124 calculates the relative position and attitude information Mwv between the first reference coordinate system X31 and the second reference coordinate system X32. In other words, in step S1207, the relative position and attitude information calculation means calculates the relative position and attitude information using the first position and attitude information received by the communication means.

[0081] Specifically, using Mv2(s) and M12(s) calculated by the second terminal and Mw1(s) received from the first terminal, the relative position and attitude information Mwv between the reference coordinate systems is calculated using Equation 2 mentioned above. Then, the second position and attitude information Mw2(t) of the first reference coordinate system X31 is calculated using Equation 1 to correct the second position and attitude information. After that, the process moves to step S1208.

[0082] Here, step S1207 functions as a second position and attitude information correction step, which corrects the second position and attitude information to the first reference coordinate system using the first position and attitude information, the second position and attitude information, and the relative position and attitude information.

[0083] In step S1208, a determination unit (not shown) determines whether or not to terminate the calculation (correction) of the second position and orientation information. If the user performs an operation to end the use of terminal T21 via an input unit (not shown) and the termination flag is entered, the processing flow in Figure 4 is terminated; otherwise, the process returns to S1202 and the calculation of the position and orientation of the second terminal T21 is repeated.

[0084] Meanwhile, in step S1209, the second position and attitude information in the first reference coordinate system X31 is calculated (corrected). That is, the second position and attitude information calculation unit 122 converts the second position and attitude Mv2(t), which is the second position and attitude calculated by the second position and attitude information calculation unit 122 at a certain time t, into the second position and attitude information Mw2(t) in the first reference coordinate system X31 using the relative position and attitude information Mwv between the reference coordinate systems.

[0085] In other words, the second position and attitude information is corrected by calculating Mw2(t) using the aforementioned Equation 1.

[0086] In this way, the position and orientation information calculated by the first terminal is transmitted to the second terminal via the communication unit. Then, by using the transmitted information in the second terminal to perform coordinate transformations to correct the reference coordinate system, it is possible to calculate the position and orientation in a reference coordinate system common to the first terminal, even if the terminals are different. Therefore, it is possible to display a virtual object at the same position in physical space on multiple terminals.

[0087] <Modified form of Embodiment 1> In Embodiment 1, the communication unit 131 broadcast the first position and orientation information Mw1(s) from the first terminal T11. However, the first terminal and the second terminal may establish a connection in advance and perform bidirectional communication such as TCP / IP. This would allow for error correction and retransmission of communication errors, enabling a more stable commonization of the reference coordinate system.

[0088] Furthermore, although the communication unit 131 has been described as having a configuration in which the first terminal T11 and the second terminal T21 directly send and receive information, information may also be sent and received via a router or another server (not shown). Alternatively, the communication unit 131 may send the necessary information to a server, and the server may calculate the reference coordinate system and send it to each terminal. In this way, it becomes unnecessary for each terminal to perform a common reference coordinate system processing, and a common reference coordinate system can be used.

[0089] In the above embodiment 1, the communication unit 131 transmits the first position and attitude information Mw1(s). However, the communication content is not limited to position and attitude information; additional information for calculating the reference coordinate system with high precision may also be transmitted.

[0090] For example, the time information of the image used to calculate the position and orientation information may also be transmitted. In this way, the second terminal T21 can select from the received time series of first position and orientation information group Mw1(r) [time r refers to multiple time points] the position and orientation information of the image that is close to the time s of the image used to calculate the relative position and orientation information M12(s) of the first and second terminals.

[0091] In other words, by using data where r≈ s, the reference coordinate system can be standardized using terminal position and orientation information from nearby times, allowing for a more accurate standardization of the reference coordinate system.

[0092] Furthermore, a signal may be transmitted to synchronize the time information of both terminals. That is, the communication means may further transmit the time information of the first terminal and the second terminal to synchronize the time of the first terminal and the second terminal. For example, the first terminal may internally maintain an NTP server, and the second terminal may synchronize its time via the communication unit 131.

[0093] Furthermore, the first acquisition means and the second acquisition means may also input the time at which the first image information and the second image information were captured, respectively, in association with the first image information and the second image information. In addition, the first position and orientation information calculation means, the second position and orientation information calculation means, and the relative position and orientation information may be stored in association with the time at which the first image information or the second image information was captured.

[0094] By doing so, the positional information used to unify the reference coordinate system can be selected using more accurately synchronized time information, thus enabling a more accurate unification of the reference coordinate system.

[0095] Furthermore, the first position and orientation information calculation means, the second position and orientation information calculation means, and the relative position and orientation information calculation means may calculate the first position and orientation information, the second position and orientation information, and the relative position and orientation information, respectively, for each image at each time point.

[0096] In other words, the first positional information, the second positional information, and the relative positional information may be calculated for each image taken at each time point using the first or second image information captured at multiple times.

[0097] Furthermore, the second position and attitude information correction means may select from the first position and attitude information, the second position and attitude information, and the relative position and attitude information calculated at multiple time points, each of which has a difference of less than or equal to a predetermined value in its associated shooting time. Based on the selected position and attitude information, the second position and attitude information may then be corrected to the first reference coordinate system.

[0098] Furthermore, this embodiment assumes that the two terminals (HMDs) have the same shape. However, even if the shapes are different, it is possible to handle this by pre-registering the shape model of the terminal to be used for commonizing the reference coordinate system in a holding unit (not shown), or by receiving it from the internet or another server or terminal (not shown).

[0099] Furthermore, if each terminal holds a shape model of its own terminal, the communication unit 131 may transmit those shape models. That is, the communication means may transmit the shape model of the first terminal held by the first terminal from the first terminal to the second terminal.

[0100] For example, the relative position and attitude information calculation means provided in the second terminal may calculate the relative position and attitude information by performing model fitting using the shape model of the first terminal received through the communication means. In this way, even if the shape model of another terminal calculated from an image is not held, a shape model can be obtained from another terminal via the communication unit 131.

[0101] Furthermore, in the above embodiment, the first reference coordinate system determination unit 112 detected a plane and set a reference coordinate system on that plane. However, the method is not limited to the above method, as long as it is possible to determine the origin in space. That is, the coordinates at which the terminal was activated may be used as the reference coordinate system, or an object with a predetermined shape or pattern may be detected from image information and used as the reference coordinate system. Alternatively, a point with a predetermined offset from these may be used as the reference coordinate system.

[0102] Furthermore, although this embodiment describes a configuration in which the second reference coordinate system X32 is calculated at the second terminal as well, it can be implemented without this calculation. That is, the second terminal may receive information for setting the first reference coordinate system X31 from the first terminal T11 via the communication unit 131, and then calculate the second position and orientation information for that reference coordinate system.

[0103] Furthermore, in this embodiment, the second position and orientation information Mv2(t) in the second reference coordinate system X32 was converted to the second position and orientation information Mw2(t) in the first reference coordinate system X31 using Equation 1. However, it can also be implemented with a configuration that corrects the coordinates of the SLAM map instead of correcting the position and orientation information.

[0104] In other words, the map information may be rewritten by multiplying the coordinates of each element of the map information, such as feature points, by Mvw (the inverse matrix of Mwv) in order to correct the relative position and orientation information Mwv between reference coordinate systems calculated in step S1207.

[0105] In other words, when the second position and orientation information correction means corrects the second position and orientation information to the first reference coordinate system, it may also correct the coordinates of each element of the map information so that they conform to the first reference coordinate system.

[0106] In this way, the coordinates calculated in step S1203 become the reference coordinate system of the first terminal, so the reference coordinate system can be standardized without the processing in step S1209, i.e., without the calculation in Equation 1.

[0107] Furthermore, although the imaging device in this embodiment was a camera, the camera may be a camera capable of acquiring color images using RGB color filters, or a camera that acquires grayscale images. In other words, it is sufficient to obtain an image that can calculate the position and orientation information of the terminal, and for example, a depth camera that acquires depth may also be used.

[0108] Furthermore, instead of a camera, a sensor device that acquires the shape of the surroundings, such as LiDAR (Light Detection And Ranging), could also be used.

[0109] Furthermore, while the shape model was created using the STL format, if positional information can be calculated from image data through model fitting, other formats such as PLY (Polygon File Format), like point clouds, may also be used.

[0110] Furthermore, the relative position and orientation information calculation unit 123 calculated the relative position and orientation information M12(s) using the shape model of the terminal. However, it is not limited to this; if the terminal is equipped with markers (e.g., AR markers) or LED arrays that can calculate position and orientation, the relative position and orientation information may be calculated by detecting these. In other words, the shape model in this embodiment includes the markers and LED arrays mentioned above.

[0111] In such cases, the communication unit 131 may transmit information for identifying these terminals. It may also transmit how to identify the terminals, the method of identification, or software. That is, the communication unit has a first terminal identification means for identifying the first terminal, and the communication means may transmit a shape model of the first terminal for identifying the first terminal based on second image information.

[0112] Furthermore, the first terminal identification means may identify the first terminal by recognizing the first terminal from the second image information using a shape model of the first terminal.

[0113] This embodiment describes a method for unifying the reference coordinate system at a certain time s. By unifying the reference coordinate system based on information acquired at multiple time points and using the average or median of these values, it is possible to unify the reference coordinate system with higher accuracy. Such a configuration can be realized if the communication unit 131 transmits position and orientation information at multiple time points, and each terminal holds this information.

[0114] In this embodiment, the second position and orientation information Mv2(t) of the second terminal T21 was corrected to match the reference coordinate system of the first terminal T11. The roles of the first terminal T11 and the second terminal T21 may be swapped.

[0115] In other words, the configuration involves two terminals, one of which operates as the first terminal T11 and the other as the second terminal T21. In such a configuration, the communication unit 131 can be realized by sending and receiving information to determine the representative terminal (the terminal that sets the reference coordinate system).

[0116] For example, two terminals each calculate a value using random numbers, and the one that calculates the larger value is designated as the first terminal (representative terminal). Alternatively, the terminal with higher position and orientation calculation accuracy or stability may be designated as the representative terminal.

[0117] High position and orientation calculation accuracy means, for example, that the reprojection error value in SLAM is small, or that the residuals of the optimization process in position and orientation calculation are small. High position and orientation calculation stability means that a large number of feature points are used in SLAM, or that the brightness changes in the image information are small.

[0118] The communication unit 131 may transmit this information (position and attitude calculation accuracy information and position and attitude calculation stability information) to set a representative terminal. In this way, the reference coordinate system can be standardized using the terminal that calculates higher accuracy as the reference.

[0119] Both the first terminal T11 and the second terminal T21 may maintain the configuration described in this embodiment, calculate relative position and orientation information M12(s) together, and integrate them to create a common reference coordinate system.

[0120] The communication unit 131 may also exchange information for calculating the positional and orientation information required by both parties. In this way, the reference coordinate system can be standardized with higher accuracy.

[0121] Furthermore, when calculating relative position and orientation information M12(s), if the first terminal T11 is being obstructed by an obstacle or the like in the second image information, information (guidance) may be presented to the user using the first terminal T11 or the second terminal T21 to move to avoid being obstructed. The communication unit 131 may also exchange such information. In this way, a more accurate reference coordinate system can be standardized.

[0122] <Embodiment 2> In Embodiment 1, the second terminal received information from the first terminal to standardize the reference coordinate system and corrected the second position and orientation information. Embodiment 2 describes an example in which the second terminal takes a picture of the first terminal, transmits the image to the first terminal, and the first terminal, upon receiving the image, corrects the second position and orientation information.

[0123] Specifically, the first terminal calculates relative position and orientation information between terminals using image information of the first terminal taken by another terminal and a shape model held by the first terminal itself, and uses this relative position and orientation information to standardize the reference coordinate system.

[0124] Figure 5 is a functional block diagram showing an example configuration of the information processing system of Embodiment 2. Components similar to those in Embodiment 1 are indicated by the same numbers and their descriptions are omitted. Information processing system 2 includes a first terminal T51 and a second terminal T52.

[0125] The differences from Embodiment 1 are that the relative position and attitude information calculation unit 213 and the second position and attitude information correction unit 223 are mounted on the first terminal T51, and the information transmitted and received by the communication unit 231 is different.

[0126] The communication unit 231 is installed in the first terminal T51 and the second terminal T52. The second position and attitude information calculated by the second position and attitude information calculation unit 122 (Mv2(s) in Figure 1) and the second image of the first terminal T51 acquired by the second acquisition unit 121 are transmitted to the relative position and attitude information calculation unit 213 via the communication unit 231.

[0127] The relative position and orientation information calculation unit 213 uses the second position and orientation information (Mv2(s)) and the second image information to calculate the relative position information (M12(s) in Figure 1) of the first terminal T51 and the second terminal T52, and outputs it to the second position and orientation information correction unit 223.

[0128] The second position and attitude information correction unit 223 receives the relative position and attitude information M12(s) mentioned above, the first position and attitude information Mw1(s) calculated by the first position and attitude information calculation unit 113, and the second position and attitude information Mv2(s) received via the communication unit 231.

[0129] Then, based on the relative position and orientation information M12(s), the first position and orientation information Mw1(s), and the second position and orientation information Mv2(s), the relative position and orientation information (Mwv in Figure 1) between the first reference coordinate system X31 and the second reference coordinate system X32 is calculated.

[0130] The relative position and orientation information Mwv is then transmitted from the first terminal T51 to the second terminal T52 via the communication unit 231.

[0131] Figure 6 is a flowchart illustrating an example of an information processing method using the information processing system of Embodiment 2. The CPUs and other components of the computers located inside the first and second terminals execute computer programs stored in memory, thereby sequentially performing each step of the flowchart in Figure 6.

[0132] Step S610 is a process performed at the first terminal T51, and step S620 is a process performed at the second terminal T52. Process steps similar to those in Embodiment 1 are indicated by the same numbers and their explanations are omitted.

[0133] Step S2201 is executed if it is determined that the second position and attitude information corrected in step S1204 has not yet been received. In step S2201, the second position and attitude information calculated by the second position and attitude information calculation unit 122, and the second image information showing the first terminal T51 acquired by the second acquisition unit 121 are transmitted to the first terminal via the communication unit 231.

[0134] In other words, the communication means transmits the second image information and second position and orientation information acquired by the second acquisition means from the second terminal to the first terminal. After transmission, proceed to S2202.

[0135] In step S2202, the corrected second position and attitude information is received. That is, the relative position and attitude information Mwv between the first reference coordinate system X31 and the second reference coordinate system X32, calculated by the second position and attitude information correction unit 223, is received and stored via the communication unit 231.

[0136] Furthermore, similar to Embodiment 1, in step S1209, the second position and attitude information calculation unit 122 performs the calculation of Equation 1 using the relative position and attitude information (Mwv) between the reference coordinate systems to calculate the second position and attitude information (Mw2(t)) with respect to the first reference coordinate system X31.

[0137] In step S2101, it is determined whether the second image and the second position and orientation information have been received. That is, it is checked whether the data transmitted in step S2201 was received by the first terminal T51. If it has been received, the process proceeds to step S2102; otherwise, it proceeds to step S1106.

[0138] In step S2102, the relative position and orientation information M12(s) of the first terminal T51 and the second terminal T52 at time s is calculated. That is, the second position and orientation information correction unit 223 calculates the above relative position and orientation information M12(s) using the second image, the second position and orientation information, and the shape model of the first terminal T51 held by a holding unit (not shown) mounted on the first terminal T51.

[0139] Specifically, the relative position and orientation information calculation means calculates relative position and orientation information using the second image information and second position and orientation information received by the communication means, and the shape model of the first terminal possessed by the first terminal.

[0140] The calculation procedure is the same as the procedure described in step S1205 of Embodiment 1. In this embodiment, as described in Embodiment 1, it is not necessary for the second terminal T52 to hold the shape model of the first terminal T51, which is another terminal; it is sufficient for the first terminal T51 to hold the shape model of the first terminal T51.

[0141] In step S2103, the second position and attitude information is corrected. Specifically, the second position and attitude information correction unit 223 calculates the relative position and attitude information (Mwv) between the first reference coordinate system X31 and the second reference coordinate system X32.

[0142] Specifically, relative position and orientation information (Mwv) is calculated using equation 2, with respect to the relative position and orientation information M12(s) of the first terminal T51 and the second terminal T52, the second position and orientation information Mv2(s), and the first position and orientation information Mw1(s).

[0143] Subsequently, in step S2104, the corrected second position and attitude information is transmitted. That is, the relative position and attitude information Mwv is transmitted to the second terminal via the communication unit 231. In this embodiment, the communication means transmits the position and attitude information corrected by the second position and attitude information correction means from the first terminal to the second terminal. After that, the process proceeds to step S1106.

[0144] In this embodiment, the image of the first terminal captured by the second terminal and the position and orientation information calculated by the second terminal are transmitted to the first terminal, and the first terminal calculates the relative position and orientation information (Mwv) between the first reference coordinate system X31 and the second reference coordinate system X32.

[0145] In this way, each terminal only needs to maintain its own terminal shape model, and even if the terminals have different shapes, communication of shape models as described in the modified example of Embodiment 1 becomes unnecessary. Furthermore, the relative position and orientation between the two terminals can be calculated even if the shapes of the two terminals do not match.

[0146] In other words, it is possible to calculate relative position and orientation information (Mwv) between the first reference coordinate system X31 and the second reference coordinate system X32. Furthermore, it is possible to calculate position and orientation in a common reference coordinate system even on different devices, and to display a virtual object at the same location in physical space on multiple devices.

[0147] <Modified form of Embodiment 2> In Embodiment 2, the relative position and orientation information Mwv between the first reference coordinate system X31 and the second reference coordinate system X32 was calculated by the first terminal T51. However, the relative position and orientation information M12(s) between the first and second terminals may be calculated by the first terminal T51, and the relative position and orientation between the reference coordinate systems may be calculated by the second terminal T52.

[0148] Specifically, relative position and orientation information M12(s) of the first terminal T51 and the second terminal T52 is transmitted from the first terminal T51 to the second terminal T52 via the communication unit 131. This reduces the computational load on the first terminal. In particular, when multiple terminals attempt to align their reference coordinate systems with the first terminal, it becomes possible to reduce the processing load on the first terminal T51.

[0149] Furthermore, the communication content of the communication unit 231 is not limited to position and orientation information; other information may be transmitted as long as it is necessary to standardize the reference coordinate system. For example, camera parameters of the first imaging device 110 may be transmitted from the first terminal T51 to the second terminal T52.

[0150] The camera parameters mentioned above include parameters representing the camera's focal length, optical center, and lens distortion. By doing so, relative position and orientation information (M12(s)) can be calculated using appropriate parameters for the captured camera, allowing for more accurate alignment of the reference coordinate system.

[0151] Furthermore, the second acquisition unit 121 may correct the second image information so as to standardize the camera parameters of the first imaging device 110 and the second imaging device 120. Standardizing the camera parameters may be done using a known method, such as the one described in Japanese Patent Application Publication No. 2022-11818. The communication unit 231 may also pre-transmit and receive the standardized camera parameters so that each terminal can utilize them.

[0152] <Embodiment 3> Embodiments 1 and 2 describe a configuration in which information is sent and received to share a common reference coordinate system between different terminals. Embodiment 3 describes a configuration in which individual terminals identify other terminals, pair them, and then disconnect them.

[0153] Figure 7 is a functional block diagram showing an example configuration of the information processing system of Embodiment 3. Components similar to those in Embodiment 1 are indicated by the same numbers and their descriptions are omitted. The information processing system 3 includes a first terminal T71 and a second terminal T72.

[0154] The differences from Embodiment 1 are that a first terminal identification unit 311, a first display unit 312, a second terminal identification unit 321, and a second display unit 322 have been added, and the information transmitted and received by the communication unit 331 has changed.

[0155] Here, the first terminal identification unit 311 functions as a first terminal identification means for identifying the first terminal. The first display unit 312 and the second display unit 322 function as display means.

[0156] The communication unit 331 is installed in the first terminal T71 and the second terminal T72, and in addition to the information transmitted and received by the communication unit 131 described in Embodiment 1, it transmits and receives the following information.

[0157] In other words, the information transmitted and received by the communication unit 131 includes identification information that the first terminal T71 uses to identify the second terminal T72, which is input and output by the first terminal identification unit 311 and the second terminal identification unit 321.

[0158] Furthermore, the information transmitted and received by the communication unit 131 includes pairing requests and unpairing requests for pairing the first terminal T71 and the second terminal T72, as well as pairing information relating to their approval or denial. In addition, the information transmitted and received by the communication unit 131 includes 3D content (virtual objects) displayed by the first display unit 312 and the second display unit 322.

[0159] The first terminal identification unit 311 outputs to the communication unit 331 the first identification information, including the first position and orientation information Mw1(t) of the first terminal T71, an identification ID (terminal number, user's username, etc.), and pairing information, to another terminal (second terminal T72).

[0160] The communication unit 331 transmits this information to the second terminal T72 via the wireless network. The first terminal identification unit 311 also receives the second identification information, described later, via the communication unit 331, determines whether to approve or deny the pairing included in the second identification information, and sends it back to the second terminal T72 via the communication unit 331.

[0161] The first display unit 312 displays to the user 3D content held by the holding unit (not shown) of the first terminal T71, and second position and orientation information Mw2(t) in the reference coordinate system of the second terminal T72, which is input via the communication unit 331. The first display unit 312 also displays information related to pairing approval (pairing information).

[0162] Variations in the UI display content will be described in detail in the modified examples 3-1 and 3-2 of Embodiment 3. Furthermore, the aforementioned 3D content is transmitted to the second terminal T72 via the communication unit 331.

[0163] The second terminal identification unit 321 outputs to the communication unit 331 the second position and orientation information Mw2(t) in the first reference coordinate system X31 and pairing information such as the identification ID (terminal number, user's username, etc.) to another terminal (the first terminal T71). The communication unit 331 inputs this information to the first terminal identification unit 311 via the wireless network.

[0164] The second display unit 322 displays to the user 3D content acquired from the first terminal T71 via the communication unit 331, the first position and orientation information Mw1(t) input via the communication unit 331, and information related to pairing approval (pairing information). Variations of the UI display content will be explained in detail in the modified examples 3-1 and 3-2.

[0165] Figure 8A is a flowchart showing an example of an information processing method using the information processing system of Embodiment 3, Figure 8B is a flowchart continuing from Figure 8A, and Figure 8C is a flowchart continuing from Figure 8B.

[0166] Furthermore, the CPUs and other components of the computers installed inside the first and second terminals execute the computer programs stored in their memory, thereby sequentially performing the operations of each step in the flowcharts shown in Figures 8A to 8C.

[0167] Step S810 is a process performed at the first terminal T71, and step S820 is a process performed at the second terminal T72. Process steps similar to those in Embodiment 1 are indicated by the same numbers and their explanations are omitted.

[0168] Steps S3101 to S3103 and S3201 to S3203 are steps for the first terminal T71 and the second terminal T72 to mutually identify the object and to standardize the reference coordinate system. Thus, in Embodiment 3, in addition to the processing steps of Embodiment 1, a step of mutually identifying the terminals is added.

[0169] In step S3201, the second terminal identification unit 321 determines whether or not it has identified the first terminal T71 using the first position and orientation information Mw1(t) of the first terminal T71 input from the communication unit 331. That is, it determines whether the first terminal T71 is shown in the assumed position in the first image information.

[0170] The assumed position is calculated using the following equation 3. That is, using the first position and orientation information Mw1(t) and the second position and orientation information Mw2(t) of the first terminal in the first reference coordinate system X31, the position of the second terminal relative to the first terminal (i.e., relative position and orientation M21(t)) is calculated using the following equation 3.

number

[0171] If the image is captured in the expected position, it is determined that the first terminal T71 has been identified, and the process proceeds to step S1207 to correct the second position and orientation information. Otherwise, the process proceeds to step S3241.

[0172] In step S3202, the identification information of the second terminal T72 is transmitted. That is, the second terminal identification unit 321 transmits the second position and orientation information Mw2(t) of the first terminal in the reference coordinate system and the identification ID to the first terminal T71 via the communication unit 331 as the identification information of the second terminal T72. After transmission, the process moves to step S3203.

[0173] In step S3203, the identification information of the first terminal T71 is registered. That is, the second terminal identification unit 321 receives the identification information of the first terminal T71 from the first terminal T71 via the communication unit 331. This identification information of the first terminal T71 is then registered by being held in a holding unit (not shown), and the process moves to step S3241.

[0174] Meanwhile, in step S3101, the first terminal identification unit 311 determines whether or not identification information for the second terminal T72 has been received via the communication unit 331. If it has been received, the process proceeds to S3102; otherwise, it proceeds to step S3141 in Figure 8B.

[0175] In step S3102, the first terminal identification unit 311 registers the identification information of the second terminal T72 in a holding unit (not shown), and then proceeds to step S3103.

[0176] In step S3103, the first terminal identification unit 311 transmits the identification information of the first terminal T71 to the second terminal T72 via the communication unit 331.

[0177] Steps S3141 and S3241 in Figure 8B are display and operation sequences for displaying 3D content and UI on the first terminal T71 and the second terminal T72, and for operating the UI. Specifically, in step S3141, the first display unit 312 displays 3D content, identification information for the second terminal T72, and various UIs on the first image information and accepts operation input.

[0178] In 3D content superposition, the 3D content is first placed in the first reference coordinate system X31. Then, the 3D content is rendered from that viewpoint using the first position and orientation information (Mw1(t)), and this is composited onto the first image information.

[0179] In the superposition of the identification information of the second terminal T72, the position of the second terminal relative to the first terminal (i.e., relative position and orientation information M12(t)) is calculated using the second position and orientation information Mw2(t) in the first reference coordinate system X31 according to equation 4 below. Then, the rendering position is determined and rendering is performed.

number

[0180] The UI displays the names of the devices that requested pairing (described later) and their approval buttons. It also accepts and inputs UI operations from the user. Once this series of processes is complete, the process moves to step S3111.

[0181] Meanwhile, in step S3241, the second display unit 322 displays the 3D content, the identification information of the first terminal T71, and various UI elements on the second image information and accepts operation input. In the superposition of 3D content, the 3D content is first placed on the first reference coordinate system X31.

[0182] Next, the 3D content viewed from that viewpoint is rendered using the second position and orientation information Mw2(t) in the first reference coordinate system X31, and this is then composited onto the second image information.

[0183] In the superposition of the identification information of the first terminal T71, the position of the second terminal relative to the first terminal (i.e., relative position and orientation M21) is calculated using Equation 3, with the second position and orientation information Mw2(t) in the first reference coordinate system X31. Then, the rendering position is determined and rendering is performed.

[0184] The UI displays the name of the device to be paired (described later) and a button to select the device. It also accepts and inputs UI operations from the user. Once this series of processes is complete, the process moves to step S3211.

[0185] Steps S3111 to S3113 and steps S3211 to S3204 are a group of steps that authorize the pairing of the first terminal and the second terminal, that is, the sharing of a common reference coordinate system, and the experience of the same 3D content.

[0186] In step S3211, it is determined whether or not the pairing destination has been determined. That is, the second terminal identification unit 321 determines whether or not to pair the first terminal T71 and the second terminal T72, i.e., to share a common reference coordinate system and experience the same 3D content.

[0187] The second terminal identification unit 321 determines whether the user has selected the terminal to pair, i.e., the first terminal T71, in the UI displayed by the second display unit 322 in step S3241. If selected, the process proceeds to step S3202; otherwise, it proceeds to step 3221.

[0188] In step S3202, the second terminal identification unit 321 sends a pairing request to the first terminal T71 via the communication unit 331. After sending the request, the process proceeds to step S3203.

[0189] In step S3203, it is determined whether the pairing is OK or not. That is, the second terminal identification unit 321 receives via the communication unit 331 whether the first terminal T71 has approved the pairing. If the pairing is approved, the process proceeds to step S3204; otherwise, it proceeds to step S3221.

[0190] In step S3204, the second terminal identification unit 321 receives the 3D content being used by the first terminal T71 via the communication unit 331. The received 3D content is held in a holding unit (not shown), and the process proceeds to step S3221.

[0191] On the other hand, in step S3111, it is determined whether or not a pairing request has been received. That is, the first terminal identification unit 311 determines, via the communication unit 331, whether or not a pairing request from the first terminal T71 has been received. If a pairing request is received, the process proceeds to step S3112; otherwise, it proceeds to step S3121.

[0192] In step S3112, the first terminal identification unit 311 determines whether or not pairing has been approved. Specifically, it transmits the pairing approval result entered by the user in the UI displayed by the first display unit 312 in step S3141 to the second terminal T72 via the communication unit 331. If pairing is approved, the process proceeds to step S3113; otherwise, it proceeds to step S3121.

[0193] In step S3113, the first terminal identification unit 311 transmits the 3D content held in a holding unit (not shown) to the second terminal T72 via the communication unit 331. After transmission, the process proceeds to step S3121.

[0194] Steps S3121 to S3123 and S3221 to S3224 are pairing splitting sequences that divide multiple paired terminals into groups.

[0195] In other words, the pairing division sequence divides a group of terminals sharing a common reference coordinate system into multiple groups. While this embodiment describes an example with two terminals, dividing three or more terminals into two groups can be achieved using a similar configuration.

[0196] In step S3121, it is determined whether or not to split the group. That is, the first terminal identification unit 311 determines whether or not a terminal to be split from the list of paired terminals has been selected. If there is a terminal that the user has selected as a target for group splitting in the UI displayed by the first display unit 312, the process proceeds to step S3122; otherwise, the process proceeds to step S3131.

[0197] In step S3122, the identification information is divided. That is, the first terminal identification unit 311 lists the terminal IDs selected by the user and randomly selects a terminal from among them to be newly assigned reference coordinates as the representative terminal. In this embodiment, the second terminal T72 is selected as the representative terminal. Once these processes are completed, the process moves to step S3123.

[0198] In step S3123, the identification information division results are transmitted. That is, the first terminal identification unit 311 transmits the list and the identification information of the representative terminal as the identification information division results to the terminals listed in step S3122 via the communication unit 331.

[0199] Meanwhile, in step S3221, the second terminal identification unit 321 determines whether or not it has received the identification information division result received via the communication unit 331. If it has received it, it proceeds to step S3222; otherwise, it proceeds to step S3231 in Figure 8C.

[0200] In step S3222, the second terminal identification unit 321 determines whether the second terminal T72 is a representative terminal based on the identification information division result received in step S3221. If it is a representative terminal, the process proceeds to step S3223; otherwise, it proceeds to step S3224.

[0201] In step S3223, the second position and orientation information calculation unit 122 calculates and sets the second reference coordinate system X32 at the second terminal T72 as described in Embodiment 1.

[0202] Furthermore, although this embodiment will not be explained, if the list includes multiple terminals, the processing flow performed by the first terminal T71 is driven, and processes are executed to make the second reference coordinate system X32 common with those terminals or to accept pairing. After calculating the second reference coordinate system X32, the process moves to step S3231 in Figure 8C.

[0203] In step S3224, the second terminal identification unit 321 identifies the representative terminal included in the list and standardizes the reference coordinate system. These processes are the same as the terminal identification and standard coordinate system standardization sequence described above, but with a different target terminal, so their explanation is omitted. After the process in step S3224 is completed, the process moves to step S3231 in Figure 8C.

[0204] In step S3231, the second terminal identification unit 321 determines whether or not to unpair the currently paired terminal. Specifically, if the user inputs an operation to unpair the terminal in the UI displayed by the second display unit 322, the process proceeds to step S3232; otherwise, or if there are no paired terminals, the process proceeds to step S1208.

[0205] In step S3232, the second terminal identification unit 321 sends a pairing cancellation request to the first terminal T71 via the communication unit 331. After sending the request, the process proceeds to step S3223.

[0206] In step S3223, the second terminal identification unit 321 deletes the 3D content received from the first terminal T71 in step S3204 from a storage unit (not shown), and also deletes the identification information of the first terminal received in step S3203. After deletion, the process moves to step S3234.

[0207] In step S3234, a second reference coordinate system is set. That is, the second position and orientation information calculation unit 122 calculates the reference coordinate system at the second terminal T72 as described in Embodiment 1. Once the calculation is complete, the process moves to step S1208.

[0208] Meanwhile, in step 3131, the first terminal identification unit 311 determines, via the communication unit 331, whether or not it has received a pairing cancellation request from the second terminal T72. If it has received the request, the process proceeds to step S3132; otherwise, it proceeds to step S1106.

[0209] In step S3132, the first terminal identification unit 311 deletes the identification information of the second terminal T72, which was registered in step S3102, from a storage unit (not shown). After deletion, the process proceeds to step S1106.

[0210] In this way, the first and second terminals communicate information for identifying, pairing, unpairing, and splitting pairing (groups) of each terminal. This allows for the identification of multiple terminals, determining which terminals to share 3D content with, creating, splitting, and deleting groups, and displaying virtual objects at the same location in physical space as the desired terminal.

[0211] <Modification 3-1 of Embodiment 3> Embodiment 3 consisted of a terminal identification / reference coordinate system unification sequence, a display / operation sequence, a pairing sequence, a pairing splitting sequence, and a pairing unpairing sequence. However, these can be selected depending on the desired configuration. In other words, not all of these are mandatory; only some functions may be executed, and the order is not limited to these.

[0212] Furthermore, while the 3D content and pairing processes were performed after standardizing the reference coordinate system, these processes could also be performed before standardizing the reference coordinate system. Moreover, the order in which 3D content is sent and received before or after the pairing process could also be changed.

[0213] This approach reduces reception waiting time because 3D content is received in advance. Furthermore, it enables configurations where information related to the standardization of the reference coordinate system and 3D content are paired before transmission, thus enhancing privacy by allowing 3D content to be displayed only to desired recipients.

[0214] Furthermore, in Embodiment 3, the process of unifying the reference coordinate system was performed using the method of Embodiment 1, but it may also be performed using the method of Embodiment 2. The configurations of Embodiment 1 and Embodiment 2 may also be used in appropriate combinations.

[0215] Furthermore, in Embodiment 3, the representative terminal during group division was selected using random numbers. However, as described in Embodiment 1, if the communication unit 331 transmits and receives position and orientation calculation accuracy information and position and orientation calculation stability information for each terminal, the terminal that calculates position and orientation information with high accuracy and stability may be selected as the representative terminal. In this way, the reference coordinate system can be standardized based on the terminal that calculates with higher accuracy.

[0216] Furthermore, the second terminal identification unit 321 may identify the terminal using time-series position and orientation information. That is, it may identify the terminal using trajectory information of the terminal's movement.

[0217] Specifically, the first position and orientation information Mw1(t) calculated by the first terminal and the position and orientation Mv1(t) of the first terminal relative to the second terminal, calculated by the second terminal, are calculated at multiple time points and used as trajectory information (referred to as the first trajectory information and the second trajectory information, respectively).

[0218] Specifically, the first position and orientation information calculation means calculates the first trajectory information of the first terminal using the first position and orientation information calculated within a predetermined time period.

[0219] Furthermore, the second position and orientation information calculation means and the relative position and orientation information calculation means calculate the second position and orientation information and relative position and orientation information calculated by the second position and orientation information within a predetermined time period to calculate the second trajectory information of the first terminal, which is the trajectory of the first terminal relative to the second terminal. The first terminal identification means may then identify the first terminal when the first trajectory information and the second trajectory information match.

[0220] In other words, if the residuals obtained when rigid body transformation is performed so that the positions of these trajectory information at the same time coincide are smaller than a predetermined value, it may be determined that the device has been identified. Furthermore, the determination may be made using only the orientation, or using the residuals of both position and orientation, not just the position. In this way, even if a device with a similar shape to the first device is shown in the second image information, it can be recognized as a different device.

[0221] Furthermore, the first terminal identification unit 311 may use the first image information to recognize the second terminal T72 and calculate the relative position and orientation information M12(t) between the terminals. In this way, the first terminal T71 and the second terminal T72 can recognize each other, and the terminals can be identified with greater accuracy. Therefore, the reference coordinate system can be made more robustly common.

[0222] Alternatively, the system may be configured to unify the reference coordinate system once, and then unify it again. In this case, the process of unifying the reference coordinate system as shown in Embodiment 3 may be re-executed, or the results of the first and second unification processes may be combined (for example, by weighted integration) and recalculated.

[0223] Alternatively, the UI could display the first reference coordinate system X31 of the first terminal and the corrected reference coordinate system used by the second terminal, and the system could be configured to recalculate using the correction amount modified by the user to adjust them.

[0224] Furthermore, it is also possible to check at any time whether the reference coordinate system has shifted. Specifically, the relative position and attitude information M12(t) between the first terminal T71 and the second terminal T72 may be recalculated. Then, the determination may be made based on the magnitude of the difference between the first position and attitude information Mw1(t) calculated by the first terminal and the position and attitude M'w1(t) calculated using the relative position and attitude information M12(t), for example using equation 5 below.

[0225] Furthermore, for the purpose of determination, the communication unit 331 may send and receive such information between the first terminal T71 and the second terminal T72.

number

[0226] By doing this, if there is a discrepancy after the reference coordinate system has been standardized, it is possible to recalculate and standardize the reference coordinate system with higher accuracy.

[0227] <Modification 3-2 of Embodiment 3> Figure 9 shows an example of a GUI displayed by the information processing system of Embodiment 3, where G10 is an example of a GUI displayed by the first display unit 312 and the second display unit 322. G101 and G102 are users belonging to Group 1, and their usernames (UserA and UserB), which are identification IDs, are displayed superimposed.

[0228] Furthermore, it is indicated that the terminals used by users belonging to Group 1 all utilize the reference coordinate system G103 and display the common 3D content G104.

[0229] Furthermore, G105 is the name of the paired group (Group1 in this case), and G106 displays a frame to identify which devices and users belong to Group1. G107 indicates to the user that they need to tap to pair in order to join this group.

[0230] G201 and G202 are users belonging to Group 2, and their usernames (UserC and UserD), which are their identification IDs, are superimposed on them. Furthermore, the terminals used by users belonging to Group 2 all use the reference coordinate system G203 and display the common 3D content G204.

[0231] Furthermore, G205 is the name of the paired group (Group2 in this case), and G206 displays a box to identify which devices and users belong to Group2. G207 indicates to the user that they need to tap to pair in order to join this group.

[0232] Furthermore, G103 and G104 are acquired by sharing a common reference coordinate system with the terminal used by G101 or G102, as described in Embodiment 3, and by receiving 3D content.

[0233] G401 is a button used to unpair an already paired device. G402 is a button used to combine multiple groups. After tapping this button, tapping G106 and G206 will unify the reference coordinate system of G103 and G203, making them a single group.

[0234] G403 is a button for splitting a group. If you tap this button and then, for example, tap G101, G101 will be separated from Group1 and moved to another group.

[0235] G501 indicates a user (UserE) who is participating in Group1. A message appears on G502 indicating that the user wants to join Group1. G503 and G504 are buttons to select whether or not to approve the user's participation in the group (i.e., pairing). Pressing G503 approves the request, while selecting G504 denies it.

[0236] Thus, in this embodiment, the display means displays terminals that share a coordinate system from among a plurality of terminals identified by the first terminal identification means, allowing the user to select them.

[0237] G601 indicates a user (UserF) whose part is hidden from view. An instruction (arrow) G602 is displayed, prompting the user to change the camera direction so that the relative position and orientation between devices can be calculated in order to standardize the reference coordinate system between devices.

[0238] G701 is the reference coordinate system of the terminal used by the user displaying this GUI10, and this is indicated by a message in G702. For example, if the reference coordinate system is misaligned when aligned with the reference coordinate system of G103, the user can adjust the reference coordinate system by dragging G701. Note that G701 does not have to be the reference coordinate system; any indicator that allows the user to recognize the misalignment will suffice, such as a computer graphics (CG) or an arrow.

[0239] In this embodiment, the display device uses the corrected second position and orientation information to superimpose the position and orientation information of the second terminal onto the first image information, or superimpose the position and orientation information of the first terminal onto the second image information.

[0240] Furthermore, this embodiment displays UIs for identifying, pairing, unpairing, and splitting paired (group) devices, as well as a UI for adjusting the reference coordinate system. In this way, users can easily perform these operations and display virtual objects at the same location in physical space as a given device without complicated operations.

[0241] In the embodiments described above, each terminal is shown to have an information processing system. However, at least a portion of the functions of the information processing system may be provided on a server outside the terminal. Also, although an HMD was described as an example of a terminal, any terminal can be used as long as it has a camera for capturing image information and functions capable of information processing.

[0242] Furthermore, the terminal could be, for example, a smartphone with a camera, or a PC equipped with a camera. Alternatively, it could be an autonomous mobile device such as an AGV equipped with a camera. Also, the types of terminals used can be different.

[0243] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible in accordance with the spirit of the present invention, and these are not excluded from the scope of the present invention. Furthermore, some of the above embodiments may be combined as appropriate.

[0244] Furthermore, the present invention includes, for example, a system that realizes the functions of the above embodiment using at least one processor such as a CPU, memory, and circuitry (e.g., an ASIC). Alternatively, multiple processors may be used for distributed processing.

[0245] Furthermore, in order to implement some or all of the control in the above embodiment, a computer program that implements the functions of the above embodiment may be supplied to an information processing system, etc., via a network or various storage media.

[0246] Furthermore, a computer (or CPU, MPU, etc.) in such an information processing system may read and execute the program. In that case, the program and the storage medium in which the program is stored constitute the present invention. The present invention includes the following combinations.

[0247] (Configuration 1) An information processing system for setting a coordinate system shared by multiple terminals, comprising: communication means for communicating between a first terminal and a second terminal; first acquisition means for acquiring first image information from a first imaging device installed in the first terminal; first reference coordinate system determination means for determining a first reference coordinate system in the first terminal based on the first image information; first position and orientation information calculation means for calculating first position and orientation information representing the position and orientation of the first terminal using the first image information and the first reference coordinate system; and second image information for acquiring second image information from a second imaging device installed in the second terminal. An information processing system characterized by comprising: a second acquisition means; a second position and orientation information calculation means for calculating second position and orientation information representing the position and orientation of the second terminal using the second image information; a relative position and orientation information calculation means for calculating relative position and orientation information representing the relative position and orientation of the first terminal and the second terminal based on the second image information and the second position and orientation information; and a second position and orientation information correction means for correcting the second position and orientation information to the first reference coordinate system using the first position and orientation information, the second position and orientation information, and the relative position and orientation information.

[0248] (Configuration 2) The information processing system according to Configuration 1, characterized in that the communication means transmits the first position and orientation information from the first terminal to the second terminal, and the relative position and orientation information calculation means calculates the relative position and orientation information using the first position and orientation information received by the communication means.

[0249] (Configuration 3) The information processing system according to Configuration 2, wherein the communication means further transmits a shape model of the first terminal from the first terminal to the second terminal, and the relative position and orientation information calculation means calculates the relative position and orientation information by performing model fitting using the shape model of the first terminal received through the communication means.

[0250] (Configuration 4) An information processing system according to any one of Configurations 1 to 3, characterized in that the communication means transmits the second image information and the second position and orientation information acquired by the second acquisition means from the second terminal to the first terminal, the relative position and orientation information calculation means calculates the relative position and orientation information using the second image information and the second position and orientation information received by the communication means and the shape model of the first terminal possessed by the first terminal, and the communication means further transmits the position and orientation information corrected by the second position and orientation information correction means from the first terminal to the second terminal.

[0251] (Configuration 5) The information processing system according to any one of Configurations 1 to 4, characterized in that the communication means further transmits time information of the first terminal and the second terminal to synchronize the time of the first terminal and the second terminal, the first acquisition means and the second acquisition means each input the time at which the first image information and the second image information were captured in association with the first image information and the second image information, respectively, and the first position and orientation information calculation means, the second position and orientation information calculation means, and the relative position and orientation information are held in association with the time at which the first image information or the second image information was captured.

[0252] (Configuration 6) An information processing system according to any one of Configurations 1 to 5, characterized in that the first position and orientation information calculation means, the second position and orientation information calculation means, and the relative position and orientation information calculation means calculate the first position and orientation information, the second position and orientation information, and the relative position and orientation information for each image at each time using the first image information or the second image information taken at multiple times, and the second position and orientation information correction means selects each position and orientation information from the first position and orientation information, the second position and orientation information, and the relative position and orientation information calculated at multiple times, for which the difference in associated shooting times is less than or equal to a predetermined value, and corrects the second position and orientation information to the first reference coordinate system.

[0253] (Configuration 7) An information processing system according to any one of Configurations 1 to 6, further comprising a first terminal identification means for identifying the first terminal.

[0254] (Configuration 8) The information processing system according to Configuration 7, wherein the communication means further transmits a shape model of the first terminal for identifying the first terminal based on the second image information, and the first terminal identification means identifies the first terminal by recognizing the first terminal from the second image information using the shape model of the first terminal.

[0255] (Configuration 9) The information processing system according to Configuration 7 or 8, characterized in that the first terminal identification means calculates first trajectory information of the first terminal calculated using the first position and orientation information calculated by the first position and orientation information calculation means within a predetermined time, and second trajectory information of the first terminal which is the trajectory of the first terminal relative to the second terminal calculated using the second position and orientation information calculated by the second position and orientation information calculation means and the relative position and orientation information within a predetermined time, and identifies the first terminal when the first trajectory information and the second trajectory information are consistent.

[0256] (Configuration 10) The information processing system according to any one of Configurations 1 to 9, further comprising a display device that superimposes the position and orientation information of the second terminal onto the first image information or the position and orientation information of the first terminal onto the second image information using the corrected second position and orientation information.

[0257] (Configuration 11) The information processing system according to any one of Configurations 7 to 10, further comprising a display means that displays terminals sharing a coordinate system from among a plurality of terminals identified by the first terminal identification means, and displays them in a way that the user can select them.

[0258] (Configuration 12) The information processing system according to any one of Configurations 1 to 11, further comprising a map information holding means for holding map information for calculating the second position and orientation information, wherein the second position and orientation information correction means corrects the coordinates of each element of the map information so that they conform to the first reference coordinate system when correcting the second position and orientation information to the first reference coordinate system.

[0259] (Method) An information processing method for setting a coordinate system to be shared by multiple terminals, comprising: a communication step of communicating between a first terminal and a second terminal; a first acquisition step of acquiring first image information from a first imaging device installed in the first terminal; a first reference coordinate system determination step of determining a first reference coordinate system in the first terminal based on the first image information; a first position and orientation information calculation step of calculating first position and orientation information representing the position and orientation of the first terminal using the first image information and the first reference coordinate system; and acquiring second image information from a second imaging device installed in the second terminal. An information processing method characterized by comprising: a second acquisition step; a second position and orientation information calculation step of calculating second position and orientation information representing the position and orientation of the second terminal using the second image information; a relative position and orientation information calculation step of calculating relative position and orientation information representing the relative position and orientation of the first terminal and the second terminal based on the second image information and the second position and orientation information; and a second position and orientation information correction step of correcting the second position and orientation information to the first reference coordinate system using the first position and orientation information, the second position and orientation information, and the relative position and orientation information.

[0260] A computer program that causes a computer to execute each step of the information processing system described in any one of the configurations 1 to 12. [Explanation of Symbols]

[0261] 1: Information Processing System 1 T11: First terminal T21: Second terminal 110: First imaging device 111: First acquisition section 112: First reference coordinate system determination unit 113: First position and attitude information calculation unit 120: Second imaging device 121: Second acquisition section 122: Second position and attitude information calculation unit 123: Relative position and attitude information calculation unit 124: Second position and attitude information correction unit 131: Communications Department

Claims

1. An information processing system for setting a coordinate system shared by multiple terminals, A communication means for communicating between a first terminal and a second terminal, A first acquisition means for acquiring first image information from a first imaging device provided in the first terminal, A first reference coordinate system determination means for determining a first reference coordinate system in the first terminal based on the first image information, A first position and orientation information calculation means calculates first position and orientation information representing the position and orientation of the first terminal using the first image information and the first reference coordinate system, A second acquisition means for acquiring second image information from a second imaging device provided in the second terminal, A second position and orientation information calculation means that calculates second position and orientation information representing the position and orientation of the second terminal using the second image information, Relative position and orientation information calculation means calculates relative position and orientation information representing the relative position and orientation of the first terminal and the second terminal based on the second image information and the second position and orientation information, A second position and attitude information correction means that corrects the second position and attitude information to the first reference coordinate system using the first position and attitude information, the second position and attitude information, and the relative position and attitude information, An information processing system characterized by having the following features.

2. The communication means transmits the first position and orientation information from the first terminal to the second terminal. The relative position and attitude information calculation means calculates the relative position and attitude information using the first position and attitude information received by the communication means. The information processing system according to feature 1.

3. The communication means further transmits a shape model of the first terminal from the first terminal to the second terminal. The relative position and orientation information calculation means calculates the relative position and orientation information by performing model fitting using the shape model of the first terminal received through the communication means. The information processing system according to feature 2.

4. The communication means transmits the second image information and the second position and orientation information acquired by the second acquisition means from the second terminal to the first terminal. The relative position and orientation information calculation means calculates the relative position and orientation information using the second image information and the second position and orientation information received by the communication means, and the shape model of the first terminal that the first terminal has. The communication means further transmits the position and orientation information corrected by the second position and orientation information correction means from the first terminal to the second terminal. The information processing system according to feature 1.

5. The communication means further transmits time information of the first terminal and the second terminal to synchronize the time of the first terminal and the second terminal. The first acquisition means and the second acquisition means each input the time at which the first image information and the second image information were captured, respectively, in association with the first image information and the second image information. The first position and orientation information calculation means, the second position and orientation information calculation means, and the relative position and orientation information are stored in association with the time when the first image information or the second image information was captured. The information processing system according to feature 1.

6. The first position and orientation information calculation means, the second position and orientation information calculation means, and the relative position and orientation information calculation means calculate the first position and orientation information, the second position and orientation information, and the relative position and orientation information for each image at each time point, using the first image information or the second image information captured at multiple times. The second position and orientation information correction means selects from the first position and orientation information, the second position and orientation information, and the relative position and orientation information calculated at multiple time points, each of which the difference in associated shooting time is less than or equal to a predetermined value, and corrects the second position and orientation information to the first reference coordinate system. The information processing system according to feature 1.

7. Furthermore, it has a first terminal identification means for identifying the first terminal, The information processing system according to feature 1.

8. The communication means further transmits a shape model of the first terminal for identifying the first terminal based on the second image information. The first terminal identification means identifies the first terminal by recognizing the first terminal from the second image information using the shape model of the first terminal. The information processing system according to feature 7.

9. The first terminal identification means calculates a first trajectory information of the first terminal calculated using the first position and orientation information calculated by the first position and orientation information calculation means within a predetermined time, and a second trajectory information of the first terminal, which is the trajectory of the first terminal relative to a second terminal calculated using the second position and orientation information calculated by the second position and orientation information calculation means and the relative position and orientation information calculated within a predetermined time, When the first trajectory information and the second trajectory information are consistent, the first terminal is identified. The information processing system according to feature 7.

10. Furthermore, the device includes a display device that superimposes the position and orientation information of the second terminal onto the first image information or the position and orientation information of the first terminal onto the second image information using the corrected second position and orientation information. The information processing system according to feature 1.

11. Furthermore, the system includes a display means that displays terminals sharing the same coordinate system from among the multiple terminals identified by the first terminal identification means, and displays them in a way that allows the user to select them. The information processing system according to feature 7.

12. Furthermore, it has a map information holding means for holding map information for calculating the second position and orientation information, The second position and orientation information correction means corrects the coordinates of each element of the map information to match the first reference coordinate system when correcting the second position and orientation information to the first reference coordinate system. The information processing system according to feature 1.

13. An information processing method for setting a coordinate system to be shared by multiple terminals, A communication step in which communication takes place between the first terminal and the second terminal, A first acquisition step of acquiring first image information from a first imaging device installed in the first terminal, A first reference coordinate system determination step in which a first reference coordinate system is determined in the first terminal based on the first image information, A first position and orientation information calculation step, which calculates first position and orientation information representing the position and orientation of the first terminal using the first image information and the first reference coordinate system, A second acquisition step of acquiring second image information from a second imaging device installed in the second terminal, A second position and orientation information calculation step, which calculates second position and orientation information representing the position and orientation of the second terminal using the second image information, A relative position and orientation information calculation step, which calculates relative position and orientation information representing the relative position and orientation of the first terminal and the second terminal based on the second image information and the second position and orientation information, An information processing method characterized by comprising: a second position and attitude information correction step of correcting the second position and attitude information to the first reference coordinate system using the first position and attitude information, the second position and attitude information, and the relative position and attitude information.

14. A computer program for causing a computer to perform each step of the information processing system described in any one of claims 1 to 12.

Images