Information processing device, information processing method, and computer program

Abstract

This invention provides an information processing device that allows multiple terminals to use a common reference coordinate system. [Solution] The information processing device is characterized by comprising: a first position and orientation acquisition means for acquiring a first position and orientation representing the relationship between a predetermined target terminal and a reference coordinate system; a second position and orientation acquisition means for acquiring a second position and orientation representing the relationship between a first terminal and the target terminal; and a position and orientation calculation means for calculating a third position and orientation representing the relationship between the first terminal and the reference coordinate system based on the first and second position and orientations.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, a computer program, etc. suitable for displaying virtual objects and the like.

Background Art

[0002] There is known a technology of MR (Mixed Reality) that superimposes and displays virtual objects (CG and characters) on an image captured by a camera. In MR, a technology of estimating the position and orientation of a terminal (an information terminal such as an HMD or a smartphone) based on video information and calculating the drawing position of a virtual object according to the movement of the terminal is used.

[0003] In order to draw a virtual object, a coordinate system serving as an origin somewhere in the three-dimensional space (hereinafter referred to as a reference coordinate system) is required. In many cases, an MR application using a marker sets the reference coordinate system at the position of the marker.

[0004] In recent years, as a method of estimating the position and orientation of a terminal, as shown in Non-Patent Document 1, a technology of SLAM (Simultaneous Localization and Mapping) that does not require a marker has spread.

[0005] Also, the reference coordinate system is often set on a plane such as the floor or a desk in the three-dimensional space, set at the initial position of the camera, or set at an arbitrary position according to user input. Note that Non-Patent Document 2 describes a method for estimating the position and orientation of an object based on model fitting.

Prior Art Documents

Non-Patent Documents

[0006]

Non-Patent Document 1

[0007] However, if markers are not used and multiple users want to experience MR, each with their own device, setting a separate reference coordinate system for each device will result in virtual objects appearing at different locations in physical space on each device.

[0008] This invention was made in view of the above-mentioned problems, and one of its objectives is to provide an information processing device that enables the use of a common reference coordinate system across multiple terminals. [Means for solving the problem]

[0009] Embodiments of the present invention are in an information processing apparatus, A first position and orientation acquisition means for acquiring a first position and orientation representing the relationship between a predetermined target terminal and a reference coordinate system, A second position and orientation acquisition means for acquiring a second position and orientation representing the relationship between the first terminal and the target terminal, A position and orientation calculation means that calculates a third position and orientation representing the relationship between the first terminal and the reference coordinate system based on the first position and orientation and the second position and orientation, It is characterized by being equipped with [the following features]. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an information processing device that enables the use of a common reference coordinate system across multiple terminals. [Brief explanation of the drawing]

[0011] [Figure 1] This is an illustrative diagram showing an example of the relationship between two users and two terminals in Embodiment 1. [Figure 2] This is an illustrative diagram showing the terminal (HMD) in Embodiment 1. [Figure 3] This is a block diagram showing an example of the hardware configuration of the information processing device in Embodiment 1. [Figure 4] This is a functional block diagram of the information processing device 510 for each terminal in Embodiment 1. [Figure 5] This flowchart shows an example of the processing flow of the information processing method executed by the information processing device 510 of each terminal in Embodiment 1. [Figure 6] This is an illustrative diagram illustrating the relationship between multiple users and multiple terminals observed by the sensor in Embodiment 2. [Figure 7] This is a functional block diagram of the information processing device 810 for each terminal in Embodiment 2. [Figure 8] This flowchart shows an example of the processing flow of the information processing method executed by the information processing device 810 in Embodiment 2. [Figure 9] This is a functional block diagram of the information processing device 1010 in Embodiment 3. [Figure 10] This flowchart shows an example of the processing flow of the information processing method executed by the information processing device 1010 in Embodiment 3. [Figure 11] This is an image diagram of the UI used to illustrate an example of guidance in the terminal of Embodiment 3. [Modes for carrying out the invention]

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each figure, the same members or elements are denoted by the same reference numerals, and redundant explanations are omitted or simplified.

[0013] <Embodiment 1> FIG. 1 is an image diagram showing an example of the relationship between two users and two terminals in Embodiment 1. In the information processing apparatus of this embodiment, two terminals are configured to be able to share one reference coordinate system.

[0014] <W 111 and 121 are users who are trying to experience MR. 211 and 221 are terminals (HMD: Head Mount Display) worn by user 111 and user 121, respectively. 301 is a reference coordinate system set in terminal 221 and is the origin for displaying virtual objects.

[0015] In addition, the relationship between terminal 221 and reference coordinate system 301 is defined as position and orientation A (302), the relationship between terminal 211 and terminal 221 is defined as position and orientation B (303), and the relationship between terminal 211 and reference coordinate system 301 is defined as position and orientation C (304). Herein, position and orientation A (302) is referred to as the first position and orientation, position and orientation B (303) is referred to as the second position and orientation, and position and orientation C (304) is referred to as the third position and orientation.

[0016] Herein, the position and orientation mentioned here is information having six degrees of freedom representing the geometric relationship between two objects, and A, B, and C are each represented by, for example, a 4×4 matrix.

[0017] In this embodiment, a predetermined terminal in which the position and orientation of reference coordinate system 301 is set is called a target terminal. In this embodiment, the target terminal is terminal 221, and the position and orientation A (302) between terminal 221 and reference coordinate system 301 is assumed to be known.

[0018] Figure 2 is an image diagram showing a terminal (HMD) in Embodiment 1. The terminal 211 includes a camera 212, a display 213 (not shown) located inside the terminal, and an information processing device 510. Images captured by the camera 212 are displayed on the display 213, allowing the user 111 to view the images displayed on the display 213.

[0019] In this embodiment, terminals 211 and 221 have the same device configuration. Similarly, terminal 221 includes a camera 222 (not shown), a display 223, and an information processing device 520. However, terminals 211 and 221 may have different configurations.

[0020] Figure 3 is a block diagram showing an example of the hardware configuration of the information processing device in Embodiment 1. 401 is the CPU, which controls various devices connected to the system bus 402. 403 is the ROM, which stores the BIOS program and boot program.

[0021] 404 is RAM and is used as the main memory of the CPU 401. 405 is external memory and stores the computer program that operates the information processing device 510. 406 is an input unit that receives input from a keyboard, mouse, or touch panel and performs processing related to the input of information.

[0022] 407 is the sensor input unit, which acquires video from the camera installed in the terminal. 408 is the communication unit, which communicates information with other terminals. 409 is the display unit, which outputs video information to the display installed in the terminal.

[0023] Figure 4 is a functional block diagram of the information processing device 510 for each terminal in Embodiment 1. Note that some of the functional blocks shown in Figure 4 are realized by having the CPU 401, etc., which acts as a computer included in the information processing device, execute computer programs stored in memory, which acts as a storage medium.

[0024] However, some or all of these can be implemented in hardware. Hardware options include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs).

[0025] Furthermore, each functional block shown in Figure 4 does not necessarily have to be housed in the same enclosure; it may be composed of separate devices connected to each other via signal paths. The above explanation regarding Figure 4 also applies to Figures 7 and 9. Terminal 221 also has a configuration similar to that of Figure 4.

[0026] 511 is a communication unit for communicating information between terminals. In this embodiment, wireless communication is performed, and information can be sent and received between terminals. 512 is a terminal identification unit that identifies a target terminal whose position and orientation in the reference coordinate system 301 have been set. The terminal identification unit 512 functions as a terminal identification means for identifying a target terminal whose position and orientation in the reference coordinate system have been set.

[0027] 513 is a first position and attitude acquisition unit that acquires a first position and attitude representing the relationship between the target terminal and the reference coordinate system 301, and functions as a first position and attitude acquisition means.

[0028] 514 is a second position and orientation acquisition unit that acquires a second position and orientation representing the relationship between its own terminal (first terminal) and the target terminal, and functions as a second position and orientation acquisition means.

[0029] 515 is a position and attitude calculation unit that calculates the position and attitude of the reference coordinate system 301 (third position and attitude) at terminal 211 based on the first position and attitude acquired by the first position and attitude acquisition unit 513 and the second position and attitude acquired by the second position and attitude acquisition unit 514.

[0030] Furthermore, the position and orientation calculation unit 515 functions as a position and orientation calculation means that calculates a third position and orientation representing the relationship between the first terminal and the reference coordinate system based on the first position and orientation and the second position and orientation.

[0031] Figure 5 is a flowchart showing an example of the processing flow of the information processing method executed by the information processing device 510 of each terminal in Embodiment 1. Note that the CPU and other components of the computer within the information processing device 510 of each terminal execute the computer program stored in memory, thereby sequentially performing the operations of each step in the flowchart of Figure 5.

[0032] Furthermore, the flowcharts illustrating the processing steps in the following explanation are not limited to examples; any combination of steps, grouping of multiple processes, or subdivision of processes is possible as long as the results of the present invention are satisfied. Additionally, each process can be individually separated and function as a single functional element, and can be used in combination with processes other than those shown.

[0033] Based on the processing flow in Figure 5, an example of processing performed within the information processing device 510 in terminal 211 will be described. In step S610, the information processing device 510 in Figure 4 is initialized. That is, a computer program is read from the external memory 405 in Figure 3, and the information processing device 510 is made operational.

[0034] In step S611, the terminal identification unit 512 identifies the target terminal whose position and orientation in the reference coordinate system 301 have been set.

[0035] In this embodiment, for example, it is assumed that the terminal 221, which is the target terminal, already has its position and orientation set in the reference coordinate system 301. Therefore, the terminal identification unit 512 confirms this via the communication unit 511 and identifies terminal 221 as the target terminal.

[0036] In step S612, the first position and attitude acquisition unit 513 acquires a position and attitude representing the relationship between the target terminal and the reference coordinate system 301 via the communication unit 511. Here, step S612 functions as a first position and attitude acquisition step that acquires a first position and attitude representing the relationship between a predetermined target terminal and the reference coordinate system.

[0037] In this embodiment, the position and orientation A(302) of the target terminal 221 and the reference coordinate system 301 is obtained. In this embodiment, the position and orientation A(302) is assumed to be the position and orientation set by the user 121 by inputting the position and orientation.

[0038] However, the position and orientation A(302) can be any position and orientation set by any method. For example, the position and orientation A(302) may be set by image recognition based on an image captured by the camera 212 of the terminal 221.

[0039] In step S613, the second position and orientation acquisition unit 514 acquires a position and orientation representing the relationship between its own terminal, terminal 211, and the target terminal, terminal 221. Here, step S613 functions as a second position and orientation acquisition step that acquires a second position and orientation representing the relationship between the first terminal and the target terminal.

[0040] In this embodiment, for example, image information captured by the camera 212 of terminal 211 is used as input to estimate the position and orientation B(303) representing the relationship between terminal 211 and terminal 221. Any method for estimating the position and orientation B(303) is acceptable as long as it is a method for estimating the position and orientation of the housing of terminal 221.

[0041] In this embodiment, the position and orientation of the terminal 221 are estimated based on model fitting using CAD data of the terminal's housing. A method for estimating the position and orientation of an object based on model fitting is known, for example, in Non-Patent Document 2.

[0042] In step S614, the position and orientation calculation unit 515 calculates the position and orientation C(304) (third position and orientation) of the reference coordinate system 301 for the terminal 211 based on position and orientation A(302) (first position and orientation) and position and orientation B(303) (second position and orientation).

[0043] Here, step S614 functions as a position and orientation calculation step that calculates a third position and orientation representing the relationship between the first terminal and the reference coordinate system based on the first and second position and orientations.

[0044] As mentioned above, position and orientation A(302) and position and orientation B(303) are each 4x4 matrices with 6 degrees of freedom. Then, the position and orientation C(304) of the reference coordinate system 301 for terminal 211 is calculated based on position and orientation A(302) and position and orientation B(303) using the following equation (1). C=AB...Formula (1)

[0045] As shown in Figure 1, position and orientation C(304) is the position and orientation of the reference coordinate system 301 as seen from terminal 211, and is calculated indirectly via position and orientation A(302) and position and orientation B(303). Furthermore, position and orientation C(304), calculated by multiplying the matrices representing position and orientation, is also a 4x4 matrix with 6 degrees of freedom.

[0046] As explained above, in this embodiment, the position and orientation A(302) of the reference coordinate system set on terminal 221, which is the target terminal, is transformed using the relative position and orientation B(303) between terminals. This allows the position and orientation C(304) of the reference coordinate system at terminal 211 to be calculated. In this way, by sharing the same reference coordinate system between, for example, two terminals, a virtual object can be displayed at the same position in physical space on each terminal.

[0047] <Variation 1-1> In Embodiment 1, the first position and orientation acquisition unit 513 acquired the position and orientation A(302), which represents the relationship between the target terminal and the reference coordinate system 301, via the communication unit 511.

[0048] Here, position and orientation A(302) is set, for example, by the user 121 of terminal 221, which is the target terminal, inputting its position and orientation. However, the position and orientation of the reference coordinate system 301 set in the target terminal can be set by any method.

[0049] For example, the position and orientation of the reference coordinate system could be set to the ground position recognized by a sensor (not shown) of terminal 221, or it could be set to the position and orientation of a marker. Alternatively, fixed values ​​predetermined could be used without using a sensor.

[0050] <Variation 1-2> In Embodiment 1, the second position and orientation acquisition unit 514 acquired a position and orientation B(303) representing the relationship between its own terminal 211 and the target terminal 221.

[0051] Specifically, the position and orientation representing the relationship between terminal 211 and terminal 221 was estimated using image information captured by the camera 212 of terminal 211 as input. However, the method for estimating the position and orientation between terminals can be any method. For example, a recognizable object such as a marker may be placed on the surface of the terminals, and the position and orientation between terminals may be estimated based on the marker, or the head of the person wearing the terminal may be recognized, and its position and orientation may be estimated.

[0052] Alternatively, the position and orientation representing the relationship between terminals 211 and 221 may be estimated by taking image information captured by the camera 212 of terminal 221, rather than terminal 211, as input. In that case, the estimated position and orientation can be acquired by terminal 211 via the communication unit 511. Alternatively, the position and orientation B(303) representing the relationship between terminals 211 and 221 may be estimated using, for example, radio waves.

[0053] <Variation 1-3> In Embodiment 1, the terminal was assumed to be an HMD (Head-Mounted Display). However, the terminal can be anything as long as it has a camera for capturing image information and the ability to process information. For example, it could be a smartphone with a camera, or a PC with a camera. Alternatively, it could be an autonomous mobile device such as an AGV (Automated Guided Vehicle) with a camera. Furthermore, the types of terminals may differ.

[0054] <Variation 1-4> In Embodiment 1, the communication unit 511 was assumed to use wireless communication, but the communication method can be any method that allows information to be sent and received between multiple terminals, and it may be wired.

[0055] <Variation 1-5> In Embodiment 1, position and orientation A(302) and position and orientation B(303) are assumed to be position and orientation set or estimated in relation to the positional relationship between terminal 211 and terminal 221 at the same time.

[0056] However, if position / attitude A(302) and position / attitude B(303) were set or estimated at different times, errors may be present due to the time difference. It is acceptable to correct position / attitude A(302) and position / attitude B(303) to account for this time difference.

[0057] For example, using SLAM as shown in Non-Patent Document 1, it is possible to calculate how the position and orientation of each terminal changed over time. Here, let t1 be the time corresponding to position and orientation A (302) and t2 be the time corresponding to position and orientation B (303).

[0058] Between t1 and t2, let F be the change in position and orientation of terminal 211, and G be the change in position and orientation of terminal 221. For example, when calculating position and orientation C(304) at time t2, the position and orientation A obtained at time t1 (not t2) will contain an error.

[0059] In this case, since position and orientation A(302) represents the relationship between terminal 221 and the reference coordinate system 301, only the change in position and orientation of terminal 221, G, needs to be corrected. The corrected position and orientation A' can be calculated using the following equation (2). A' = GA ... Equation (2)

[0060] <Variation 1-6> In Embodiment 1, the information processing device 510 is provided on the terminal 211. However, in the present invention, the device for calculating the position and orientation C(304) of the reference coordinate system at the terminal 211 does not necessarily have to be located on the terminal 211.

[0061] The information processing device 510 may reside, for example, on a separately prepared server, or on the target terminal, terminal 221. If the information processing device 510 is located elsewhere than terminal 211, the image from the camera 212 of terminal 211 shall be transmitted to the server or terminal 221 via the communication unit 511.

[0062] As described above, according to this embodiment, even when using multiple HMDs or other terminals, virtual objects can be displayed at the same location in physical space.

[0063] <Embodiment 2> While Embodiment 1 described two users and two terminals, this embodiment describes an example with three users and three terminals. When there are two terminals other than the user's own terminal, it is not possible to uniquely identify which terminal the sensor is observing. This embodiment describes an example of determining which terminal the sensor is observing when there are three or more such terminals.

[0064] Figure 6 is an illustrative diagram illustrating the relationship between multiple users and multiple terminals observed by the sensor in Embodiment 2. Image 701 is an image taken from terminal 211 of users 121 and 131 and their respective terminals 221 and 231, while image 702 is an image taken from terminal 221 of users 131 and 111 and their respective terminals 231 and 211.

[0065] In the diagram, 111, 121, and 131 represent users, and 211, 221, and 231 represent terminals. In image 701, the user taking the picture is 111, and the terminal is 211. In image 702, the user taking the picture is 121, and the terminal is 221. In embodiment 2, terminals 211, 221, and 231 have the same equipment configuration, but terminals 211, 221, and 231 may include different configurations from each other.

[0066] Figure 7 is a functional block diagram of the information processing device 810 in each terminal in Embodiment 2. Note that 811-813 and 816 in Figure 7 are the same functional blocks as 511-513 and 515 in Embodiment 1, respectively.

[0067] 814 is a second position and orientation acquisition unit that acquires the position and orientation representing the relationship between its own terminal and the target terminal. In this embodiment, assuming that multiple terminals are observed, multiple position and orientation values ​​are output as candidates for the position and orientation of the target terminal.

[0068] 815 is an individual determination unit that determines whether the position and orientation acquired by the second position and orientation acquisition unit 814 corresponds to the second position and orientation of the target terminal. The individual determination unit 815 functions as a determination means for determining whether the position and orientation representing the relationship between the first terminal and other terminals is the second position and orientation.

[0069] Figure 8 is a flowchart showing an example of the processing flow of the information processing method executed by the information processing device 810 in Embodiment 2. The CPU and other components within the information processing device 810 execute computer programs stored in memory, thereby sequentially performing the operations of each step in the flowchart in Figure 8.

[0070] Based on Figure 8, an example of a processing flow executed within the information processing device 810 in terminal 211 will be described. Here, steps S910, S911, S912, and S914 in Figure 8 are the same processing steps as steps S610, S611, S612, and S614 in Embodiment 1, respectively. Steps S913 and S914 differ from those in Embodiment 1 as described below.

[0071] In step S913, the second position and orientation acquisition unit 814 acquires a position and orientation representing the relationship between its own terminal 211 and the terminal as a target terminal candidate.

[0072] First, the second position and orientation acquisition unit 814 estimates the position and orientation of the terminal shown in the image, similar to Embodiment 1. In this embodiment, terminal 221 is assumed to be the target terminal, and the subsequent position and orientation calculation unit 816 uses the position and orientation B(303) of that target terminal.

[0073] However, if there are three terminals, it is possible that a terminal other than the target terminal, such as terminal 221 (for example, terminal 231), may be observed in the image from the camera 212 of terminal 211 itself. Specifically, consider the case where, for example, as shown in image 701 of Figure 6, two terminals, the target terminal 221 and terminal 231, are observed in the image.

[0074] The second position and orientation acquisition unit 814 acquires the position and orientation of each terminal 221 and 231 on the image, treating them as candidate target terminals. Here, the acquired position and orientation of each terminal 221 and 231 are denoted as B1 and B2. The position and orientation of each terminal are estimated and acquired based on model fitting, as in Embodiment 1.

[0075] In step S914, individual determination is performed. That is, the individual determination unit 815 determines whether the position and orientation acquired by the second position and orientation acquisition unit 814 is the position and orientation of, for example, terminal 221, which is the target terminal.

[0076] Specifically, the second position and orientation acquisition unit 814 determines the position and orientation corresponding to terminal 221 as the target terminal from among the position and orientation groups (B1, B2) acquired by the second position and orientation acquisition unit 814. In this embodiment, the determination is made by confirming the consistency of the mutual position and orientation estimation of terminal 211 and terminal 221.

[0077] To that end, terminal 221 first performs a second position and orientation acquisition in the same manner as shown in step S913. That is, terminal 221 acquires the position and orientation of terminals 231 and 211 as they appear on image 702. The acquired position and orientation are denoted as D1 and D2, respectively.

[0078] D1 and D2 are acquired by the information processing device 810 via the communication unit 811. Then, their respective positions and orientations are compared to find a pair of matching positions and orientations. There are four candidate pairs: (B1, D1), (B2, D1), (B1, D2), and (B2, D2). One of these pairs is the one whose position and orientation are mutually estimated by terminals 211 and 221.

[0079] If the terminals estimate each other's position and orientation, then the position and orientation of one terminal will match the inverse matrix of the other terminal's position and orientation. In this embodiment, a pair of terminals 211 and 221 whose position and orientations have been mutually estimated is determined by determining whether the similarity between the position and orientation of one terminal of a pair and the inverse matrix of the other terminal's position and orientation is within a threshold.

[0080] Thus, in this embodiment, based on the images captured by the camera of the first terminal and the camera of the other terminal, it is determined whether or not the positional orientation representing the relationship between the first terminal and the other terminal is the second positional orientation.

[0081] Furthermore, the above determination is made by comparing the position and orientation matrix of the other terminals obtained based on the camera image of the first terminal with the position and orientation matrix of the first terminal obtained based on the camera image of the other terminals.

[0082] In this embodiment, assume that (B1, D1) is a pair that satisfies the conditions. In this case, it can be determined that B1 is the position and orientation of terminal 221 as seen from terminal 211. Thereafter, by using B1 as the position and orientation B(303) of terminal 221 and terminal 221, the position and orientation calculation unit 816 can calculate the position and orientation C(304) of the reference coordinate system 301 for terminal 211.

[0083] As explained above, in this embodiment, even if there are, for example, three users and three terminals, the reference coordinate system can be shared between certain terminals (terminal 211 and terminal 221). Furthermore, the remaining terminal 231 can share coordinates by performing the same processing as in this embodiment for terminal 211 or terminal 221.

[0084] This allows virtual objects to be displayed at the same location in physical space on each of the three terminals. Furthermore, calculations can be performed similarly even with four or more terminals, allowing any number of terminals to share a reference coordinate system.

[0085] <Variation 2-1> In Embodiment 2, the individual determination unit 815 made the determination by confirming the consistency of the relative positions and orientations of terminals 211 and 221.

[0086] However, the above determination method can be any method that determines which terminal is being observed by the sensor. It could be determined by attaching a marker that can uniquely identify each terminal and recognizing it, or by checking whether the time-series movement of the terminal observed by the sensor matches the movement obtained by each terminal's own self-position and orientation estimation.

[0087] <Embodiment 3> In Embodiment 1, the second position and orientation acquisition unit estimates the position and orientation based on the image. When estimating the position and orientation from an image, the estimation accuracy becomes low under conditions such as when part of the terminal is hidden at the edge of the screen. Therefore, in Embodiment 3, only the position and orientation between terminals that can be estimated with good accuracy are used.

[0088] Figure 9 is a functional block diagram of the information processing device 1010 in Embodiment 3. Here, blocks 1011-1014 and 1016 in Figure 9 are the same functional blocks as 511-514 and 515 in Embodiment 1, respectively.

[0089] 1015 is an accuracy determination unit that determines whether the position and attitude acquired by the second position and attitude acquisition unit 1014 meets a predetermined accuracy requirement. Here, the accuracy determination unit 1015 functions as an accuracy determination means that determines whether the position and attitude acquired by the second position and attitude acquisition unit meets a predetermined accuracy requirement.

[0090] Figure 10 is a flowchart showing an example of the processing flow of the information processing method executed by the information processing device 1010 in Embodiment 3. The CPU and other components within the information processing device 1010 execute computer programs stored in memory, thereby sequentially performing each step of the flowchart in Figure 10.

[0091] Based on Figure 10, an example of a processing flow executed within the information processing device 1010 in terminal 211 will be described. Here, the first steps S1110 to S1113 and S1115 in Figure 12 are the same processing steps as steps S610 to S613 and S614 in Embodiment 1, respectively. Step S1114 differs from that in Embodiment 1 as shown below.

[0092] In step S1114, the accuracy determination unit 1015 determines whether the position and orientation of the terminal, which is a candidate for the target terminal, acquired by the second position and orientation acquisition unit 1014, satisfies the accuracy conditions.

[0093] Specifically, in this embodiment, it is determined whether the acquired position and orientation of a terminal as a target terminal candidate falls within a predetermined range. First, it is considered that the conditions for estimating the position and orientation with the highest accuracy are that the terminal as a target terminal candidate is in the center of the image and that the terminal is facing directly towards the camera of terminal 211.

[0094] If we define the position and orientation as X, we can expect that the accuracy will decrease as the distance from position and orientation X increases. For example, if part of the device's image extends beyond the screen, if the device is too far away in the depth direction and its image is too small on the screen, or if the device is turned sideways and its shape is difficult to recognize, the accuracy will be judged to be low.

[0095] In this embodiment, the position and orientation acquired by the second position and orientation acquisition unit 1014 is designated as B(303), and the accuracy condition is determined by determining whether the difference between the above-mentioned position and orientation X and position and orientation B(303) is within a predetermined value.

[0096] If the conditions are met, the position and orientation B(303) can be used thereafter, allowing the position and orientation calculation unit 1016 to accurately calculate the position and orientation of the reference coordinate system 301 for the terminal 211.

[0097] As described above, in Embodiment 3, the accuracy determination unit 1015 can accurately estimate the position and orientation of the reference coordinate system 301 on the terminal 211 by using only the position and orientation that satisfies the accuracy conditions. As a result, virtual objects can be displayed accurately at the same position in physical space for each terminal.

[0098] <Variation 3-1> In Embodiment 3, the accuracy determination unit 1015 determined whether the position and orientation acquired by the second position and orientation acquisition unit 1014 was within a predetermined range.

[0099] However, any method can be used to determine whether the acquired position and orientation are highly accurate. For example, if the device on the screen moves quickly, motion blur can make it difficult to accurately estimate its position and orientation.

[0100] Taking advantage of this trend, it may be determined that the accuracy of the position and attitude is high when the amount of change in the time series of the position and attitude acquired by the second position and attitude acquisition unit 1014 is small, and conversely, that the accuracy of the position and attitude is low when the movement speed is fast.

[0101] In other words, the accuracy determination unit 1015, as an accuracy determination means, should determine whether the position and attitude acquired by the second position and attitude acquisition unit satisfies at least one of the following conditions: a predetermined position and attitude range and a predetermined speed range.

[0102] Alternatively, the percentage of occlusion (obstruction) by hands or other objects can be calculated, and a low percentage can be used to determine high accuracy in position and orientation. Furthermore, a combination of the above methods may be employed.

[0103] <Modification 3-2> The information processing device 1010 may include a guidance unit 1017 (not shown) that provides guidance to the user to improve the accuracy of position and orientation. That is, a guidance means may be provided that provides guidance so that the position and orientation acquired by the second position and orientation acquisition unit falls within a predetermined range of position and orientation, based on the determination result of the accuracy determination unit 1015, which is an accuracy determination means.

[0104] Figure 11 is an image diagram of the UI to illustrate an example of guidance in a terminal of Embodiment 3. 703 is an image displayed on the display 213 of terminal 211. A frame 1201 may be displayed in the center of image 703, and characters 1202 or arrows 1203 may be displayed there as guidance to move the display position of other terminals.

[0105] By viewing this guidance, users can understand the direction in which the camera on device 211 should be pointed, thereby improving the accuracy of acquiring positional orientation between devices.

[0106] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible in accordance with the spirit of the present invention, and these are not excluded from the scope of the present invention. Furthermore, some of the above embodiments may be combined as appropriate.

[0107] Furthermore, the present invention includes, for example, a system that realizes the functions of the above embodiment using at least one processor such as a CPU, memory, and circuitry (e.g., an ASIC). Alternatively, multiple processors may be used for distributed processing.

[0108] Furthermore, in order to implement some or all of the control in the above embodiment, a computer program that implements the functions of the above embodiment may be supplied to an information processing device, etc., via a network or various storage media.

[0109] Furthermore, the computer (or CPU, MPU, etc.) in the information processing device may read and execute the program. In that case, the program and the storage medium in which the program is stored constitute the present invention. The present invention includes the following combinations.

[0110] (Configuration 1) An information processing device comprising: a first position and orientation acquisition means for acquiring a first position and orientation representing the relationship between a predetermined target terminal and a reference coordinate system; a second position and orientation acquisition means for acquiring a second position and orientation representing the relationship between a first terminal and the target terminal; and a position and orientation calculation means for calculating a third position and orientation representing the relationship between the first terminal and the reference coordinate system based on the first and second position and orientations.

[0111] (Configuration 2) The information processing apparatus according to Configuration 1, characterized in that it has terminal identification means for identifying the target terminal whose position and orientation in the reference coordinate system is set.

[0112] (Configuration 3) The information processing apparatus according to Configuration 1 or 2, characterized in that it has a determination means for determining whether the positional orientation representing the relationship between the first terminal and the other terminal is the second positional orientation.

[0113] (Configuration 4) The information processing apparatus according to Configuration 3, characterized in that the determination means determines whether the positional orientation representing the relationship between the first terminal and the other terminal is the second positional orientation, based on images taken by the camera of the first terminal and the camera of the other terminal, respectively.

[0114] (Configuration 5) The information processing apparatus according to Configuration 4, characterized in that the determination means performs the determination by comparing a matrix of the position and orientation of the other terminal obtained based on the camera image of the first terminal with a matrix of the position and orientation of the first terminal obtained based on the camera image of the other terminal.

[0115] (Configuration 6) An information processing device according to any one of Configurations 1 to 5, characterized in that it comprises an accuracy determination means for determining whether the position and orientation acquired by the second position and orientation acquisition means satisfies a predetermined accuracy.

[0116] (Configuration 7) The information processing apparatus according to Configuration 6, characterized in that the accuracy determination means determines whether the position and attitude acquired by the second position and attitude acquisition means satisfies at least one of a predetermined position and attitude range and a predetermined speed range.

[0117] (Configuration 8) A guidance means that provides guidance so that the position and attitude acquired by the second position and attitude acquisition means falls within a predetermined range of position and attitude, based on the determination result of the accuracy determination means. The information processing apparatus according to configuration 6 or 7, characterized by comprising the above.

[0118] (Method) An information processing method characterized by comprising: a first position and orientation acquisition step of acquiring a first position and orientation representing the relationship between a predetermined target terminal and a reference coordinate system; a second position and orientation acquisition step of acquiring a second position and orientation representing the relationship between a first terminal and the target terminal; and a position and orientation calculation step of calculating a third position and orientation representing the relationship between the first terminal and the reference coordinate system based on the first and second position and orientations.

[0119] A computer program that causes a computer to execute each of the means of the information processing device described in any one of configurations 1 to 8. [Explanation of Symbols]

[0120] 111, 121, 131: Users 211, 221, 231: Terminal (HMD) 212: Camera 301: Reference Coordinate System

Claims

1. A first position and orientation acquisition means for acquiring a first position and orientation representing the relationship between a predetermined target terminal and a reference coordinate system, A second position and orientation acquisition means for acquiring a second position and orientation representing the relationship between the first terminal and the target terminal, A position and orientation calculation means that calculates a third position and orientation representing the relationship between the first terminal and the reference coordinate system based on the first position and orientation and the second position and orientation, An information processing device characterized by comprising:

2. The information processing apparatus according to claim 1, characterized in that it has terminal identification means for identifying the target terminal whose position and orientation in the aforementioned reference coordinate system is set.

3. The information processing apparatus according to claim 1, further comprising determination means for determining whether the positional orientation representing the relationship between the first terminal and the other terminal is the second positional orientation.

4. The information processing apparatus according to claim 3, characterized in that the determination means determines whether the positional orientation representing the relationship between the first terminal and the other terminal is the second positional orientation, based on images taken by the camera of the first terminal and the camera of the other terminal, respectively.

5. The information processing apparatus according to claim 4, characterized in that the determination means performs the determination by comparing a matrix of position and orientation of the other terminal obtained based on the camera image of the first terminal with a matrix of position and orientation of the first terminal obtained based on the camera image of the other terminal.

6. The information processing apparatus according to claim 1, further comprising an accuracy determination means for determining whether the position and orientation acquired by the second position and orientation acquisition means satisfies a predetermined accuracy.

7. The information processing apparatus according to claim 6, characterized in that the accuracy determination means determines whether the position and attitude acquired by the second position and attitude acquisition means satisfies at least one of a predetermined position and attitude range and a predetermined speed range.

8. A guidance means provides guidance to ensure that the position and orientation acquired by the second position and orientation acquisition means falls within a predetermined range of position and orientation, based on the determination result of the accuracy determination means. The information processing apparatus according to claim 6, characterized by comprising:

9. A first position and orientation acquisition step is to acquire a first position and orientation representing the relationship between a predetermined target terminal and a reference coordinate system, A second position and orientation acquisition step for acquiring a second position and orientation representing the relationship between the first terminal and the target terminal, A position and orientation calculation step that calculates a third position and orientation representing the relationship between the first terminal and the reference coordinate system based on the first position and orientation and the second position and orientation, An information processing method characterized by comprising:

10. A computer program for causing a computer to execute each means of the information processing apparatus described in any one of claims 1 to 8.

Images