Method, program, storage medium, and information processing system

Abstract

A method for synchronizing a coordinate system between a first device and a second device employing different platform systems in a mixed reality (MR) space includes the steps of: generating each world coordinate from each 3D system; acquiring a relative coordinate position for each of the first device and the second device from a 3D object; transmitting the relative coordinates to the host computer; converting the coordinate information into a cross-platform coordinate rule; instructing the host computer to transmit an MR visual to the second device on the basis of the coordinates of the second device; and displaying the correct MR content on the display unit of the second device.

Description

Method, program, storage medium, and information processing system

[0001] The present invention relates to a method, a program, a storage medium, and an information processing system.

[0002] Previously, mixed reality (MR) connectivity was only possible on the same platform system. For example, two Hololens (MR glasses) could be connected simultaneously, but multiple devices on different platform systems could not be connected simultaneously. This is because Hololens' 3D coordinate information is completely different from other platforms. Hololens (Windows) has its own 3D coordinate information, while iOS (iPhone, iPad) has a different 3D coordinate information. As a result, each platform system could not communicate with each other or share spatial data with each other, making cross-platform connectivity impossible.

[0003] It is the ability to connect multiple devices with different platform systems simultaneously.

[0004] [1] A method according to the present disclosure is a method for synchronizing coordinate systems between a first device and a second device employing different platform systems in an MR (Mixed Reality) space, the method including the steps of: generating each world position from each 3D system; obtaining a relative coordinate position from a 3D object relative to the device itself; transmitting the relative coordinates to a host computer; converting the coordinate information into a cross-platform coordinate rule; transmitting the cross-platform coordinate information from the host computer to the second device; instructing the host computer to transmit MR visuals to the second device based on the coordinates of the second device; and displaying correct MR content.

[0005] [2] The method according to the present disclosure in [1], wherein the content is MR content, and further includes a step of transmitting the MR content from a host computer to the first device and the second device.

[0006] [3] The method according to the present disclosure in [2] further includes the steps of: transmitting the first relative coordinate information and the second relative coordinate information to a host computer; and setting a cross-platform coordinate rule based on the first relative coordinate information and the second relative coordinate information.

[0007] [4] The method according to the present disclosure in [2] further includes the steps of: transmitting the first relative coordinate information and the second relative coordinate information to the host computer; setting, by the host computer, a cross-platform coordinate rule based on the first relative coordinate information and the second relative coordinate information; adjusting, by the host computer, the first coordinate system according to the cross-platform coordinate rule; adjusting, by the host computer, the second coordinate system according to the cross-platform coordinate rule; generating, by the host computer, second content based on the adjusted second coordinate system and transmitting the second content to the second device; and displaying MR visual content on the second device, wherein the MR visual content corresponds to Hololens coordinates.

[0008] FIG. 1 is a conceptual diagram showing an example of the configuration of an information processing system according to the present embodiment; FIG. 2 is a diagram showing an example of the hardware configuration of the information processing system according to the present embodiment; FIG. 3 is a diagram showing an example of the operation of the information processing system according to the present embodiment; FIG. 4 is a diagram showing the positional relationship between each user (each device) and a 3D object; and FIG. 5 is a conceptual diagram of a case where user A (MR glasses) and a 3D object are viewed from the viewpoint of user B (tablet terminal).

[0009] <System Configuration> FIG. 1 is a diagram showing a schematic configuration of an information processing system 1 according to this embodiment. As shown in FIG. 1, the information processing system 1 according to this embodiment includes MR glasses 2 used by user A, a tablet terminal 3 used by user B, and a server 4. Here, the MR glasses 2 are, for example, Hololens (registered trademark), and the tablet terminal 3 is, for example, an iPad (registered trademark), which is an iOS device. Note that other iOS devices (such as an iPhone) may be used instead of the iPad. The MR glasses 2 and the tablet terminal 3 are assumed to employ different platform systems for MR connection. Furthermore, the server 4 may be a host computer.

[0010] The MR glasses 2, tablet terminal 3, and server 4 are connected to each other so as to be able to communicate with each other via a network 5 such as the Internet. The network 5 may be either a wired line or a wireless line, and the type and form of the line do not matter. The MR glasses 2 are worn on the user's head and provide an MR (mixed reality) space in the user's field of view. It is assumed that there are multiple MR glasses 2 and tablet terminals 3, depending on the number of users who use the information processing system 1. <Hardware Configuration> Next, the hardware configuration of the information processing system 1 according to this embodiment will be described. Figure 2 is a block diagram showing an example of the hardware configuration of the tablet terminal 3 (iOS device) and server 4 included in the information processing system 1 according to this embodiment.

[0011] In the tablet terminal 3, the CPU 301 is a processing device that controls the overall operation of the tablet terminal 3. The ROM 302 is a non-volatile memory that stores control programs executed by the CPU 301 and various data. The RAM 303 is a volatile memory that is used as a load area and work area for programs executed by the CPU 301. The storage device 304 is a storage means for storing various types of information, and may be one that is built into the tablet terminal 3 itself or one whose storage medium is removable. The display 305 is a display device that displays various types of information in the MR space. The imaging device 306 includes an imaging element that captures an image of a subject. The communication I / F (interface) 307 is an interface for connecting to the network 5. The bus 208 is a bus line that interconnects the above components.

[0012] In the server 4, the CPU 401 is a processing device that controls the overall operation of the server 4. The ROM 402 is a non-volatile memory that stores the control programs executed by the CPU 401 and various data. The RAM 403 is a volatile memory used as a load area and work area for the programs executed by the CPU 401. The storage device 404 is a storage means for storing various information and may be built into the server 4 itself or may have a removable storage medium. The communication I / F (interface) 405 is an interface for connecting to the network 5. The bus 406 is a bus line that interconnects the above components. <Functional Configuration> This method allows the same MR visuals to be shared based on the same coordinates between Hololens (Windows) and iPhone / iPad (iOS).

[0013] Sharing MR space data between Hololens and iPads can achieve several goals: 1. A first device used by user A and a second device used by user B can establish a real-time connection when interacting in the same MR space. 2. User B can monitor what is happening in user A's viewpoint. 3. When user A and user B are in different coordinate ranges, the computer (PC) can save computing resources. 4. Budget investments efficiently. If you want to create a multi-person real-time MR experience with a limited budget, you can allocate a primary budget for multiple Hololens devices and a secondary budget for the remaining iPad devices.

[0014] FIG. 3 is a diagram showing an example of the operation of the information processing system 1 according to this embodiment.

[0015] First, the MR glasses 2 (first device) and the tablet terminal 3 (second device) each activate their cameras (step S1). Each device generates separate world coordinates based on the results of scanning the real space.

[0016] Next, the server 4 (computer) acquires the respective relative coordinates (step S2). The reason for acquiring "relative coordinates" rather than "absolute coordinates" here is that using "relative coordinates" can save computational resources compared to using "absolute coordinates."

[0017] The server 4 acquires the coordinate information (relative coordinates) from the MR glasses 2 and the tablet terminal 3, and converts the two pieces of coordinate information based on a coordinate conversion system described below. The converted coordinate information defines cross-platform coordinate rules (step S3).

[0018] The server 4 synchronizes and transmits the common coordinate rule to the MR glasses 2 and the tablet terminal 3, and the MR glasses 2 and the tablet terminal 3 are connected in real time using the converted coordinate information rule (step S4).

[0019] Next, the tablet terminal 3 requests the server 4 to deliver 3D content corresponding to the coordinate system of the tablet terminal 3, and the 3D content corresponding to the coordinate system of the MR glasses 2 is displayed on the image captured by the imaging unit of the tablet terminal 3 (step S5).

[0020] If the coordinate positions of the MR glasses 2 and the tablet terminal 3 are maintained within the same range, the server 4 acquires the coordinate positions from the MR glasses 2 and the tablet terminal 3. In steps S4 to S5, both the MR glasses 2 and the tablet terminal 3 continue to send coordinate information to the server 4, and after the server 4 acquires the coordinate information from both devices, it synchronizes the data on both platforms. After acquiring the data from both platforms, the server 4 synchronizes the data with each other, so that the two cross-platform devices can always maintain a real-time connection. Note that steps S4 to S5 are executed only when the tablet terminal 3 (user B) enters the coordinate range of the MR glasses 2 (user A).

[0021] On the other hand, if the MR glasses 2 and the tablet terminal 3 are in different coordinate ranges (for example, if the MR glasses 2 (user A) perform MR interaction in the coordinate system of the MR glasses 2, and the tablet terminal 3 (user B) is viewing different MR 3D content in the coordinate system of the tablet terminal 3), the server 4 does not need to distribute the MR content of the MR glasses 2 to the tablet terminal 3. The server 4 only needs to distribute the 3D content corresponding to each coordinate. This saves computational resources.

[0022] In the field of MR technology, each software system has its own coordinate rules. However, if the world coordinate needs to be used for cross-platform MR connection, the transformed coordinate must be taken into consideration. Therefore, this disclosure proposes a coordinate transformation method for synchronized MR connection.

[0023] Rather than limiting the iPad to AR (Augmented Reality) use, this cross-platform MR processing allows the iPad (iOS) to be used in MR connected experiences.

[0024] For example, as shown in Figure 4, when user A (HoloLens) stands in front of user B (iPad), the two users are positioned in a straight line.

[0025] In this case, whether the 3D object is located between User A and User B or in front of User A, User B (iPad) can only see the 3D object superimposed on User A, as shown in Figure 5. This is because in AR, the 3D object does not have "distance data," so User B cannot recognize the distance from the 3D object to himself.

[0026] In such cases, conventionally, AR 3D objects are only displayed in the foreground of user A, but with this disclosure, Hololens position data can be subtracted, allowing the 3D object to be displayed in the correct position.

[0027] (Coordinate Transformation System) Next, we will explain the coordinate transformation rules of the information processing system 1 in this embodiment. The processing device of the server 4 in the information processing system 1 in this embodiment sets (generates) cross-platform coordinate transformation rules from first relative coordinate information acquired from the MR headset (first device) and second relative coordinate information acquired from the tablet terminal 3 (second device), and transforms each original relative coordinate into common coordinate information in accordance with the cross-platform coordinate transformation rules based on the cross-platform coordinate transformation rules.

[0028] The processor on Server 4 (the host computer) operates based on the relative coordinates from the first and second devices. These relative coordinates are sent from the first and second devices to the server. The server then sends this cross-platform coordinate rule to the first and second devices. If the iOS device requests mixed reality content that matches those coordinates, the server sends the corresponding mixed reality visual to the iOS device.

[0029] In addition, the processor of the server 4 (host computer) may further execute a step of transmitting content (MR content / AR content) around the first device to the second device in advance when the second device is in the vicinity of the first device (within a predetermined range, for example, within 10 m).

[0030] The processor of the server 4 (host computer) also receives interactive operation commands from at least one first device in the MR space and generates corresponding content in response to the interactive operation commands. When an interactive operation command sent from the first device is received, the first content including the interactive operation command is sent from the server to the iPhone or iPad for display. <Use Case> The information processing system 1 according to this embodiment can be used in any situation where multiple people need to connect to and interact with an MR experience. For example, the information processing system 1 can be used in a museum tour, an amusement park tour, or an amusement park entertainment game.

[0031] Specifically, it is applied to parent-child interaction, where the parent wears an MR headset and the child (under 12 years old) can interact with the parent using a tablet, allowing all age groups to experience MR in real time.

[0032] Also, if you need effective budget management, if 10 people participate in a museum tour at the same time, there is no need to invest in 10 MR headsets, i.e., some participants can use tablets instead of MR headsets.

[0033] Any part or all of the functional units described in this specification may be realized by a program. The program mentioned in this specification may be non-transitoryly recorded on a computer-readable recording medium.

[0034] Such a program may be installed on a computer (a so-called native application), in which case the program may be downloaded to the computer via a communication line (including wireless communication) such as the Internet, or may be distributed in a state where it is installed on the computer.

[0035] Based on the above description, a person skilled in the art may be able to conceive additional effects and various modifications of the present invention, but the aspects of the present invention are not limited to the individual embodiments described above. For example, inventions that extract only a part of each embodiment or inventions that combine multiple embodiments are naturally envisioned. Various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present invention, which can be derived from the content defined in the claims and their equivalents.

[0036] For example, what is described in this specification as one device (or component, the same applies hereinafter) (including what is depicted as one device in the drawings) may be realized by multiple devices. Conversely, what is described in this specification as multiple devices (including what is depicted as multiple devices in the drawings) may be realized by one device. Alternatively, some or all of the means or functions included in one device may be included in another device. Furthermore, a "system" may be composed of one device, or two or more devices.

[0037] Furthermore, not all of the features described in this specification are essential requirements. In particular, features described in this specification but not included in the claims can be considered optional additional features.

[0038] The problem that the present invention aims to solve should be identified by considering the entire specification. For example, if the specification states that a specific effect is achieved by a specific configuration, it can also be said that the invention solves a problem that is the reverse of the specific effect. However, it is not intended that such a specific configuration is necessarily a required requirement.

[0039] REFERENCE SIGNS LIST 1 Information processing system 2 MR glasses 3 Tablet terminal 301 CPU 302 ROM 303 RAM 304 Storage device 305 Display 306 Imaging device 307 Communication interface 308 Bus 4 Server 401 CPU 402 ROM 403 RAM 404 Storage device 405 Communication interface 406 Bus 5 Network

Claims

1. A method for synchronizing coordinate systems between a first device and a second device employing different platform systems in an MR (Mixed Reality) space, comprising the steps of: generating world coordinates from each 3D system; obtaining relative coordinate positions for the first device and the second device from a 3D object; transmitting the relative coordinates to a host computer; converting the coordinate information into cross-platform coordinate rules; instructing the host computer to transmit MR visuals to the second device based on the coordinates of the second device; and displaying the correct MR content on a display unit of the second device.

2. The method according to claim 1, wherein the content is MR content, and further comprising the step of transmitting the MR content from a host computer to the first device and the second device.

3. The method of claim 2, further comprising: a step of transmitting the first relative coordinate information and the second relative coordinate information to a host computer; and a step of setting a cross-platform coordinate rule based on the first relative coordinate information and the second relative coordinate information.

4. The method of claim 2, further comprising the steps of: transmitting the first relative coordinate information and the second relative coordinate information to the host computer; setting a cross-platform coordinate rule based on the first relative coordinate information and the second relative coordinate information by the host computer; adjusting the first coordinate system according to the cross-platform coordinate rule by the host computer; adjusting the second coordinate system according to the cross-platform coordinate rule by the host computer; generating second content based on the adjusted second coordinate system and transmitting the second content to the second device by the host computer; and displaying the first content on the first device based on the adjusted first coordinate system.

Images