A method for accessing the same server as a secondary terminal from the primary terminal, the secondary terminal, the primary terminal, and the program.

Abstract

To provide a method that makes it possible for a main terminal to access a server that is the same as a sub terminal that satisfies predetermined conditions in real space, a sub terminal, a main terminal, and a program.SOLUTION: A server access method for a system that is capable of communicating by radio waves between a main terminal and a sub terminal has a first step in which the sub terminal determines whether a radio wave received from the main terminal satisfies a predetermined condition and notifies the main terminal of a permission signal when it determines the condition to be true and a second step in which the main terminal accesses the same server as the sub terminal when the permission signal is acquired from the sub terminal. The sub terminal may determine that the predetermined condition of the first step is true when the distance measured by a TOF (Time-of-Flight) radio wave is equal to or less than a first predetermined threshold, or determine that the predetermined condition of the first step is true when the reception strength of the radio wave is equal to or greater than a second predetermined threshold.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to technologies of virtual reality (VR), augmented reality (AR), or mixed reality (MR).

Background Art

[0002] Virtual reality refers to a technology in which an HMD (Head Mount Display) terminal is worn on a user's head and a three-dimensional virtual space is displayed in the user's field of view. The user can obtain a sense of immersion in the virtual space while moving himself / herself. Augmented reality refers to a technology in which CG (Computer Graphics) images are displayed in the real space reflected in the user's field of view. For example, by superimposing CG on a see-through display or on an image obtained by photographing the real space with a camera, the real space is extended. Mixed reality refers to a technology in which a virtual space artificially created by CG is merged with the real space. Objects in the real space are integrated with the virtual space, and the user's movements mutually affect both the real space and the virtual space. According to these technologies, HMD terminals and AR / MR terminals detect the user's movements by means of an attitude sensor, and acquire and display virtual space information by accessing a virtual space server.

[0003] Conventionally, there is a technology for an experience sharing system among multiple users (see, for example, Patent Document 1). According to this technology, it has a "smart glasses" that wirelessly transmits an image captured by a camera, and a "VR device" that receives the image from the smart glasses and displays a virtual reality image to the user.

[0004] Furthermore, there are technologies that provide services via augmented reality (or virtual reality) space (see, for example, Patent Document 2). According to this technology, an IoT device with an identification code attached and an AR (or VR) device that photographs the identification code are used. The AR device reads the identification code and displays the augmented reality space that serves as its reference point to the user, while the IoT device works in conjunction with the augmented reality space to provide services to the user. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Special Publication No. 2020-506565 [Patent Document 2] Japanese Patent Publication No. 2016-71496 [Non-patent literature]

[0006] [Non-Patent Document 1] SteamVR · Lighthouse, [online], [searched October 1, 2021], Internet <URL:https: / / www.moguravr.com / terms / index-r / lighthouse / #:~:text=Lighthouse%E3%81%A8%E3%81%AF%E3%80%81PC%E5%90%91%E3%81%91,%E3%81%99%E3%82%8B%E3%81%93%E3%81%A8%E3%81%8C%E5%8F%AF%E8%83%BD%E3%81%A7%E3%81%99%E3%80%82> [Non-Patent Document 2] "Lighthouse, a VR tracking system that allows users to walk around a room, has been licensed," [online], [searched October 1, 2021], Internet<URL:https: / / www.moguravr.com / lighthouse-tracking-vr / > [Non-Patent Document 3] Rotation matrix, [online], [searched October 1, 2021], Internet<URL:https: / / ja.wikipedia.org / wiki / %E5%9B%9E%E8%BB%A2%E8%A1%8C%E5%88%97> [Non-Patent Document 4] ARToolkit Overview, [online], [searched October 1, 2021], Internet<URL:https: / / im-lab.net / artoolkit-overview / > [Non-Patent Document 5] OpenCV Detection of ArUco Markers, [online], [searched October 1, 2021], Internet<URL:https: / / docs.opencv.org / master / d5 / dae / tutorial_aruco_detection.html> [Non-Patent Document 6] SLAM, [online], [searched October 1, 2021], Internet<URL:https: / / ja.wikipedia.org / wiki / SLAM> [Non-Patent Document 7] Proposal for an AR illustration system to support understanding of pointing behavior, [online], [searched October 1, 2021], Internet <URL:http: / / www.res.kutc.kansai-u.ac.jp / ~yone / research / pdf_graduate_thesis / 201703g_UENO_Kaede.pdf> [Non-Patent Document 8] Collision overview, [online], [searched October 1, 2021], Internet <URL:https: / / docs.unrealengine.com / ja / InteractiveExperiences / Physics / Collision / Overview / index.html> [Disclosure of the Invention] [Problems that the invention aims to solve]

[0007] Figure 1 is a system configuration diagram showing that multiple sub-terminals exist in the real world visible to the operator.

[0008] As shown in Figure 1, the operator is observing multiple users a-c in the real world. Each of the users a-c is wearing a secondary terminal 2, such as an HMD terminal, on their head, and each HMD terminal is accessing the virtual space server 3. This allows each secondary terminal 2 to recreate the virtual space desired by the user.

[0009] The virtual world played through an HMD (Head-Mounted Display) is an experience driven by the user wearing the HMD. Ultimately, only the user wearing the HMD can interact with that virtual world, making it an extremely personal experience.

[0010] As shown in Figure 1, for example, we envision a scenario where a physically disabled person requiring care (user) is fitted with an HMD terminal (secondary terminal 2) to experience a virtual space. In this scenario, the person requiring care can immerse themselves in the virtual space, but the caregiver (operator) cannot perceive the virtual space that the person requiring care is viewing through the HMD terminal. Naturally, the caregiver cannot assist the person requiring care with their operations within the virtual space. These challenges are not limited to caregiving settings. For example, the relationship between vocational trainees wearing HMD devices and instructors who provide training in a virtual space is similar.

[0011] The operator, for example, can enter the same virtual space as the secondary terminal 2 through a see-through display by wearing the main terminal (e.g., an AR terminal) on their head. However, an unspecified number of users who do not exist in the secondary terminal's real world can also enter the virtual space, making it difficult to distinguish them. In contrast, the inventor of the present application considered that, on the condition that the main terminal exists within a visible distance from the sub-terminal, it may be good to allow the user of the main terminal to enter the virtual space of the sub-terminal. That is, if the caregiver (operator) can enter the virtual space visually recognized by the care recipient using the sub-terminal by using the main terminal, the caregiver may be able to assist the care recipient in operating the virtual space, it was thought.

[0012] Therefore, an object of the present invention is to provide a method for the main terminal to access the same server as the sub-terminal in the real space, a sub-terminal, a main terminal, and a program.

Means for Solving the Problems

[0013] According to the present invention, there is provided a method for entering a virtual space of a system having a main terminal and a sub-terminal, a first step in which the main terminal determines whether a sub-terminal or its peripheral device exists within a visible distance in the real space; a second step in which when the main terminal determines it is true, it accesses the same virtual space server as the sub-terminal and enters the same virtual space, characterized by comprising.

[0014] According to another embodiment of the virtual space entry method of the present invention, the system further has a search server that stores in association a user identifier and a server address of a virtual space server in which a user corresponding to the user identifier is currently participating, Regarding the first step, the main terminal acquires a user identifier from a sub-terminal or its peripheral device existing within a visible distance in the real space, Regarding the second step, the main terminal transmits the user identifier acquired from the sub-terminal to the search server, the search server responds to the main terminal with the server address corresponding to the user identifier, and the main terminal logs in to the virtual space server based on the server address, also being preferable.

[0015] According to another embodiment of the virtual space entry method of the present invention, Regarding the second step, the main terminal further transmits the user identifier acquired from the secondary terminal to the virtual space server, It is also preferable to associate the secondary terminal in the real space with the terminal in the virtual space for the virtual space server. This is also preferable.

[0016] According to another embodiment of the virtual space entry method of the present invention, Regarding the first step, The main terminal acquires the server address of the virtual space server from a secondary terminal or its peripheral device existing within a visually recognizable distance in the real space, Regarding the second step, It is also preferable that the main terminal logs in to the virtual space server based on the server address. This is also preferable.

[0017] According to another embodiment of the virtual space entry method of the present invention, The main terminal acquires the user identifier or the server address from the secondary terminal by receiving by radio waves for short-distance communication from the secondary terminal, or photographing and reading a marker attached to the secondary terminal and describing the user identifier or the server address with a camera. This is also preferable.

[0018] According to another embodiment of the virtual space entry method of the present invention, Regarding the first step, the main terminal is equipped with a camera and has previously stored the object shape or marker of the secondary terminal or its peripheral device. When the secondary terminal or its peripheral device is detected as an object by object recognition from the camera image, is equipped with a distance sensor or a depth camera and has previously stored the object shape or marker of the secondary terminal or its peripheral device. When the measured distance to the secondary terminal or its peripheral device is less than or equal to a first predetermined threshold, Having a short-range communication interface, pre-storing the terminal identifier of a sub-terminal, and when the received signal strength of the short-range communication radio waves received from the sub-terminal is equal to or greater than a second predetermined threshold, Having a short-range communication interface capable of communicating with an access point, and having pre-stored area identifiers of sub-terminals, if the area identifier detected by the location of the main terminal is the same, Having a short-range communication interface capable of communicating with an access point, and pre-storing the access point identifier of the sub-terminal, if it is the same as the access point identifier of the access point of the radio waves received by the main terminal, or Having a short-range communication interface capable of communicating with an access point, pre-storing the terminal identifier of the secondary terminal, and if the radio waves received by the primary terminal are within the access point's range, It is determined that the secondary terminal or its peripheral devices are within visible distance in real space. That is also desirable.

[0019] According to another embodiment of the virtual space entry method of the present invention, The main terminal is further equipped with a camera and attitude sensor. At predetermined intervals, the attitude TH2 of the sub-terminal in the sub-terminal coordinate system H is received from each sub-terminal. At predetermined intervals, object recognition is performed on the camera image to detect the attitude TA2 of each sub-terminal in the real space, based on the attitude sensor of the main terminal, in the main terminal coordinate system A. At predetermined intervals, for each sub-terminal, the displacement ΔTH2 of the sub-terminal attitude TH2 in the sub-terminal coordinate system H and the displacement ΔTA2 of the sub-terminal attitude TA2 in the main terminal coordinate system A are detected. For each sub-terminal, identify the sub-terminal whose displacement ΔTA2 in the main terminal coordinate system A is closest to the displacement ΔTH2 of the sub-terminal attitude TH2 in the sub-terminal coordinate system H, and associate the terminal identifier of that sub-terminal with the sub-terminal recognized from the video. Among multiple sub-terminals in the real world captured by the camera's image, the operator selects an object on one of the sub-terminals. That is also desirable.

[0020] According to another embodiment of the virtual space entry method of the present invention, The secondary terminal is also equipped with an attitude sensor. The secondary terminal orientation TA2 in the primary terminal coordinate system A is derived from the position v and tilt r of the secondary terminal in the video relative to the front facing the user. The attitude TH2 of the sub-terminal in the sub-terminal coordinate system H is detected by the attitude sensor of the sub-terminal. That is also desirable.

[0021] According to another embodiment of the virtual space entry method of the present invention, Regarding the secondary terminal orientation TA2 in the primary terminal coordinate system A, the front side of the secondary terminal that the user is facing, as seen in the video, is determined based on the position of a predetermined marker placed on the secondary terminal. That is also desirable.

[0022] According to another embodiment of the virtual space entry method of the present invention, The position v and tilt r based on the sub-terminal coordinate system H are received, and the attitude matrix consisting of the rotation matrix R based on the tilt r and the position v is set as the sub-terminal attitude TH2 in the sub-terminal coordinate system H. From the camera's video feed, the position v and tilt r are detected for each sub-terminal based on the main terminal coordinate system A. The attitude matrix, consisting of the rotation matrix R based on the tilt r and the position v, is defined as the sub-terminal attitude TA2 in the main terminal coordinate system A. That is also desirable.

[0023] According to the present invention, a method for entering a virtual space of a system having a main terminal and a sub-terminal, The first step involves the secondary terminal determining whether the primary terminal or its peripheral devices are within visible distance in the real world, and if it determines that this is true, notifying the primary terminal of a permission signal. The second step is that when the primary terminal receives a permission signal from the secondary terminal, it accesses the same virtual space server as the secondary terminal and enters the same virtual space. It is characterized by having the following features.

[0024] According to another embodiment of the virtual space entry method of the present invention, The system further includes a search server that stores user identifiers in association with the server addresses of the virtual space servers that the user corresponding to that user identifier is currently using. Regarding the first step, The secondary terminal notifies the primary terminal of the user identifier as an authorization signal. Regarding the second step, The main terminal sends the user identifier obtained from the secondary terminal to the search server. The search server responds to the main terminal with the server address corresponding to the user identifier. The main terminal logs in to the virtual space server based on the server address. That is also desirable.

[0025] According to another embodiment of the virtual space entry method of the present invention, In the second step, the primary terminal further transmits the user identifier obtained from the secondary terminal to the virtual space server. The virtual space server is linked to the corresponding sub-terminal in the real world and the terminal in the virtual world. That is also desirable.

[0026] According to another embodiment of the virtual space entry method of the present invention, Regarding the first step, The secondary terminal notifies the primary terminal of the server address as a permission signal. Regarding the second step, The main terminal logs in to the virtual space server based on the server address. That is also desirable.

[0027] According to another embodiment of the virtual space entry method of the present invention, The secondary terminal notifies the primary terminal of the user identifier or server address, It is transmitted to the main terminal by short-range radio waves, or, A marker containing a user identifier or server address, attached to the secondary terminal, is photographed and read by the camera of the primary terminal. That is also desirable.

[0028] According to another embodiment of the virtual space entry method of the present invention, Regarding the first step, the secondary terminal is: Equipped with a camera, it pre-stores the object shape or marker of the main terminal or its peripheral device, and when the sub-terminal or its peripheral device is detected as an object in the camera's image through object recognition, Equipped with a distance sensor or depth camera, and pre-stores the object shape or marker of the main terminal or its peripheral device, if the measured distance to the main terminal or its peripheral device is less than or equal to a first predetermined threshold, Having a short-range communication interface, pre-storing the terminal identifier of the main terminal, and when the received signal strength of the short-range communication radio waves received from the main terminal is equal to or greater than a second predetermined threshold, Having a short-range communication interface capable of communicating with an access point, and having pre-stored area identifiers of the main terminal, if the area identifier detected by the location of the sub-terminal is the same, Having a short-range communication interface capable of communicating with an access point, and pre-storing the access point identifier of the main terminal, if the access point identifier of the access point of the radio waves received by the sub-terminal is the same, or Having a short-range communication interface capable of communicating with an access point, pre-storing the terminal identifier of the main terminal, and if the radio waves received by the sub-terminal are within the access point, It is determined that the main terminal or its peripheral devices are within visible distance in real space. That is also desirable.

[0029] According to another embodiment of the virtual space entry method of the present invention, The secondary terminal is further equipped with a camera and attitude sensor. At predetermined intervals, the main terminal attitude TH2 in the main terminal coordinate system H is received from each terminal. At predetermined intervals, object recognition is performed on the camera footage to detect the orientation TA2 of the main terminal in the real world for each main terminal, based on the orientation sensor of the sub-terminal. At predetermined intervals, for each main terminal, the displacement ΔTH2 of the main terminal orientation TH2 in the main terminal coordinate system H and the displacement ΔTA2 of the main terminal orientation TA2 in the secondary terminal coordinate system A are detected. For each main terminal, the main terminal whose displacement ΔTA2 in the secondary terminal coordinate system A is closest to the displacement ΔTH2 of the main terminal orientation TH2 in the main terminal coordinate system H is identified, and the terminal identifier of that main terminal is associated with the main terminal recognized from the video. Among multiple main terminals in the real world captured by the camera's image, an object on one main terminal is selected through operator control. That is also desirable.

[0030] According to another embodiment of the virtual space entry method of the present invention, The main terminal is equipped with an attitude sensor. The primary terminal orientation TA2 in the secondary terminal coordinate system A is derived from the position v and tilt r of the secondary terminal in the video relative to the front facing the user. The attitude TH2 of the main terminal in the main terminal coordinate system H is detected by the attitude sensor of the main terminal. That is also desirable.

[0031] According to another embodiment of the virtual space entry method of the present invention, Regarding the main terminal orientation TA2 in the secondary terminal coordinate system A, the front side of the main terminal that the user is facing, as seen in the video, is determined based on the position of a predetermined marker placed on the main terminal. That is also desirable.

[0032] According to another embodiment of the virtual space entry method of the present invention, The system receives the position v and tilt r based on the main terminal coordinate system H, and uses the attitude matrix consisting of the rotation matrix R based on the tilt r and the position v as the main terminal attitude TH2 in the main terminal coordinate system H. From the camera's video feed, the position v and tilt r based on the sub-terminal coordinate system A are detected for each main terminal. The attitude matrix consisting of the rotation matrix R based on the tilt r and the position v is defined as the main terminal attitude TA2 in the sub-terminal coordinate system A. That is also desirable.

[0033] According to another embodiment of the virtual space entry method of the present invention, Regarding the second step, the main terminal is: To a virtual space server located on an external network via wide-area communication, To a locally located virtual server via short-range communication, or To the virtual space server installed on the secondary terminal via short-range communication Access is also desirable.

[0034] According to another embodiment of the virtual space entry method of the present invention, The primary or secondary terminal is a glasses-type, contact lens-type, or head-mounted terminal equipped with a display, which allows the operator to see multiple terminals in the real world through a see-through design or by displaying images captured by a camera. That is also desirable.

[0035] According to the present invention, in a main terminal capable of communicating with a sub-terminal, A means for determining whether a secondary terminal or its peripheral device exists within a visible distance in real space, When the visual distance determination means determines that it is true, the server access means accesses the same virtual space server as the secondary terminal and enters the same virtual space. It is characterized by having the following features.

[0036] According to the present invention, in a system having a main terminal and a sub-terminal capable of communicating with the main terminal, The secondary terminal determines whether the primary terminal or its peripheral devices are within visible distance in the real world, and if it determines that this is true, it notifies the primary terminal of a permission signal. When the primary terminal receives a permission signal from the secondary terminal, it accesses the same virtual space server as the secondary terminal and enters the same virtual space. It is characterized by the following:

[0037] According to the present invention, in a sub-terminal capable of communicating with a main terminal, A permission signal notification means that determines whether the main terminal or its peripheral devices are present within visible distance in real space, and notifies the main terminal of a permission signal when true. It has, The primary terminal is made to access the same virtual space server as the secondary terminal, and to enter the same virtual space. It is characterized by the following:

[0038] According to the present invention, in a program that enables the operation of a computer installed in a main terminal capable of communicating with a sub-terminal, A means for determining whether a secondary terminal or its peripheral device exists within a visible distance in real space, When the visual distance determination means determines that it is true, the server access means accesses the same virtual space server as the secondary terminal and enters the same virtual space. It is characterized by enabling the computer to function in this way.

[0039] According to the present invention, in a program that enables the operation of a computer installed in a sub-terminal capable of communicating with a main terminal, A permission signal notification means that determines whether the main terminal or its peripheral devices are present within visible distance in real space, and notifies the main terminal of a permission signal when true. This enables the computer to function, allowing the primary terminal to access the same virtual space server as the secondary terminal, and to enter the same virtual space. It is characterized by the following:

[0040] According to the present invention, a server access method for a system that enables communication between a main terminal and a sub-terminal by radio waves, The secondary terminal is accessing the virtual space server. The sub-terminal performs a first step in which it determines whether the radio waves received from the main terminal meet predetermined conditions, and if it determines that the conditions are true, it notifies the main terminal of a permit signal. When the main terminal receives a permission signal from the sub-terminal, it uses the same signal as the sub-terminal. virtual space Access the server and enter the same virtual space The second step is to It is characterized by having the following features.

[0041] According to another embodiment of the server access method of the present invention, Regarding the predetermined conditions of the first step, the sub-terminal determines that the condition is true if the distance measured by the Time-of-Flight (TOF) radio waves is less than or equal to the first predetermined threshold. That is also desirable.

[0042] According to another embodiment of the server access method of the present invention, Regarding the predetermined conditions of the first step, the sub-terminal determines that the condition is true if the distance measured by radio waves reflected from the main terminal using a distance sensor or depth camera is less than or equal to the first predetermined threshold. That is also desirable.

[0043] According to another embodiment of the server access method of the present invention, Regarding the predetermined conditions of the first step, the sub-terminal determines that the condition is true if the radio wave reception strength is equal to or greater than the second predetermined threshold. That is also desirable.

[0045] According to the present invention, a server access method for a system in which a primary terminal and a secondary terminal are capable of communicating by radio waves and an area tracking reference anchor is located, The secondary terminal is accessing the virtual space server. The sub-terminal determines whether the area identifier of the main terminal, which it has stored in advance, is the same as the area identifier determined by radio waves received from the reference anchor. If it determines that this is true, it notifies the main terminal of a permit signal. When the main terminal receives a permission signal from the sub-terminal, it uses the same signal as the sub-terminal. virtual spaceAccess the server and enter the same virtual space The second step is to It is characterized by having the following features.

[0046] According to the present invention, a server access method for a system in which a main terminal and a sub-terminal are able to communicate by radio waves and an access point is located, The secondary terminal is accessing the virtual space server. The sub-terminal determines whether the access point identifier of the main terminal, which it has stored in advance, is the same as the access point identifier included in the radio waves it has received from the access point. If it determines that this is true, it notifies the main terminal of a permission signal. (First step) When the primary terminal receives a permission signal from the secondary terminal, the same as the secondary terminal virtual space Access the server and enter the same virtual space The second step is to It is characterized by having the following features.

[0047] According to another embodiment of the server access method of the present invention, The system uses a user identifier and the user corresponding to that user identifier to indicate that the user is currently participating. virtual space The system further includes a search server that stores the server's server address in association with the server's address. Regarding the first step, The secondary terminal notifies the primary terminal of the user identifier as an authorization signal. Regarding the second step, The main terminal sends the user identifier obtained from the secondary terminal to the search server. The search server responds to the main terminal with the server address corresponding to the user identifier. The main terminal is based on the server address. virtual space Access the server That is also desirable.

[0048] According to another embodiment of the server access method of the present invention, In the second step, the main terminal uses the user identifier obtained from the secondary terminal. virtual space Send further to the server That is also desirable.

[0049] According to another embodiment of the server access method of the present invention, Regarding the first step, The secondary terminal notifies the primary terminal of the server address as a permission signal. Regarding the second step, The main terminal is based on the server address. virtual space Access the server That is also desirable.

[0050] According to the present invention, a server access method for a system that enables communication between a main terminal and a sub-terminal by radio waves, The secondary terminal is accessing the virtual space server. The main terminal performs a first step of determining whether the radio waves received from the sub-terminal meet predetermined conditions, When the primary terminal determines that it is true, it is identical to the secondary terminal. virtual space Access the server and enter the same virtual space The second step is to It is characterized by having the following features.

[0051] According to another embodiment of the server access method of the present invention, Regarding the predetermined conditions of the first step, the main terminal determines that the condition is true if the distance measured by the Time-of-Flight (TOF) radio waves is less than or equal to the first predetermined threshold. That is also desirable.

[0052] According to another embodiment of the server access method of the present invention, Regarding the predetermined conditions of the first step, the main terminal determines that the condition is true if the distance measured by radio waves reflected from the sub-terminal using a distance sensor or depth camera is less than or equal to the first predetermined threshold. That is also desirable.

[0053] According to another embodiment of the server access method of the present invention, Regarding the predetermined conditions of the first step, the main terminal determines that the condition is true if the radio wave reception strength is equal to or greater than the second predetermined threshold. That is also desirable.

[0055] According to the present invention, a server access method for a system in which a primary terminal and a secondary terminal are capable of communicating by radio waves and an area tracking reference anchor is located, The secondary terminal is accessing the virtual space server. The main terminal performs a first step of determining whether the area identifier of the sub-terminal, which has been stored in advance, is the same as the area identifier determined by radio waves received from the reference anchor. When the primary terminal determines that it is true, it is identical to the secondary terminal. virtual space Access the server and enter the same virtual space The second step is to It is characterized by having the following features.

[0056] According to the present invention, a server access method for a system in which a main terminal and a sub-terminal are able to communicate by radio waves and an access point is located, The secondary terminal is accessing the virtual space server. The first step is to determine whether the access point identifier of the sub-terminal, which has been stored in advance, is the same as the access point identifier included in the radio waves received by the main terminal from the access point. When the primary terminal determines that it is true, the secondary terminal is identical. virtual space Access the server and enter the same virtual space The second step is to It is characterized by having the following features.

[0057] According to another embodiment of the server access method of the present invention, The system uses a user identifier and the user corresponding to that user identifier to indicate that the user is currently participating. virtual space The system further includes a search server that stores the server's server address in association with the server's address. Regarding the first step, The main terminal obtains the user identifier from the sub-terminal, Regarding the second step, The main terminal sends the user identifier obtained from the secondary terminal to the search server. The search server responds to the main terminal with the server address corresponding to the user identifier. The main terminal is based on the server address. virtual space Log in to the server That is also desirable.

[0058] According to another embodiment of the server access method of the present invention, In the second step, the main terminal uses the user identifier obtained from the secondary terminal. virtual space Send further to the server That is also desirable.

[0059] According to another embodiment of the server access method of the present invention, Regarding the first step, The main terminal is accessed from the secondary terminal. virtual space Obtain the server address of the server, Regarding the second step, The main terminal is based on the server address. virtual space Access the server That is also desirable.

[0060] According to the present invention, communication with the main terminal is possible via radio waves. And it is accessing the virtual space server. It is a secondary terminal, The system has a permission signal notification means that determines whether the radio waves received from the main terminal meet predetermined conditions, and when it determines that the condition is true, it notifies the main terminal of a permission signal. The main terminal has the same specifications as the secondary terminal. virtual space Access the server This allows them to enter the same virtual space. Let It is characterized by the following:

[0061] According to the present invention, a main terminal capable of communicating with a sub-terminal by radio waves, Accessing a virtual server A means for determining whether radio waves received from a sub-terminal meet predetermined conditions, When determined to be true by the viewing distance determination means, the same as the sub-terminal. virtual space Access the server and enter the same virtual space Server access means and It is characterized by having the following features.

[0062] According to the present invention, communication with the main terminal is possible via radio waves. And it is accessing the virtual space server. A program that enables the computer installed in the secondary terminal to function, A permission signal notification means that determines whether the radio waves received from the main terminal meet predetermined conditions, and when it is determined to be true, notifies the main terminal of a permission signal. to make the computer work, The main terminal has the same specifications as the secondary terminal. virtual space Access the server This allows them to enter the same virtual space. Let It is characterized by the following:

[0063] According to the present invention, a program for operating a computer mounted on a main terminal that is capable of communicating with a sub-terminal by radio waves, Accessing a virtual server A means for determining whether radio waves received from a sub-terminal meet predetermined conditions, When determined to be true by the viewing distance determination means, the same as the sub-terminal. virtual space Access the server and enter the same virtual space Server access means and It is characterized by enabling the computer to function in this way. [Effects of the Invention]

[0064] According to the method, sub-terminal, main terminal, and program of the present invention, the main terminal can access the same server as the sub-terminal in real space. [Brief explanation of the drawing]

[0065] [Figure 1] This is a system configuration diagram showing that multiple sub-terminals exist in the real world visible to the operator. [Figure 2] This is a system configuration diagram for the present invention. [Figure 3]This is the first sequence diagram showing how the main terminal determines the viewing distance to the sub-terminal. [Figure 4] This is the first sequence diagram showing how the main terminal obtains the user ID and / or server address from the secondary terminal. [Figure 5] This is a diagram illustrating the functional configuration of the main terminal and sub-terminal in the first sequence. [Figure 6] This is a second sequence diagram showing how the secondary terminal determines the viewing distance to the primary terminal. [Figure 7] This is a second sequence diagram showing how the primary terminal obtains a user ID or server address from the secondary terminal. [Figure 8] This is a diagram illustrating the functional configuration of the main and sub-terminals in the second sequence. [Figure 9] This is a system configuration diagram in real space showing the view from the main terminal to the secondary terminal. [Figure 10] Figure 9 is a flowchart of the main terminal. [Figure 11] This is an explanatory diagram illustrating the relationship between the primary terminal coordinate system and the secondary terminal coordinate system. [Figure 12] This is an explanatory diagram illustrating the relationship between real space and virtual space. [Modes for carrying out the invention]

[0066] The embodiments for carrying out the present invention will be described in detail below with reference to the drawings.

[0067] Figure 2 is a system configuration diagram of the present invention.

[0068] According to existing technologies, virtual space servers generally allow multiple users located in remote locations to exist in the same virtual space. In contrast, according to the present invention, the main terminal 1 held by the operator and the sub-terminal 2 held by the user are required to be "at a distance where they can be seen from each other in real space." In other words, the present invention is characterized by being realized in the immediate vicinity of real space, and for this reason, it is possible for a caregiver (operator) wearing the main terminal 1 to assist a care recipient wearing the sub-terminal 2.

[0069] The main terminal 1 is, for example, a glasses-type AR (Augmented Reality) terminal that can display computer graphics of a virtual space on its see-through display. The operator wears the main terminal 1 and can see multiple users a to c through the see-through display. On the other hand, secondary terminal 2 is, for example, an HMD (Head Mount Display) terminal and can access the virtual space server 3. Let's assume that users a to c are wearing HMDs and experiencing the virtual space.

[0070] According to Figure 2, the main terminal 1 is an AR terminal, but it may also be an MR (Mixed Reality) terminal, and is not limited to these. The main terminal 1 may be a contact lens type, or it may be an HMD terminal with a camera that also plays images of the outside world. The operator can see multiple sub-terminals 2 that exist in the real world either through a see-through or through a display that shows images captured by the camera. On the other hand, while secondary terminal 2 is also described as an HMD terminal, it could, of course, be an AR terminal or an MR terminal, and is not limited to these. The primary terminal 1 and secondary terminal 2 may be devices equipped with a camera and display, such as a smartphone, or they may be tablets or personal computers.

[0071] The main terminal 1 communicates with the secondary terminal 2 via short-range communication and also accesses the virtual space server 3. There are three possible locations for the virtual space server 3, for example: (1) A virtual space server (Dedicated Server) located on an external network (Internet) (2) Locally located virtual space server (Dedicated Server) (3) Virtual space server (Listen Server) installed on the secondary terminal In this case, if the virtual space server 3 is located on the internet, the main terminal 1 connects to the internet via wide-area communication.

[0072] Figure 3 is a first sequence diagram showing how the main terminal determines the viewing distance to the sub-terminal.

[0073] According to Figure 3, the main terminal 1 can enter the same virtual space as the sub-terminal 2, provided that the sub-terminal 2 is located within a visible distance in the real world. (S1) The main terminal 1 determines whether or not the sub-terminal 2 or its peripheral devices are within a distance visible in real space. (S2) When the primary terminal 1 determines that S1 is true, it accesses the same virtual space server as the secondary terminal 2 and enters the same virtual space.

[0074] <Method for determining visible distance> The main terminal 1 can determine whether or not the sub-terminal 2 is within visible distance in real space by, for example, the following determinations 1 to 5.

[0075] (Determination 1) The system is equipped with a camera and has pre-stored the object shape or marker of the sub-terminal 2 or its peripheral device. If the sub-terminal 2 or its peripheral device is detected as an object by object recognition from the camera's image, it is determined that the sub-terminal 2 is at a distance visible in real space from the perspective of the main terminal 1. Even though the secondary terminal 2 is the same object in real space, it appears in various shapes in the image depending on the viewpoint. Robust feature extraction techniques are used so that it can be detected as the same object even when its shape changes. For example, an object recognition engine that has been pre-trained on the appearance of secondary terminal 2 (e.g., the appearance of the HMD terminal) may be used to recognize the object of secondary terminal 2. Generally, examples include SIFT (Scale-Invariant Feature Transform) and deep neural network classification techniques. Such feature extraction techniques are possible for both 2D images and 3D images (or point clouds). In its simplest form, for 2D images, object recognition may involve recognizing a marker attached to, for example, the sub-terminal 2 itself or its peripheral device. The marker may be, for example, a two-dimensional QR (Quick Response) code (registered trademark). At a minimum, the marker only needs to be attached to the sub-terminal 2 in a position visible from the main terminal 1.

[0076] (Determination 2) If the device is equipped with a distance sensor or depth camera and has pre-stored the object shape or marker of the sub-terminal 2 or its peripheral device, and the measured distance to the sub-terminal 2 or its peripheral device is less than or equal to the first predetermined threshold, it is determined that the sub-terminal 2 is at a distance where it can be seen in real space from the perspective of the main terminal 1. A distance sensor works by emitting light from a light source (such as an LED or laser diode) and receiving the reflected light from an object to be measured, such as sub-terminal 2, using a photodetector. The distance is calculated from the time it takes from the emission of light from the light source to the reception of the reflected light. A depth camera is a camera equipped with a depth sensor that acquires information about depth. One such method is TOF (Time-of-Flight), which diffuses the illumination range to obtain distance information as an image, similar to a camera.

[0077] (Determination 3) If the sub-terminal has a short-range communication interface, stores the terminal identifier of the sub-terminal 2 in advance, and the received strength of the short-range communication radio waves received from the sub-terminal 2 is equal to or greater than the second predetermined threshold, it is determined that the sub-terminal 2 is at a distance where it can be seen in real space from the perspective of the main terminal 1. For "short-range communication," examples include Bluetooth®, Zigbee®, and UWB (Ultra Wide Band). In the case of Bluetooth, for example, BLE (Bluetooth Low Energy), which has a particularly narrow radio wave range, is suitable. As a low-power version of Bluetooth, it operates with about one-third the power, which can keep the power consumption of the main terminal 1 and sub-terminal 2 low. For example, sub-terminal 2, acting as a tag device, continuously transmits advertising packets. These advertising packets are transmitted periodically, for example, at intervals of 100ms. According to the BLE standard, sub-terminal 2 functions as an "advertiser," and the advertising packets include a "beacon ID" as the terminal ID. By receiving the advertising packets, primary terminal 1 detects the received signal strength and can assume that sub-terminal 2 is close to being visible in the real world.

[0078] (Determination 4) If the sub-terminal 2 has a short-range communication interface that can communicate with the access point, and the area identifier of the sub-terminal 2 is pre-stored, and it is the same as the area identifier detected by the location of the main terminal, then it is determined that the sub-terminal 2 is at a distance where it can be seen in real space from the main terminal 1. For example, area tracking technology such as Lighthouse® can be employed (see, for example, Non-Patent Documents 1 and 2). Alternatively, positioning may be determined by receiving radio waves from multiple reference anchors placed indoors using ultra-wideband wireless communication such as UWB (Ultra Wide Band). The distance between the two terminals can be determined to be close if the area identifier tracking both the main terminal 1 and the sub-terminal 2 is the same. In the case of UWB, it is possible to perform accurate indoor positioning with low power consumption, a range of about 30 meters, and an error of several tens of centimeters.

[0079] (Determination 5) If the sub-terminal 2 has a short-range communication interface capable of communicating with an access point, and the access point identifier of the sub-terminal 2 is pre-stored, and the access point identifier of the access point of the radio waves received by the main terminal 1 is the same as that of the access point, then it is determined that the sub-terminal 2 is at a distance within the visible space from the main terminal 1. For example, in the case of a wireless LAN, the primary terminal 1 determines that the two terminals are close together if the access point identifier of the wireless router currently receiving the radio waves is the same as the access point identifier of the secondary terminal 2. In this case, the distance between the two terminals may also be determined to be close if the difference between the radio wave strength of the primary terminal 1 and the radio wave strength of the secondary terminal 2, based on the radio wave strength received from the same wireless router, is below a threshold.

[0080] (Determination 6) If the sub-terminal 2 has a short-range communication interface capable of communicating with the access point, and the terminal identifier of the sub-terminal 2 is stored in advance, and the radio waves received by the main terminal 1 are within the range of the access point, then it is determined that the sub-terminal 2 is at a distance within the line of sight of the main terminal 1 in real space. For example, in the case of a wireless LAN, the primary terminal 1 can receive a response containing the terminal identifier from the secondary terminal 2 by distributing a query request via multicast. The terminal identifier obtained in the response can be used to determine whether both terminals are in the same area.

[0081] Figure 4 is a first sequence diagram showing how the primary terminal obtains the user ID and / or server address from the secondary terminal.

[0082] (S1) The main terminal 1 determines that the sub-terminal 2 or its peripheral device is located within a visible distance in real space, and obtains a "User ID (Identifier)" from the sub-terminal 2 or its peripheral device. Here, there are two possible embodiments for the main terminal 1 to obtain the "User ID" from the secondary terminal 2. (S1-1) The signal is received via radio waves for short-range communication from the sub-terminal 2. For example, in the case of Bluetooth, the public announcement packet distributed from the sub-terminal 2 can also include the user ID. (S1-2) The marker on the secondary terminal 2, on which the user ID is written, is photographed and read by the camera.

[0083] Here, the system further includes a search server 4. The search server 4 stores the "user ID" and the "server address" of the virtual space server that the user corresponding to that user ID is currently in.

[0084] (S2) If S1 determines that the main terminal 1 is true, then the user I obtained from the secondary terminal 2 D is sent to search server 4. In response, search server 4 searches for the server address corresponding to the user ID. Search server 4 responds to the main terminal 1 with the server address corresponding to the user ID. The main terminal 1 can then log in to the virtual space server based on that server address.

[0085] The main terminal 1 may also transmit the user ID obtained from the secondary terminal 2 to the virtual space server 3. This allows the virtual space server 3 to associate the secondary terminal in the real world with the terminal in the virtual world.

[0086] Furthermore, the secondary terminal 2 may be capable of communicating with the user's smartphone. The user possesses a smartphone, and that smartphone communicates with the secondary terminal 2. The secondary terminal 2 can receive user information from the smartphone and notify the primary terminal 1 of the user ID based on that user information. If the user ID is to be displayed on a marker, the marker must be displayed in a variable manner depending on the user wearing the secondary terminal 2.

[0087] In another embodiment, the server address may be used instead of the user ID. (S1) The main terminal 1 may obtain the "server address" from the sub-terminal 2 or its peripheral devices. The server address includes the URL (Uniform Resource Locator), IP address, port number, and other information necessary to access the virtual space server. The server address may also be the virtual space room name or an identifier in a social VR platform such as VRChat®. (S2) If S1 determines that the primary terminal 1 is true, it can log in to the virtual space server based on the server address.

[0088] Figure 5 is a diagram showing the functional configuration of the main terminal and sub-terminal in the first sequence.

[0089] According to Figure 5, the hardware of the main terminal 1 includes a display for the operator's vision, a camera capable of capturing images of the sub-terminal 2 as real-world footage, a short-range communication interface for communication with the sub-terminal 2, and a wide-area communication interface for accessing the virtual space server 3 and the search server 4. Furthermore, according to Figure 5, the software of the main terminal 1 includes a viewing distance determination unit 11 and a server access unit 12. These functional components are realized by executing a program that enables the computer installed in the main terminal to function. The processing flow of these functional components can also be understood as a method for entering the virtual space.

[0090] [Visibility distance determination unit 11] The visibility distance determination unit 11 determines whether the sub-terminal 2 or its peripheral devices are within a visible distance in real space. It performs the same processing as in S1 described above. [Server Access Section 12] When the viewing distance determination unit 11 determines that the condition is true, the server access unit 12 accesses the same virtual space server 3 as the sub-terminal 2 and enters the same virtual space. It then performs the same processing as S2 described above.

[0091] Figure 6 is a second sequence diagram showing how the secondary terminal determines the viewing distance to the primary terminal.

[0092] As shown in Figure 6, the secondary terminal 2 can allow the primary terminal 1 to enter the same virtual space, provided that the secondary terminal 2 is located within a visible distance in the real world. (S1) Sub-terminal 2 determines whether or not the main terminal 1 or its peripheral devices are within visible distance in the real world, and if it determines that this is true, it notifies the main terminal 1 of a permission signal. (S2) When the primary terminal 1 receives a permission signal from the secondary terminal 2, it accesses the same virtual space server 3 as the secondary terminal 2 and enters the same virtual space.

[0093] Furthermore, the method for determining the visible distance from the secondary terminal 2 to the primary terminal 1 is exactly the same as the method for determining the visible distance from the primary terminal 1 to the secondary terminal 2, as described above. In other words, the only difference is whether the subject is the primary terminal 1 or the secondary terminal 2.

[0094] Figure 7 is a second sequence diagram showing how the primary terminal obtains a user ID or server address from the secondary terminal.

[0095] (S1) Sub-terminal 2 determines that the main terminal 1 or its peripheral device is within visible distance in the real world, and notifies the main terminal 1 or its peripheral device of the "User ID". Here, there are two possible embodiments for the secondary terminal 2 to notify the primary terminal 1 of the "User ID". (S1-1) The user ID is transmitted to the main terminal 1 as a permission signal via radio waves for short-range communication from the sub-terminal 2. For example, in the case of Bluetooth, the user ID can also be included as a permission signal in the public announcement packet distributed from the sub-terminal 2. (S1-2) Sub-terminal 2 displays a marker on which the user ID is written, and the camera of main terminal 1 takes a picture of it and reads it.

[0096] Similar to Figure 4, Figure 7 also includes a search server 4 as part of the system. (S2) The main terminal 1 sends the user ID received from the sub-terminal 2 along with the permission signal to the search server 4. In response, the search server 4 searches for the server address corresponding to the user ID. The search server 4 replies to the main terminal 1 with the server address corresponding to the user ID. The main terminal 1 can then log in to the virtual space server based on that server address.

[0097] Of course, the main terminal 1 may also send the user ID obtained from the secondary terminal 2 to the virtual space server 3. This allows the virtual space server 3 to associate the secondary terminal in the real world with the terminal in the virtual world.

[0098] Furthermore, in another embodiment, as in Figure 4, the server address may be used instead of the user ID in Figure 7. (S1) The secondary terminal 2 may notify the primary terminal 1 of the "server address". (S2) The primary terminal 1 can log in to the virtual space server 3 based on the server address notified by the secondary terminal 2.

[0099] Figure 8 is a diagram showing the functional configuration of the main terminal and sub-terminal in the second sequence.

[0100] As shown in Figure 8, compared to Figure 5, the sub-terminal 2 is equipped with a viewing distance determination unit 21. The function of the viewing distance determination unit 21 is exactly the same as the function of the viewing distance determination unit 11 of the main terminal 1 in Figure 5. If the viewing distance determination unit 21 determines that the sub-terminal 2 is true, it notifies the main terminal 1 of the user ID and / or server address along with a permission signal via the short-range communication interface. Of course, this could also be done by changing a marker such as a QR code.

[0101] <An embodiment in which a secondary terminal is selected from the perspective of the primary terminal (or the primary terminal from the perspective of the secondary terminal)> The main terminal 1 worn by the operator can communicate with multiple sub-terminals 2 worn by the user. However, it is difficult for the operator to select a desired sub-terminal 2 from among multiple sub-terminals 2 that are visible in the real world. If the primary terminal is an AR terminal, the operator might be able to select secondary terminal 2 by pointing to the object on that terminal with their finger. However, this selection is merely based on the object visible in the image captured by the operator's AR terminal's camera. In this case, it is not possible to identify the address (terminal ID) for communicating with the desired secondary terminal 2. An address is a terminal ID such as an IP address, MAC (Media Access Control) address, or beacon ID. It is difficult for the operator's primary terminal 1 to arbitrarily select the user's secondary terminal 2 and communicate directly with it.

[0102] Figure 9 is a diagram of the system configuration in real space, showing the view from the main terminal to the secondary terminal.

[0103] As shown in Figure 9, both the main terminal 1 and the sub-terminal 2 further have attitude sensors. [Posture Sensor] The attitude sensor constantly detects its own attitude (Transform), specifically its "position v" and "tilt r," while it is running. Figure 9 shows the following posture: TA1: Main terminal orientation in main terminal coordinate system A TH2: Sub-terminal orientation of sub-terminal coordinate system H Furthermore, the primary terminal coordinate system A shall be the reference coordinate system set when the device is started up. Similarly, the secondary terminal coordinate system H shall also be the reference coordinate system set when the device is started up.

[0104] The attitude sensor incorporates an IMU (Inertial Measurement Unit) to detect the "tilt r." This is the same type of IMU found in typical smartphones and other devices. Furthermore, the attitude sensor incorporates a tracking device mounted on the head-mounted display, such as SteamVR's Lighthouse®, to detect "position v" (see, for example, Non-Patent Document 1). This device can track position v within a predetermined range by communicating with anchors (base stations) placed in real space. Alternatively, the attitude sensor can simultaneously track its own position v and tilt r using SLAM (Simultaneous Localization And Mapping) by incorporating the aforementioned IMU and camera (see, for example, Non-Patent Document 6).

[0105] In this invention, "orientation" is defined as "orientation T" (4x4 matrix) from "position v" (3 matrix) in real space (3-dimensional space) and "rotation matrix R" (3x3 matrix) calculated from the inclination r (3 matrix), as follows (see, for example, Non-Patent Document 3).

number

[0106] Figure 10 is a flowchart of the main terminal in Figure 9.

[0107] [Sub-terminal detection unit 101] The sub-terminal detection unit 101 obtains a terminal ID from each of the sub-terminals 2. The terminal ID can be, for example, a beacon ID, an IP address, or a MAC (Media Access Control) address. For example, if short-range communication is BLE, the sub-terminal 2 periodically transmits a terminal ID, and the sub-terminal detection unit 101 of the main terminal 1 can receive that terminal ID. For example, if the short-range communication is via Wi-Fi, the sub-terminal detection unit 101 of the main terminal 1 can receive the terminal ID from the sub-terminal 2 by distributing a query request via multicast. The acquired terminal ID is output to the sub-terminal attitude receiving unit 102.

[0108] [Sub-terminal attitude receiving unit 102] The sub-terminal attitude receiving unit 102 receives the sub-terminal attitude TH2 (position v and tilt r) in the sub-terminal coordinate system H from each sub-terminal 2 already detected by the sub-terminal detection unit 101 at predetermined time intervals (Δt). The sub-terminal attitude TH2 in the sub-terminal coordinate system H is detected by the attitude sensor of the sub-terminal 2. TH2: Sub-terminal orientation of sub-terminal coordinate system H The attitude TH2 of the sub-terminal coordinate system H at each of the sub-terminals 2 at predetermined time intervals is output to the attitude displacement detection unit 104.

[0109] The object recognition unit 112 recognizes multiple sub-terminals 2 in real space as objects from the video captured by the camera.

[0110] Even though the secondary terminal 2 is the same object in real space, it appears in various shapes in the image depending on the viewpoint. Robust feature extraction techniques are used so that it can be detected as the same object even when its shape changes. For example, an object recognition engine that has been pre-trained on the appearance of secondary terminal 2 (e.g., the appearance of the HMD terminal) may be used to recognize the object of secondary terminal 2. Generally, examples include SIFT (Scale-Invariant Feature Transform) and deep neural network classification techniques. Such feature extraction techniques are possible for both 2D images and 3D images (or point clouds). In the simplest case, if it is a 2D image, the object recognition unit 112 may recognize, for example, a marker attached to the sub-terminal 2 itself. The marker may be, for example, a two-dimensional QR code (registered trademark).

[0111] [Object attitude detection unit 103] The object attitude detection unit 103 detects the attitude TA2 of each sub-terminal (object) in the main terminal coordinate system A based on the attitude sensor of the main terminal 1 at predetermined time intervals (Δt). TA2: Secondary terminal orientation in primary terminal coordinate system A The object attitude detection unit 103 detects the position v and tilt r based on the main terminal coordinate system A for each object of the sub-terminal 2 recognized by the object recognition unit 112 from the video captured by the camera. Then, it detects an attitude matrix consisting of a rotation matrix R based on the tilt r and the position v as the sub-terminal attitude TA2 in the main terminal coordinate system A. As shown in Figure 5 above, the secondary terminal orientation TA2 in the primary terminal coordinate system A is derived from the position v and tilt r of the secondary terminal 2 (HMD terminal) relative to the front facing the user wearing it, as seen in the video. The front facing the user is determined based on the position of a predetermined marker placed on the secondary terminal 2.

[0112] When an object (e.g., a marker) is captured in a 2D image by a camera, the object will appear from different angles depending on the camera's orientation. The orientation (position v and tilt r) of the object can be detected from the shape of the image of the object captured in the video (see, for example, Non-Patent Document 4). Specifically, a position vector indicating the center position of the marker and a rotation axis vector such that the x and y axes are parallel to the edges of the marker and the z axis is perpendicular to them are detected.

[0113] A rotation axis vector is a vector whose "direction" is the axis of rotation when rotating an object to achieve a desired tilt according to Rodrigues' rotation formula, and whose "norm" is the angle of rotation. Specifically, the aruco marker detection function of OpenCV (registered trademark) can be used (see, for example, Non-Patent Document 5).

[0114] Furthermore, since the secondary terminal attitude TA2 of the primary terminal coordinate system A is greatly affected by the object detection location (e.g., marker placement location), it is preferable to introduce an attitude correction matrix. TA2 = TA1T1sTS2Tc TA1: Main terminal attitude in main terminal coordinate system A T1s: Attitude of the camera (sensor) coordinate system in the main terminal attitude coordinate system. TS2: The orientation of the secondary terminal as recognized in the camera (sensor) coordinate system. Tc: Posture correction matrix Here, the principal terminal attitude coordinate system is a coordinate system that uses the principal terminal attitude in principal terminal coordinate system A as the reference. Thus, in practice, a process is required to convert from the sensor coordinate system to the main terminal coordinate system.

[0115] [Posture displacement detection unit 104] At a predetermined time interval (Δt), the attitude displacement detection unit 104 receives the attitude TH2 of the sub-terminal coordinate system H for each sub-terminal from the sub-terminal attitude receiving unit 102, and also receives the attitude TA2 of the sub-terminal coordinate system A from the object attitude detection unit 103. (time t) Sub-terminal 21's attitude TH21(t) in sub-terminal coordinate system H Sub-terminal 22's attitude TH22(t) in sub-terminal coordinate system H Sub-terminal 23's attitude TH23(t) in sub-terminal coordinate system H The secondary terminal orientation TA2a(t) of the primary terminal coordinate system A of the image-recognized object a. The secondary terminal orientation TA2b(t) of the image-recognized object b in the primary terminal coordinate system A. The secondary terminal orientation TA2c(t) of the recognized object c in the primary terminal coordinate system A. (Time t+Δt) Sub-terminal 21's attitude TH21(t+Δt) in sub-terminal coordinate system H Sub-terminal 22's attitude TH22(t+Δt) in sub-terminal coordinate system H Sub-terminal 23's attitude TH23(t+Δt) in sub-terminal coordinate system H The secondary terminal orientation TA2a(t+Δt) of the recognized object a in the primary terminal coordinate system A. The secondary terminal orientation TA2b(t+Δt) of the recognized object b in the primary terminal coordinate system A. The secondary terminal orientation TA2c(t+Δt) of the recognized object c in the primary terminal coordinate system A.

[0116] The attitude displacement detection unit 104 then detects the displacement ΔTH2 of the sub-terminal attitude TH2 in the sub-terminal coordinate system H and the displacement ΔTA2 of the sub-terminal attitude TA2 in the main terminal coordinate system A. Displacement of the secondary terminal orientation TH21 of secondary terminal 21 in the secondary terminal coordinate system H: ΔTH21=(TH21(t))-1TH21(t+Δt) Displacement of the secondary terminal TH22 orientation in the secondary terminal coordinate system H of secondary terminal 22: ΔTH22 = (TH22(t)) - 1TH22(t + Δt) Displacement of the secondary terminal TH23 orientation in the secondary terminal coordinate system H of secondary terminal 23: ΔTH23 = (TH23(t)) - 1TH23(t + Δt) Displacement of the secondary terminal orientation TA2a of the image-recognized object a in the primary terminal coordinate system A: ΔTA2a=(TA2a(t))-1TA2a(t+Δt) Displacement of the secondary terminal orientation TA2b of the image-recognized object b in the primary terminal coordinate system A: ΔTA2b=(TA2b(t))-1TA2b(t+Δt) Displacement of the secondary terminal orientation TA2c of the image-recognized object c in the primary terminal coordinate system A: ΔTA2c=(TA2c(t))-1TA2c(t+Δt)

[0117] Figure 11 is an explanatory diagram illustrating the relationship between the primary terminal coordinate system and the secondary terminal coordinate system. Figure 12 is an explanatory diagram illustrating the relationship between real space and virtual space.

[0118] As shown in Figure 11, even a sub-terminal as a single object will have a different orientation if it is based on a different coordinate system. Furthermore, the sub-terminal coordinate system H also has an orientation based on a different primary terminal coordinate system A. TA1: Main terminal orientation in main terminal coordinate system A TA2: Secondary terminal orientation in primary terminal coordinate system A TH2: Sub-terminal orientation of sub-terminal coordinate system H TAH: Attitude of the secondary terminal coordinate system H as seen from the primary terminal coordinate system A. THA: The attitude of the primary terminal coordinate system A as seen from the secondary terminal coordinate system H.

[0119] The attitude TA1 of the main terminal in the main terminal coordinate system A was detected by the attitude sensor mounted on the main terminal 1. The secondary terminal attitude TH2 in the secondary terminal coordinate system H is also detected by the attitude sensor mounted on secondary terminal 2. Primary terminal 1 receives the secondary terminal attitude TH2 in the secondary terminal coordinate system H from secondary terminal 2. The secondary terminal orientation TA2 in the main terminal coordinate system A is detected by the object orientation detection unit 114 from an object captured in the video image taken by the main terminal 1's camera.

[0120] The relative orientation in real space is calculated from the secondary terminal orientation TA2 of the primary terminal coordinate system A and the secondary terminal orientation TH2 of the secondary terminal coordinate system H, for example, the relative orientation TAH of the secondary terminal coordinate system H as seen from the primary terminal coordinate system A. This is calculated as follows: TAH = TA2TH2 - 1 TA2: Secondary terminal orientation in primary terminal coordinate system A TH2-1: Inverse matrix of the sub-terminal orientation TH2 in sub-terminal coordinate system H

[0121] As shown in Figure 11, the relationship between real space and virtual space can be made to coincide by using the inverse matrix of the orientation TAH of the secondary terminal coordinate system H as seen from the primary terminal coordinate system A, and the orientation THA of the primary terminal coordinate system A as seen from the secondary terminal coordinate system H. TAH: Attitude of the secondary terminal coordinate system H as seen from the primary terminal coordinate system A. THA: The attitude of the primary terminal coordinate system A as seen from the secondary terminal coordinate system H. TAH=THA-1

[0122] [Sub-terminal identification unit 105] The sub-terminal identification unit 105 identifies the sub-terminal whose displacement ΔTA2 in the main terminal coordinate system A is closest to the displacement ΔTH2 of the sub-terminal attitude TH2 in the sub-terminal coordinate system H, and associates the terminal identifier of that sub-terminal with the sub-terminal recognized from the video. (Secondary terminal ID) (Image-recognized object) Sub-terminal 21 <-> Sub-terminal a Sub-terminal 22 <-> Sub-terminal c Sub-terminal 23 <-> Sub-terminal b

[0123] The sub-terminal identification unit 105 may use a "matrix norm," which is the sum of the absolute values ​​of the differences between each element in each ΔTA and each ΔTH, to associate sub-terminals with the smallest displacement. Alternatively, the unit may use a "vector norm," which is the magnitude of the vector difference between the position v or slope r extracted from each ΔTA and the position v or slope r extracted from each ΔTH, or it may calculate a rotation axis vector from the slope r extracted from each ΔTA and the slope r extracted from each ΔTH, and use its norm. These norms are matched to the sub-terminals that are below the threshold and have the smallest size.

[0124] [Sub-terminal selection unit 106] The sub-terminal selection unit 106 selects one of the sub-terminals 2 by operator input. The terminal ID of the selected sub-terminal 2 is output to the visibility distance determination unit 11.

[0125] The sub-terminal selection unit 106 allows the operator to select an object of one sub-terminal from among multiple sub-terminals in the real space captured in the video footage by the camera. At this time, the display shows the object in the camera footage seen by the operator and the acquired terminal ID in association, making it easier for the operator to make a selection. In addition, a pre-set user identification name can be associated with each terminal ID and displayed on the display.

[0126] As shown in Figure 7 above, the sub-terminal selection unit 106 can select sub-terminal 2 in the video by pointing with the operator's finger when the operator's finger is captured by the camera (see, for example, Non-Patent Documents 7 and 8). Specifically, it determines the collision between the object in the video recognized as sub-terminal 2 and the object in the video recognized as the finger.

[0127] Furthermore, the aforementioned embodiment of selecting a secondary terminal from the perspective of the primary terminal can be implemented in reverse as an embodiment of selecting a primary terminal from the perspective of the secondary terminal. In other words, the only difference is whether the primary terminal is primary terminal 1 or secondary terminal 2.

[0128] As described in detail above, the method, sub-terminal, main terminal, and program of the present invention allow the main terminal to access the same server as the sub-terminal in real space.

[0129] As described above, various embodiments of the present invention can be easily modified, altered, and omitted within the scope of the technical concept and viewpoint of the present invention by those skilled in the art. The above description is merely illustrative and not intended to limit the invention in any way. The present invention is limited only to what is defined in the claims and their equivalents. [Explanation of Symbols]

[0130] 1 Main terminal 101 Sub-terminal detection unit 102 Sub-terminal attitude receiving unit 103 Object attitude detection unit 104 Attitude displacement detection unit 105 Sub-terminal identification section 106 Sub-terminal selection section 11. Visibility distance determination unit 12 Server Access Section 2 Sub-terminals 21. Visibility distance determination unit 3. Virtual Space Server 4 Search Server

Claims

1. A server access method for a system that enables communication between a main terminal and a sub-terminal by radio waves, The secondary terminal is accessing the virtual space server. The sub-terminal performs a first step in which it determines whether the radio waves received from the main terminal meet predetermined conditions, and if it determines that the conditions are true, it notifies the main terminal of a permit signal. The second step is that when the primary terminal receives a permission signal from the secondary terminal, it accesses the same virtual space server as the secondary terminal and enters the same virtual space. A server access method characterized by having the following features.

2. Regarding the predetermined conditions of the first step, the sub-terminal determines that the condition is true if the distance measured by the Time-of-Flight (TOF) radio waves is less than or equal to the first predetermined threshold. The server access method according to feature 1.

3. Regarding the predetermined conditions of the first step, the sub-terminal determines that the condition is true if the distance measured by radio waves reflected from the main terminal using a distance sensor or depth camera is less than or equal to the first predetermined threshold. The server access method according to feature 1.

4. Regarding the predetermined conditions of the first step, the sub-terminal determines that the condition is true if the radio wave reception strength is equal to or greater than the second predetermined threshold. The server access method according to feature 1.

5. A server access method for a system in which communication is possible between a primary terminal and a secondary terminal by radio waves, and in which a reference anchor for area tracking is located, The secondary terminal is accessing the virtual space server. The sub-terminal determines whether the area identifier of the main terminal, which it has stored in advance, is the same as the area identifier determined by radio waves received from the reference anchor. If it determines that this is true, it notifies the main terminal of a permit signal. (First step) The second step is that when the primary terminal receives a permission signal from the secondary terminal, it accesses the same virtual space server as the secondary terminal and enters the same virtual space. A server access method characterized by having the following features.

6. A server access method for a system where a primary terminal and a secondary terminal can communicate by radio waves and an access point is located, The secondary terminal is accessing the virtual space server. The sub-terminal determines whether the access point identifier of the main terminal, which it has stored in advance, is the same as the access point identifier included in the radio waves it has received from the access point. If it determines that this is true, it notifies the main terminal of a permission signal. (First step) The second step is that when the primary terminal receives a permission signal from the secondary terminal, it accesses the same virtual space server as the secondary terminal and enters the same virtual space. A server access method characterized by having the following features.

7. The system further includes a search server that stores user identifiers in association with the server addresses of the virtual space servers that the user corresponding to that user identifier is currently using. Regarding the first step, The secondary terminal notifies the primary terminal of the user identifier as an authorization signal. Regarding the second step, The main terminal sends the user identifier obtained from the secondary terminal to the search server. The search server responds to the main terminal with the server address corresponding to the user identifier. The main terminal accesses the virtual server based on the server address. A server access method according to any one of claims 1 to 6.

8. In the second step, the primary terminal further transmits the user identifier obtained from the secondary terminal to the virtual space server. The server access method according to feature 7.

9. Regarding the first step, The secondary terminal notifies the primary terminal of the server address as a permission signal. Regarding the second step, The main terminal accesses the virtual server based on the server address. A server access method according to any one of claims 1 to 6.

10. A server access method for a system that enables communication between a main terminal and a sub-terminal by radio waves, The secondary terminal is accessing the virtual space server. The main terminal performs a first step of determining whether the radio waves received from the sub-terminal meet predetermined conditions, When the primary terminal determines that the condition is true, it takes a second step of accessing the same virtual space server as the secondary terminal and entering the same virtual space. A server access method characterized by having the following features.

11. Regarding the predetermined conditions of the first step, the main terminal determines that the condition is true if the distance measured by the Time-of-Flight (TOF) radio waves is less than or equal to the first predetermined threshold. The server access method according to feature 10.

12. Regarding the predetermined conditions of the first step, the main terminal determines that the condition is true if the distance measured by radio waves reflected from the sub-terminal using a distance sensor or depth camera is less than or equal to the first predetermined threshold. The server access method according to feature 10.

13. Regarding the predetermined conditions of the first step, the main terminal determines that the condition is true if the radio wave reception strength is equal to or greater than the second predetermined threshold. The server access method according to feature 10.

14. A server access method for a system in which communication is possible between a primary terminal and a secondary terminal by radio waves, and in which a reference anchor for area tracking is located, The secondary terminal is accessing the virtual space server. The main terminal performs a first step of determining whether the area identifier of the sub-terminal, which has been stored in advance, is the same as the area identifier determined by radio waves received from the reference anchor. When the primary terminal determines that the condition is true, it takes a second step of accessing the same virtual space server as the secondary terminal and entering the same virtual space. A server access method characterized by having the following features.

15. A server access method for a system where a primary terminal and a secondary terminal can communicate by radio waves and an access point is located, The secondary terminal is accessing the virtual space server. The first step is to determine whether the access point identifier of the sub-terminal, which has been stored in advance, is the same as the access point identifier included in the radio waves received by the main terminal from the access point. The second step is that when the primary terminal determines that it is true, it accesses the same virtual space server as the secondary terminal and enters the same virtual space. A server access method characterized by having the following features.

16. The system further includes a search server that stores user identifiers in association with the server addresses of the virtual space servers that the user corresponding to that user identifier is currently using. Regarding the first step, The main terminal obtains the user identifier from the sub-terminal, Regarding the second step, The main terminal sends the user identifier obtained from the secondary terminal to the search server. The search server responds to the main terminal with the server address corresponding to the user identifier. The main terminal logs in to the virtual space server based on the server address. A server access method according to any one of claims 10 to 15.

17. In the second step, the primary terminal further transmits the user identifier obtained from the secondary terminal to the virtual space server. The server access method according to feature 16.

18. Regarding the first step, The primary terminal obtains the server address of the virtual space server from the secondary terminal. Regarding the second step, The main terminal accesses the virtual server based on the server address. A server access method according to any one of claims 10 to 15.

19. A secondary terminal that can communicate with the main terminal via radio waves and accesses a virtual space server, The system has a permission signal notification means that determines whether the radio waves received from the main terminal meet predetermined conditions, and when it determines that the condition is true, it notifies the main terminal of a permission signal. The primary terminal is made to access the same virtual space server as the secondary terminal, and to enter the same virtual space. A secondary terminal characterized by the following features.

20. A main terminal capable of communicating with a sub-terminal by radio waves, A means for determining whether radio waves received from a secondary terminal accessing a virtual space server meet predetermined conditions, When the visual distance determination means determines that it is true, the server access means accesses the same virtual space server as the secondary terminal and enters the same virtual space. A main terminal characterized by having the following features.

21. A program that enables communication via radio waves with a main terminal and functions a computer installed in a secondary terminal that is accessing a virtual space server, A permission signal notification means that determines whether the radio waves received from the main terminal meet predetermined conditions, and when it is determined to be true, notifies the main terminal of a permission signal. to make the computer work, The primary terminal is made to access the same virtual space server as the secondary terminal, and to enter the same virtual space. A program for a secondary terminal characterized by the following features.

22. A program that enables the operation of a computer installed in a main terminal that can communicate with a secondary terminal by radio waves, A means for determining whether radio waves received from a secondary terminal accessing a virtual space server meet predetermined conditions, When the visual distance determination means determines that it is true, the server access means accesses the same virtual space server as the secondary terminal and enters the same virtual space. A program for a main terminal characterized by enabling the computer to function.

Images