Information processing apparatus, information processing method, and recording medium

Abstract

The disclosed information processing device includes a control unit for controlling a display terminal that arranges virtual objects in a real space using a coordinate system associated with the real space. The control unit determines whether to update the first spatial information of the first display terminal and the second spatial information of the second display terminal for determining the position of the virtual objects, based on a first operating state of a first display terminal and a second operating state of a second display terminal that shares virtual objects with the first display terminal.

Description

Technical Field

[0001] This disclosure relates to information processing devices, information processing methods, and recording media. Background Technology

[0002] In recent years, a technology called Augmented Reality (AR) has attracted attention, in which additional information (hereinafter also referred to as virtual objects) is overlaid on real space, and the result is presented to the user. The presentation of information using AR technology utilizes display terminals such as head-mounted displays (HMDs). In some cases, the position in a global coordinate system is assigned to the presented virtual object. In this case, the display terminal estimates its own position in the global coordinate system and then determines the display position of the virtual object, such that the virtual object is displayed at the position assigned to it in the global coordinate system.

[0003] Among the various existing methods for estimating the position of a display terminal, Patent Document 1 below discloses a method using Simultaneous Localization and Mapping (SLAM) technology based on keyframes.

[0004] Reference List

[0005] Patent Literature

[0006] Patent Document 1: JP 2016-528476 A Summary of the Invention

[0007] Technical issues

[0008] The estimation result of the display terminal's own position may include errors. This may result in an error between the position assigned to the virtual object in the global coordinate system and the position of the virtual object displayed by the display terminal in the global coordinate system. Here, when the same virtual object is to be displayed on multiple display terminals, variations in the accuracy and results of the estimation of its own position on each display terminal cause differences in the position of the virtual object in the global coordinate system displayed on each display terminal.

[0009] In view of this situation, the present disclosure proposes novel and improved information processing apparatus, information processing methods and recording media that can reduce the difference in display position when the same virtual object is displayed on multiple display terminals.

[0010] Solution to the problem

[0011] According to this disclosure, an information processing apparatus is provided, which includes a control unit that controls a display terminal to arrange virtual objects in a real space using a coordinate system associated with the real space. The control unit determines whether to update the first spatial information of the first display terminal and the second spatial information of the second display terminal for determining the position of the virtual object based on a first operating state of a first display terminal and a second operating state of a second display terminal that shares the virtual object with the first display terminal.

[0012] Furthermore, according to this disclosure, an information processing apparatus is provided, comprising: an output unit including a first output device for presenting a virtual object to a first user; and a control unit for determining the position of the virtual object to be arranged in a coordinate system associated with real space, wherein the control unit determines whether to update the first spatial information of the first output device and the second spatial information of the second output device for determining the position of the virtual object based on a first operating state of the first output device and a second operating state of the second output device, wherein the second output device is disposed externally and shares the virtual object with the first output device.

[0013] Furthermore, according to this disclosure, an information processing method to be executed by a processor is provided, the method comprising: controlling a display terminal that arranges virtual objects in real space using a coordinate system associated with real space; and determining whether to update first spatial information of the first display terminal and second spatial information of the second display terminal for determining the position of the virtual objects based on a first operating state of a first display terminal and a second operating state of a second display terminal that shares virtual objects with the first display terminal.

[0014] Furthermore, according to this disclosure, a recording medium is provided that records a program for a computer to perform processing, the processing including: controlling a display terminal to arrange virtual objects in real space using a coordinate system associated with real space; and determining whether to update first spatial information of the first display terminal and second spatial information of the second display terminal for determining the position of the virtual objects based on a first operating state of a first display terminal and a second operating state of a second display terminal that shares virtual objects with the first display terminal. Attached Figure Description

[0015] Figure 1 This is a diagram illustrating an example system configuration of an information processing system according to an embodiment of the present disclosure.

[0016] Figure 2 This is a block diagram illustrating an example of the hardware configuration of an AR terminal according to an implementation method.

[0017] Figure 3 This is a block diagram illustrating an example of the functional configuration of an information processing system according to an embodiment.

[0018] Figure 4 This is a flowchart illustrating the standard positioning process.

[0019] Figure 5 This is a flowchart illustrating the positioning process according to an embodiment of the present disclosure.

[0020] Figure 6 This is a flowchart illustrating the process of keyframe update determination according to an implementation method.

[0021] Figure 7 This is a sequence diagram illustrating the processing flow in an information processing system according to an embodiment.

[0022] Figure 8A This is a diagram illustrating the positional relationships between multiple users according to a first specific example of an implementation.

[0023] Figure 8B This is a diagram showing the views of various users according to a first specific example of an implementation.

[0024] Figure 9A This is a diagram illustrating the positional relationships between multiple users according to a second specific example of an implementation.

[0025] Figure 9B This is a diagram showing the views of various users according to a second specific example of the implementation.

[0026] Figure 10A This is a diagram illustrating the positional relationships between multiple users according to a third specific example of an implementation.

[0027] Figure 10B This is a diagram showing the views of various users according to a third specific example of the implementation.

[0028] Figure 11 This is a graph showing the time-series changes of keyframes according to a fourth specific example of the implementation.

[0029] Figure 12 This is a flowchart illustrating the keyframe update determination process according to the first modified example of the implementation.

[0030] Figure 13 This is a flowchart illustrating the keyframe update determination process according to a second modified example of the implementation.

[0031] Figure 14 This is a block diagram illustrating an example of the hardware configuration of an information processing apparatus according to an embodiment. Detailed Implementation

[0032] In the following, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configuration will be indicated by the same reference numerals, and redundant descriptions will be omitted.

[0033] Note that the descriptions will be provided in the following order.

[0034] 1. First Implementation Method

[0035] 2. Second Implementation Method

[0036] 3. Hardware Configuration Example

[0037] 4. Supplement

[0038] <<1. First Implementation Method>>

[0039] The technology according to embodiments of this disclosure relates to an information processing apparatus for controlling the display of the same virtual object on multiple display terminals. In the following, as an example, it is assumed that the display terminals are implemented as AR terminals and the information processing apparatus is implemented as a server.

[0040] <1-1 System Configuration Example>

[0041] First, refer to Figure 1 and Figure 2 An example system configuration of an information processing system described according to an embodiment of this disclosure. Figure 1 This is a diagram illustrating an example system configuration of an information processing system according to an embodiment of this disclosure. (See diagram for example.) Figure 1 As shown, the information processing system 1000 according to this embodiment includes an AR terminal 10, a server 20, and a network 30. Note that in this embodiment, as... Figure 1 As shown, assume user 12a wears AR terminal 10a (first display terminal), while user 12b wears AR terminal 10b (second display terminal). Furthermore, AR terminal 10a and AR terminal 10b perform shared display of the same virtual object.

[0042] (1) AR terminal 10

[0043] AR terminal 10 is a terminal that displays virtual objects. According to this embodiment, AR terminal 10 is implemented, for example, by a head-mounted display (HMD). Examples of HMDs suitable for AR terminal 10 include perspective HMDs, video perspective HMDs, and retinal projection HMDs.

[0044] The see-through HMD uses, for example, a translucent mirror or a transparent light guide plate to hold a virtual image optical system, in which transparent light guide units are formed, in front of the user 12's eyes, so as to display an image within the virtual image optical system. Therefore, the user 12 wearing the see-through HMD can view the external scene while viewing the image displayed within the virtual image optical system. With this configuration, the see-through HMD can also, for example, based on the recognition results of at least one of the see-through HMD's position or posture, superimpose an image of a virtual object onto an optical image of a real object located in real space using AR technology. A specific example of a see-through HMD is a device called a glasses-type wearable device, which has a portion configured as a virtual image optical system corresponding to the lenses of glasses.

[0045] When worn on the head or face of user 12, the video perspective HMD is worn to cover the eyes of user 12, so that a display unit, such as a display component, can be held in front of the eyes of user 12. Furthermore, the video perspective HMD includes an imaging unit for capturing images of the surrounding scene, and the display unit displays the image of the scene in front of user 12 captured by the imaging unit. With this configuration, user 12 wearing the video perspective HMD cannot easily view the external scene directly, but can still confirm the external scene through the image displayed on the display unit. Additionally, the video perspective HMD can, for example, overlay virtual objects onto the image of the external scene based on AR technology, according to the recognition results of at least one of the video perspective HMD's position or posture.

[0046] The retinal projection HMD is configured such that the projector is held in front of the user's eyes, and an image is projected from the projector toward the user's eyes, so that the image is superimposed on the external scene. More specifically, the retinal projection HMD projects the image from the projector directly onto the retina of the user's eyes, so that the image can be formed on the retina. With this configuration, even nearsighted or farsighted users can view clearer images. Furthermore, the user wearing the retinal projection HMD can view the external scene while viewing the image projected from the projector. With this configuration, the retinal projection HMD can also, for example, based on the recognition results of at least one of the retinal projection HMD's position or posture, superimpose virtual object images onto optical images of real objects located in real space using AR technology.

[0047] In addition to the examples mentioned above, there are also HMDs known as immersive HMDs. Similar to video perspective HMDs, immersive HMDs are worn to cover the eyes of the user 12, so that the display unit, such as the display component, can be held in front of the user 12's eyes. Therefore, the user 12 wearing an immersive HMD cannot directly view the external scene (i.e., the scene of the real world), and only the image displayed on the display unit enters the field of vision. With this configuration, immersive HMDs can give the user 12 viewing the image a sense of immersion. Therefore, immersive HMDs are suitable for situations, for example, where information is presented primarily based on virtual reality (VR) technology.

[0048] Note that the AR terminal 10 is not limited to the HMD described above. For example, the AR terminal 10 can be a terminal device with a display unit, such as a smartphone, tablet, wearable device, or agent device.

[0049] AR terminal 10 connects to server 20 via network 30 and can send and receive information from server 20.

[0050] (Hardware configuration of AR Terminal 10)

[0051] Here, we will refer to Figure 2 Describes an example of the hardware configuration of AR terminal 10. Figure 2 This is a block diagram illustrating an example hardware configuration of an AR terminal 10 according to an embodiment of the present disclosure. Note that the hardware configuration example of the AR terminal 10 is not limited to this example.

[0052] like Figure 2 As shown, the AR terminal 10 includes a central processing unit (CPU) 100, a read-only memory (ROM) 102, and a random access memory (RAM) 104. Furthermore, the AR terminal 10 includes a communication interface (I / F) controller 106, an input image controller 108, a camera 110, a surrounding environment information input controller 112, a surrounding environment information detection sensor 114, an output image controller 116, and a display 118.

[0053] The CPU 100 functions as, for example, an arithmetic processing device or a control device, and controls all or part of the operation of each component based on various programs recorded in the ROM 102, RAM 104, or storage devices. The ROM 102 is a device that stores programs loaded by the CPU 100, data used for calculation, etc. The RAM 104 temporarily or permanently stores, for example, programs loaded by the CPU 100, various parameters that change appropriately during program execution, etc.

[0054] The communication I / F controller 106 is a controller that performs communication-related controls. For example, the communication I / F controller 106 controls communication with external devices via network 30.

[0055] The input image controller 108 is a controller that performs controls related to the input of an image (still image or moving image). For example, the input image controller 108 controls the input of photographic images captured by the camera 110. Furthermore, the input image controller 108 can control the imaging processing performed by the camera 110.

[0056] Camera 110 is an imaging device that images the surrounding environment of AR terminal 10. Through the imaging device, AR terminal 10 can acquire photographic images indicating the surrounding environment of user 12. There is no specific limitation on the number of cameras 110 provided in AR terminal 10; any number of cameras 110 can be provided. For example, the AR terminal 10 according to this embodiment is provided with a right-eye camera and a left-eye camera.

[0057] The surrounding environment information input controller 112 is a controller that performs control related to the input of information indicating the state around the AR terminal 10 (hereinafter also referred to as "surrounding environment information"). For example, the surrounding environment information input controller 112 controls the input of surrounding environment information acquired by the surrounding environment information detection sensor 114. In addition, the surrounding environment information input controller 112 can control the surrounding environment information acquisition processing performed by the surrounding environment information detection sensor 114.

[0058] The surrounding environment information detection sensor 114 is a sensor device that senses surrounding environment information. For example, the surrounding environment information detection sensor 114 may acquire photographic images from a camera that indicate the conditions surrounding the AR terminal 10, serving as surrounding environment information. Note that the camera may be the aforementioned camera 110, or it may be a different camera than camera 110.

[0059] The output image controller 116 is a controller that performs controls related to the output of an image. For example, the output image controller 116 controls the output of an output image on the display 118. Furthermore, the output image controller 116 can control the processing related to the generation of the output image output from the display 118.

[0060] Display 118 is a device for outputting images. Examples of display 118 include CRT display devices, liquid crystal display devices, plasma display devices, and EL display devices. Note that there is no specific limitation on the number of displays 118 provided in AR terminal 10, and any number of displays can be provided. For example, AR terminal 10 according to this embodiment is provided with a right-eye display and a left-eye display.

[0061] (2) Server 20

[0062] Server 20 is an information processing device with the function of controlling the display of virtual objects in multiple AR terminals 10. For example, server 20 performs a determination process for controlling the display of virtual objects in AR terminals 10 based on information indicating the operating state of AR terminals 10. Specifically, server 20 performs a process to determine whether to update keyframes (spatial information) used to determine the position of virtual objects based on the operating state of AR terminal 10a (first operating state) and the operating state of AR terminal 10b (second operating state). Subsequently, AR terminals 10 determine and display the position of virtual objects while sequentially updating keyframes according to the determination made by server 20. In the following text, the keyframe used to determine the position of virtual objects will also be referred to as "position determination keyframe". In the following text, the process of determining whether to update the position determination keyframe will also be referred to as "keyframe update determination process". If server 20 has determined to update the position determination keyframe in the keyframe update determination process, both the position determination keyframe (first spatial information) of AR terminal 10a and the position determination keyframe (second spatial information) of AR terminal 10b are updated.

[0063] Keyframes may include photographic images, feature points of the photographic images, and information indicating the camera's own position when the photographic image is captured. For example, each time the camera 110 of the AR terminal 10 captures an image, a keyframe is stored in a storage unit included in the AR terminal 10. In the following text, multiple keyframes stored in the storage unit of the AR terminal 10 will also be referred to as a "keyframe group".

[0064] Each of the multiple AR terminals 10 acquires candidate keyframes (hereinafter referred to as "update destination candidate keyframes") from a keyframe group to be used for updating the location determination keyframes. For example, AR terminal 10a acquires its update destination candidate keyframes (third spatial information) from its keyframe group. Similarly, AR terminal 10b acquires its update destination candidate keyframes (fourth spatial information) from its keyframe group. Server 20 receives the update destination candidate keyframes acquired by each of the multiple AR terminals 10 and then performs keyframe update decision processing based on the received update destination candidate keyframes.

[0065] AR terminal 10 acquires updated destination candidate keyframes based on information indicating the operating state of AR terminal 10. An example of the operating state of AR terminal 10 is its pose. The pose of AR terminal 10 may also include information indicating its own position. An example of information indicating the operating state is a photographic image acquired by camera 110 included in AR terminal 10. For example, AR terminal 10 performs a match between feature points of a photographic image captured immediately before the keyframe update determination process and feature points of the photographic images included in each keyframe of the keyframe group, and then acquires the successfully matched keyframes as updated destination candidate keyframes.

[0066] Server 20 is connected to AR terminal 10 via network 30 and can send and receive information from AR terminal 10. For example, server 20 receives updated destination candidate keyframes from AR terminals 10a and 10b. Furthermore, server 20 sends the decision results obtained in the decision processing to AR terminals 10a and 10b.

[0067] (3) Network 30

[0068] Network 30 has the function of connecting AR terminal 10 and server 20 to each other. Furthermore, network 30 may have the function of connecting multiple AR terminals 10 to each other. For example, network 30 connects AR terminal 10a and AR terminal 10b to each other.

[0069] Network 30 may include public networks such as the Internet, telephone networks, and satellite communication networks, or various local area networks (LANs) and wide area networks (WANs) including Ethernet (registered trademark). Furthermore, network 30 may include private networks, such as Internet Protocol Virtual Private Networks (IP-VPNs). Network 30 may also include wireless communication networks such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0070] <1-2. Function Configuration Example>

[0071] The system configuration example according to this embodiment has been described above. Next, we will refer to... Figure 3 This document describes a functional configuration example of an information processing system 1000 according to an embodiment of the present disclosure. Figure 3 This is a block diagram illustrating an example of the functional configuration of an information processing system 1000 according to an embodiment of the present disclosure.

[0072] <1-2-1. AR Terminal Functional Configuration>

[0073] like Figure 3As shown, the AR terminal 10 according to this embodiment includes a communication unit 120, a sensor unit 130, a control unit 140, a storage unit 150, and an output unit 160.

[0074] (1) Communication Unit 120

[0075] The communication unit 120 has the function of communicating with external devices. For example, in communication with external devices, the communication unit 120 outputs information received from the external devices to the control unit 140. For example, in communication with the server 20 via the network 30, the communication unit 120 receives a determination result from the server 20 and then outputs the determination result to the control unit 140.

[0076] For example, in communication with an external device, the communication unit 120 sends information input from the control unit 140 to the external device. As an example, the communication unit 120 sends updated destination candidate keyframes to the server 20.

[0077] (2) Sensor unit 130

[0078] The sensor unit 130 has the function of sensing information for processing in the control unit 140. After sensing, the sensor unit 130 outputs the sensing information acquired by various sensors to the control unit 140.

[0079] Sensor unit 130 may include various sensor devices. As examples, sensor unit 130 may include a camera, depth sensor, accelerometer, gyroscope, etc. The sensor devices included in sensor unit 130 are not limited to the examples above and may include any other sensor devices.

[0080] A camera is an imaging device that includes a lens system, a drive system, and an imaging element, and captures images as photographic images, such as an RGB camera. Such an imaging device can be included in the AR terminal 10 to capture images of the surrounding environment of the AR terminal 10. With this configuration, the sensor unit 130 can acquire photographic images of the environment surrounding the AR terminal 10. Note that when the user 12 wears or carries the AR terminal 10, the photographic images of the surrounding environment of the AR terminal 10 acquired by the sensor unit 130 are also photographic images of the user 12's surrounding environment.

[0081] A depth sensor is a device that acquires depth information, such as an infrared distance measuring device, an ultrasonic distance measuring device, a laser imaging detection and ranging (LiDAR) system, or a stereo camera. Sensor unit 130 can acquire the location information of AR terminal 10 based on the information acquired by the depth sensor. When user 12 wears or carries AR terminal 10, the location information of AR terminal 10 acquired by sensor unit 130 is also the location information of user 12.

[0082] An accelerometer is a device for acquiring acceleration. For example, an accelerometer measures acceleration, which is the rate of change of velocity of a moving target. A gyroscope is a device for acquiring angular velocity. For example, a gyroscope measures angular velocity, which is the amount of change in the target's posture. In the following text, the information acquired by the accelerometer and gyroscope will also be referred to as "inertial information." With this configuration, sensor unit 130 can acquire the inertial information of AR terminal 10. When user 12 wears or carries AR terminal 10, the inertial information acquired by sensor unit 130 is also the inertial information of user 12.

[0083] (3) Control Unit 140

[0084] The control unit 140 has the function of controlling the entire operation of the AR terminal 10. To achieve this function, such as... Figure 3 As shown, the control unit 140 includes a surrounding condition information acquisition unit 1402, a surrounding condition information management unit 1404, a spatial information acquisition unit 1406, a spatial information management unit 1408, an image output control unit 1410, and an output image generation unit 1412.

[0085] (3-1) Surrounding Condition Information Acquisition Unit 1402

[0086] The surrounding environment information acquisition unit 1402 has the function of controlling the acquisition of surrounding environment information about the AR terminal 10. For example, the surrounding environment information acquisition unit 1402 controls the operation of the sensor unit 130 and causes the sensor unit 130 to acquire surrounding environment information. In addition, the surrounding environment information acquisition unit 1402 can control the operation of the communication unit 120 and can acquire surrounding environment information from external devices connected to the AR terminal 10 via the communication unit 120. Examples of external devices include sensor devices installed outside the AR terminal 10, server 20, etc.

[0087] (3-2) Surrounding Conditions Information Management Unit 1404

[0088] The surrounding conditions information management unit 1404 has the function of managing surrounding conditions information. For example, the surrounding conditions information management unit 1404 performs update processing on the database (hereinafter also referred to as "surrounding conditions information DB") that stores surrounding conditions information included in the storage unit 150. For example, the surrounding conditions information management unit 1404 causes the storage unit 150 to add the surrounding conditions information acquired by the surrounding conditions information acquisition unit 1402 to the surrounding conditions information DB.

[0089] (3-3) Spatial Information Acquisition Unit 1406

[0090] The spatial information acquisition unit 1406 has the function of controlling the acquisition of information related to the spatial information of the AR terminal 10. For example, the spatial information acquisition unit 1406 controls the operation of the sensor unit 130 and causes the sensor unit 130 to acquire information related to the spatial information. As an example, the spatial information acquisition unit 1406 controls the camera installed in the sensor unit 130 to capture photographic images indicating the operating state of the AR terminal 10.

[0091] (3-4) Spatial Information Management Unit 1408

[0092] The spatial information management unit 1408 has the function of managing spatial information. For example, the spatial information management unit 1408 controls the processing related to spatial information. For example, the spatial information management unit 1408 performs processing such as generating query images, acquiring updated destination candidate keyframes, updating location determination keyframes, initial pose estimation, and final pose estimation.

[0093] (Generation of query image)

[0094] The spatial information management unit 1408 performs processing related to the generation of the query image. For example, the spatial information management unit 1408 generates the query image based on photographic images acquired by the spatial information acquisition unit 1406. The query image is used when obtaining updated destination candidate keyframes from the keyframe group.

[0095] When generating the query image, the spatial information management unit 1408 estimates the camera's own position (the AR terminal 10's own position) at the time of imaging based on the photographic image. In addition, the spatial information management unit 1408 performs feature point detection and feature quantity description on the photographic image, and then obtains two-dimensional feature points.

[0096] (Updating the acquisition of candidate destination keyframes)

[0097] The spatial information management unit 1408 acquires updated destination candidate keyframes for the AR terminal 10 based on the operational state of the AR terminal 10. For example, the spatial information management unit 1408 uses a query image generated from a photographic image indicating the operational state of the AR terminal 10 captured by the camera of the sensor unit 130 to search for updated destination candidate keyframes from a keyframe group stored in the storage unit 150. While searching for updated destination candidate keyframes, the spatial information management unit 1408 performs a matching process between two-dimensional feature points acquired during the generation of the query image and the two-dimensional feature points included in each keyframe of the keyframe group. The spatial information management unit 1408 then acquires the keyframes where the two-dimensional feature points are successfully matched as updated destination candidate keyframes.

[0098] (Update of location-determining keyframes)

[0099] The spatial information management unit 1408 performs the location determination keyframe update process based on the determination result of the keyframe update determination process in the server 20. When the keyframe update determination process has determined to update the location determination keyframe, the spatial information management unit 1408 will update the location determination keyframe by using the updated destination candidate keyframe. In contrast, when the keyframe update determination process has determined not to update the location determination keyframe, the spatial information management unit 1408 will not update the location determination keyframe.

[0100] (Initial pose estimation)

[0101] The spatial information management unit 1408 estimates the initial pose of the AR terminal 10 based on the position determination keyframe. The pose indicates the position and orientation of the AR terminal 10. Furthermore, the initial pose indicates the position and orientation of the AR terminal 10 when the camera captures a photographic image. For example, the spatial information management unit 1408 estimates the initial pose of the AR terminal 10 based on information indicating its own position associated with the position determination keyframe.

[0102] At this time, the spatial information management unit 1408 performs feature point matching between the two-dimensional feature points of the photographic image captured by the camera of the sensor unit 130 and the two-dimensional feature points of the location determination keyframe. When the feature point matching is successful, the spatial information management unit 1408 uses the information indicating its own position included in the location determination keyframe as the initial pose of the AR terminal 10. When the feature point matching is unsuccessful, the spatial information management unit 1408 updates the location determination keyframe and repeats the feature point matching until the feature point matching is successful.

[0103] (Final pose estimation)

[0104] The spatial information management unit 1408 corrects the estimated initial pose to estimate the final pose of the AR terminal 10. The final pose indicates the position and pose of the AR terminal 10 at the current time. Regarding the final pose, for example, the spatial information management unit 1408 performs image matching between photographic images captured by the camera of the sensor unit 130 and photographic images included in the location determination keyframe. When a difference is detected between the two images as a result of the image matching, the spatial information management unit 1408 corrects the initial pose based on the difference and determines the corrected pose as the final pose of the AR terminal 10.

[0105] (3-5) Image Output Control Unit 1410

[0106] The image output control unit 1410 has the function of controlling the output of images. For example, the image output control unit 1410 causes the output unit 160 to output the output image generated by the output image generation unit 1412. In the case where the AR terminal 10 is provided with a right-eye display and a left-eye display, the image output control unit 1410 causes each of the displays to output an output image in a format corresponding to each of the displays.

[0107] (3-6) Output image generation unit 1412

[0108] The output image generation unit 1412 has the function of generating images. For example, the output image generation unit 1412 generates an output image to be output by the output unit 160. For example, based on the final pose of the AR terminal 10 estimated by the spatial information management unit 1408, the output image generation unit 1412 generates an output image in which virtual objects are arranged.

[0109] (4) Storage unit 150

[0110] Storage unit 150 has the function of storing information related to processing in AR terminal 10. For example, under the control of surrounding situation information management unit 1404, storage unit 150 stores the surrounding situation information acquired by surrounding situation information acquisition unit 1402 into surrounding situation information DB. In addition, storage unit 150 stores pre-prepared map DB (keyframe group) into spatial information DB.

[0111] Note that the information to be stored in storage unit 150 is not limited to this example. For example, storage unit 150 can store programs such as various applications.

[0112] (5) Output unit 160

[0113] The output unit 160 has the function of outputting an image. For example, under the control of the image output control unit 1410, the output unit 160 outputs the output image generated by the output image generation unit 1412.

[0114] <1-2-2. Server Functional Configuration>

[0115] like Figure 3 As shown, the server 20 according to this embodiment includes a communication unit 210, a control unit 220, and a storage unit 230.

[0116] (1) Communication Unit 210

[0117] The communication unit 210 has the function of communicating with external devices. For example, in communication with external devices, the communication unit 210 outputs information received from the external devices to the control unit 220. As an example, in communication with the AR terminal 10 via the network 30, the communication unit 210 receives updated destination candidate keyframes from the AR terminal 10 and outputs the received updated destination candidate keyframes to the control unit 220.

[0118] For example, in communication with an external device, the communication unit 210 sends information input from the control unit 220 to the external device. For example, the communication unit 210 sends the determination result to the AR terminal 10.

[0119] (2) Control Unit 220

[0120] The control unit 220 is an information processing device that controls the entire operation of the server 20. For example, the control unit 220 has the function of performing processing related to controlling the arrangement of virtual objects in the AR terminal 10. As an example of processing, the control unit 220 performs keyframe update determination processing.

[0121] (Keyframe update determination and processing)

[0122] In the keyframe update determination process, the control unit 220 determines whether to update the position determination keyframes of AR terminal 10a and AR terminal 10b based on the operating states of AR terminal 10a and AR terminal 10b. The determination of whether to update the position determination keyframes of AR terminal 10a and AR terminal 10b is based on whether the operating states of AR terminal 10a and AR terminal 10b are similar.

[0123] The similarity of the operating states indicates that the viewing direction of user 12a is similar to that of user 12b. Therefore, when the operating states of AR terminal 10a and AR terminal 10b are similar, the control unit 220 determines not to update the position determination keyframe of AR terminal 10a or AR terminal 10b. By not updating the keyframe, AR terminals 10a and 10b can continue to use at least the position determination keyframe (more precisely, the keyframe obtained under similar operating states) when their operating states are similar. This allows for a reduction in the accuracy of the self-position estimation and the variation in the self-position estimation results of each of the AR terminals 10, and simultaneously reduces the differences in the display positions of the same virtual objects among multiple AR terminals 10.

[0124] In contrast, the dissimilarity in operating states indicates a dissimilarity between the viewing directions of user 12a and user 12b. Therefore, when the operating states of AR terminal 10a and AR terminal 10b are dissimilarity, the control unit 220 determines and updates the position determination keyframes of AR terminal 10a and AR terminal 10b. With this configuration, virtual objects can be displayed at positions corresponding to the viewing directions of each user.

[0125] - Similarity-based determination

[0126] Based on a comparison of the similarity between the operating states of AR terminal 10a and AR terminal 10b calculated by control unit 220 and a first threshold, it is determined whether the operating states of AR terminal 10a and AR terminal 10b are similar. When the similarity is equal to or greater than the first threshold, control unit 220 determines that the operating states of AR terminal 10a and AR terminal 10b are similar. Conversely, when the similarity is not equal to or greater than the first threshold, control unit 220 determines that the operating states of AR terminal 10a and AR terminal 10b are not similar.

[0127] The control unit 220 calculates similarity based on location determination keyframes acquired by AR terminal 10a in the operating state of AR terminal 10a and location determination keyframes acquired by AR terminal 10b in the operating state of AR terminal 10b. Specifically, the control unit 220 performs feature point matching between feature points included in the location determination keyframes acquired by AR terminal 10a and feature points included in the location determination keyframes acquired by AR terminal 10b, and then calculates similarity based on the number of successfully matched feature points. For example, the control unit 220 calculates a score as a specific value indicating the similarity. For example, the control unit 220 calculates the number of successfully matched feature points as a score.

[0128] Note that the method for calculating similarity is not limited to this example. For instance, the similarity can be calculated as the matching degree of the position and pose matrices (3×4 matrices) in the 3D coordinate system of the camera corresponding to the updated destination candidate keyframes. Note that the matching degree of the matrices is the sum of the squares of each element of the matrix.

[0129] Furthermore, there are no particular limitations on the value set for the first threshold, and it can be set to any value. For example, when it is desirable to further reduce the difference in display position when the same virtual object is displayed on multiple AR terminals, it is desirable to set the first threshold to high.

[0130] Furthermore, the first threshold can be changed dynamically. For example, the control unit 220 changes the first threshold based on the positional relationship between AR terminal 10a and AR terminal 10b (i.e., the positional relationship between user 12a and user 12b).

[0131] When users 12a and 12b are in close proximity, they are likely to perceive a difference in the display position of the virtual object. Therefore, when users 12a and 12b are in close proximity, the control unit 220 can set the first threshold to high. By setting this threshold, the difference in display position is reduced, making it less likely that users 12a and 12b will perceive the device at the display position of the virtual object.

[0132] In contrast, when users 12a and 12b are geographically distant, they are unlikely to perceive the difference in the display position of the virtual object. Therefore, when users 12a and 12b are geographically distant, the control unit 220 can set the first threshold to a low value. With this setting, although the difference in display position increases, it can maintain a state where the device at the display position of the virtual object is unlikely to be perceived by users 12a and 12b.

[0133] - Determining the maximum similarity

[0134] Whether to update the position determination keyframe in the keyframe update decision process can be determined based on whether the similarity score is at the maximum value. For example, the control unit 220 also determines whether to update the position determination keyframe of AR terminal 10a and the position determination keyframe of AR terminal 10b based on whether the similarity exceeds the maximum value (third threshold).

[0135] The maximum value is the maximum similarity calculated within a time period that meets predetermined conditions. The predetermined time period is, for example, the organization of continuously determining without updating the location determination keyframes in the movement distance determination described below.

[0136] When the similarity result exceeds the maximum value, the control unit 220 determines to update the location determination keyframe of AR terminal 10a and AR terminal 10b. When the similarity does not exceed the maximum value, the control unit 220 determines not to update the location determination keyframe of AR terminal 10a or AR terminal 10b.

[0137] The state where the similarity of the updated destination candidate keyframes is at its maximum indicates that the similarity of the updated destination candidate keyframes is higher than the similarity of the currently used location-determining keyframes. Therefore, by updating the currently used location-determining keyframes using the updated destination candidate keyframes with the maximum similarity, the discrepancy in the display position of virtual objects can be reduced.

[0138] - Determining the distance traveled

[0139] In the keyframe update determination process, the decision to update the location determination keyframe can be based on the distance the AR terminal 10 has moved since the last update of the location determination keyframe. For example, the control unit 220 calculates the distance the AR terminal 10a has moved since the last update of the location determination keyframe and the distance the AR terminal 10b has moved since the last update of the location determination keyframe.

[0140] Note that the movement distance is the distance between the position of the AR terminal 10 at the time of the last update of the location determination keyframe and the position where the current keyframe update determination process is being executed. For example, the movement distance can be calculated based on the difference between the AR terminal 10's own position at the time of the last update of the location determination keyframe and the AR terminal 10's own position when the current keyframe update determination process is being executed. Therefore, the control unit 220 obtains the AR terminal 10's own position at the time of the location determination keyframe update and stores its own position in the storage unit 230.

[0141] After calculating the movement distance, the control unit 220 compares the calculated movement distance with a second threshold and determines whether to update the position determination keyframe based on the comparison result.

[0142] For example, when at least one of the movement distances of AR terminal 10a and AR terminal 10b is a second threshold or greater, control unit 220 determines to update the location determination keyframes of AR terminal 10a and AR terminal 10b. Conversely, if none of the movement distances of AR terminal 10a and AR terminal 10b are a second threshold or greater, control unit 220 determines not to update the location determination keyframes of either AR terminal 10a or AR terminal 10b.

[0143] Note that if the movement distance of at least one AR terminal 10 among the multiple AR terminals 10 is a second threshold or greater, the control unit 220 determines to update the position determination keyframes of all AR terminals 10.

[0144] Furthermore, when comparing the results of similarity-based determination processing with those of distance-based determination processing, the results of distance-based determination processing can be given higher priority. For example, even if multiple AR terminal 10 candidate keyframes for updating the destination are determined to be similar in the similarity-based determination processing, the control unit 220 will still determine the updated location determination keyframe if the distance is determined to be a second threshold or greater in the distance-based determination processing.

[0145] This is because the AR terminal 10 may move a predetermined distance or more, causing errors in its own position estimation. Even when multiple AR terminals 10 have similar updated destination candidate keyframes, there is a possibility that the display position of virtual objects may differ due to errors.

[0146] Note that, for example, the self-position estimation in the AR terminal uses tracking processing via SLAM. The tracking processing calculates the movement distance of the AR terminal 10 based on the coordinate position of the camera corresponding to the keyframe and the movement distance calculated based on measurements from sensors such as an inertial measurement unit. When the movement distance of the AR terminal 10 is a second threshold or greater, the self-position estimation result of the AR terminal 10 is highly likely to depend on the movement distance portion calculated based on measurements from sensors such as an inertial measurement unit. Furthermore, the estimation result is likely to include a significant amount of drift error.

[0147] Therefore, if the movement distance of at least one of the AR terminals 10 is a second threshold or greater, the control unit 220 determines to update the location determination keyframes of all AR terminals 10. Furthermore, even when the candidate destination keyframes of multiple AR terminals 10 are determined to be similar in the similarity-based determination process, the control unit 220 determines whether to update the location determination keyframes based on the result of the movement distance-based determination process. This configuration allows for a reduction in the difference in the display position of virtual objects.

[0148] (3) Storage unit 230

[0149] Storage unit 230 has the function of storing information related to processing in server 20. For example, storage unit 230 stores the maximum value of the score calculated by control unit 220. In addition, storage unit 230 stores programs such as various applications and data. Note that the information stored in storage unit 230 is not limited to the above examples.

[0150] <1-3. Processing Examples>

[0151] The functional configuration example according to this embodiment has been described above. Next, we will refer to... Figures 4 to 7 An example of processing in an information processing system 1000 according to an embodiment of the present disclosure is described.

[0152] (1) Positioning processing

[0153] First, refer to Figure 4 and Figure 5 Describe the location processing. Figure 4 This is a flowchart illustrating the standard positioning process. Figure 5 This is a flowchart illustrating the location processing flow according to an embodiment of the present disclosure. Note that standard location processing should be performed by a standard AR terminal. Furthermore, the location processing according to this embodiment should be performed by the AR terminal 10 according to this embodiment.

[0154] (1-1) Standard positioning processing

[0155] like Figure 4 As shown, the standard AR terminal first performs feature point detection and feature quantity description on the photographed image, and then acquires two-dimensional feature points (S102). Next, the standard AR terminal uses the acquired two-dimensional feature points to search for keyframes on the keyframe group of the map DB stored in the standard AR terminal's storage unit, thereby acquiring keyframes to be used for generating the output image (S104). Next, the standard AR terminal performs feature point matching between the feature points of the acquired keyframes and the feature points of the photographed image, and then estimates the initial pose of the standard AR terminal (S106). Subsequently, the standard AR terminal performs image matching between the acquired keyframes and the photographed image, and then estimates the final pose of the standard AR terminal (S108).

[0156] In the standard positioning process described above, a standard AR terminal does not share information such as keyframes with another standard AR terminal. Therefore, a standard AR terminal does not consider its positional relationship with another standard AR terminal when determining the display position of virtual objects.

[0157] (1-2) Positioning processing according to this embodiment

[0158] like Figure 5As shown, AR terminal 10 first performs feature point detection and feature description on the photographed image, and then acquires two-dimensional feature points (S202). Next, AR terminal 10 uses the acquired two-dimensional feature points to search for keyframes in the keyframe group of the map DB stored in the storage unit 150 of AR terminal 10, and acquires updated destination candidate keyframes (S204). Next, AR terminal 10 sends the updated destination candidate keyframes to server 20, and causes server 20 to perform keyframe update determination processing (S206). Note that the details of the keyframe update determination processing will be described below.

[0159] After the keyframe update determination process, AR terminal 10 confirms whether the position determination keyframe has been determined to be updated during the determination process (S208). When it is determined during the determination process that the position determination keyframe has been updated (S208 / "Yes"), AR terminal 10 updates the position determination keyframe (S210) and proceeds to the process in S212. When it is determined during the determination process that the position determination keyframe should not be updated (S208 / "No"), AR terminal 10 proceeds to the process in S212 without updating the position determination keyframe.

[0160] In S212, the AR terminal 10 performs feature point matching between the feature points of the location determination keyframe and the feature points of the photographed image (S212). After feature point matching, the AR terminal 10 confirms whether the feature point matching is successful (S214).

[0161] When no feature point match is achieved (S214 / “No”), AR terminal 10 repeats the processing that started from S210. When a feature point match is successful (S214 / “Yes”), AR terminal 10 estimates its initial pose (S216). Subsequently, AR terminal 10 performs image matching between the position determination keyframe and the captured image, and estimates its final pose (S218).

[0162] In the positioning process described above according to this embodiment, AR terminal 10 and another AR terminal 10 share keyframes on server 20. Furthermore, AR terminal 10 determines the display position of the virtual object based on the result of the keyframe update determination process performed by server 20, taking into account its positional relationship with another standard AR terminal. With this configuration, the difference in display position can be reduced when multiple AR terminals display the same virtual object.

[0163] (2) Keyframe update determination and processing

[0164] Next, we will refer to Figure 6 The flow of keyframe update determination processing according to an embodiment of this disclosure is described. Figure 6This is a flowchart illustrating the keyframe update determination process according to an embodiment of the present disclosure. Furthermore, the keyframe update determination process according to this embodiment should be performed by the server 20 according to this embodiment.

[0165] like Figure 6 As shown, firstly, server 20 calculates a score indicating the similarity between the updated destination candidate keyframe acquired by AR terminal 10a and the updated destination candidate keyframe acquired by AR terminal 10b (S302). After calculating the score, server 20 confirms whether the score is a first threshold or greater (S304).

[0166] When the score is not the first threshold or greater (S304 / “No”), the server 20 determines the key frame to be updated (S314) and ends the key frame update determination process.

[0167] When the score is at or above the first threshold (S304 / "Yes"), server 20 confirms whether the score is at or above the maximum value (S306). When the score is at or above the maximum value (S306 / "Yes"), server 20 updates the maximum score (S308). After updating the maximum value, server 20 determines the update position to determine the keyframe (S314) and ends the keyframe update determination process.

[0168] When the score is not the maximum value or greater (S306 / “No”), the server 20 confirms whether the movement distance of AR terminal 10a or AR terminal 10b since the last update of the location determination keyframe is the second threshold or greater (S310).

[0169] When the movement distance is the second threshold or greater (S310 / “Yes”), server 20 updates the maximum score to 0 (S312). After updating the maximum score, server 20 determines the update position to determine the keyframe (S314) and ends the keyframe update determination process.

[0170] When the moving distance is not the second threshold or greater (S310 / “No”), the server 20 determines not to update the position determination key frame (S316) and ends the key frame update determination process.

[0171] (3) Processing in Information Processing System 1000

[0172] Subsequently, refer to Figure 7 The process flow in the information processing system 1000 according to an embodiment of the present disclosure is described. Figure 7 This is a sequence diagram illustrating the processing flow in an information processing system 1000 according to an embodiment of the present disclosure.

[0173] like Figure 7As shown, firstly, AR terminal 10a acquires updated destination candidate keyframes by performing processes S202 to S204 (S402). After acquisition, AR terminal 10a sends the acquired updated destination candidate keyframes to server 20 (S404). Similarly, AR terminal 10b acquires updated destination candidate keyframes by performing processes S202 to S204 (S406). After acquisition, AR terminal 10b sends the acquired updated destination candidate keyframes to server 20 (S408).

[0174] After receiving updated destination candidate keyframes from AR terminals 10a and 10b, server 20 performs keyframe update determination processing (S410) by executing S206. After the keyframe update determination processing, server 20 sends the determination result to AR terminals 10a and 10b (S412 and S416).

[0175] After receiving the determination result from the server 20, the AR terminal 10a performs the processing steps S208 to S218 (S414). Similarly, after receiving the determination result from the server 20, the AR terminal 10b performs the processing steps S208 to S218 (S418).

[0176] <1-4 Specific Examples>

[0177] The configuration example according to this implementation method has been described above. Next, we will refer to... Figures 8A to 11 A specific example according to this embodiment is described below. A specific example regarding determining the keyframe for updating the position will be described below.

[0178] (1) First specific example

[0179] First, refer to Figure 8A and Figure 8B This describes an example of updating the position determination keyframe when user 12a's viewing direction is the same as user 12b's viewing direction. Figure 8A This is a diagram illustrating the positional relationships among a plurality of users 12 in a first specific example of an implementation according to the present disclosure. Figure 8B This is a diagram showing views of various users 12 in a first specific example of an implementation according to this disclosure. Note that the time in the first specific example should be time t1.

[0180] like Figure 8A As shown, the virtual object of ghost 50 and the real object of indoor plant 51 are within the field of vision 40a of user 12a wearing AR terminal 10a. At this moment, user 12a is viewing ghost 50 and indoor plant 51, as... Figure 8B As shown on the left.

[0181] Additionally, the virtual object of Ghost 50 and the real object of Indoor Plant 51 are also within the field of vision 40b of user 12b wearing AR terminal 10b. At this moment, user 12b is viewing Ghost 50 and Indoor Plant 51, as... Figure 8B As shown on the right.

[0182] When performing keyframe update determination processing at time tl, AR terminal 10a is based on Figure 8B The photographic image 60a shown on the left is used to determine updated destination candidate keyframes. Furthermore, the AR terminal 10b is based on... Figure 8B The image 60b shown on the right is used to obtain updated destination candidate keyframes. Since image 60a and image 60b are similar, the updated destination candidate keyframes obtained based on each image are also similar. Therefore, the keyframe update determination process in server 20 determines that AR terminal 10 will not update the location determination keyframe. In addition, AR terminal 10a and AR terminal 10b will not update the location determination keyframe and will continue to use the same keyframe.

[0183] (2) Second specific example

[0184] Next, we will refer to Figure 9A and Figure 9B This describes an example of updating the position determination keyframe when the viewing direction of user 12a is different from that of user 12b. Figure 9A This is a diagram illustrating the positional relationships among a plurality of users 12 in a second specific example of an implementation according to the present disclosure. Figure 9B This is a diagram showing views of each user 12 in a second specific example of an implementation according to this disclosure. Note that the time in the second specific example should be time t2.

[0185] Relative to the time t of the first specific example l The user 12a did not change the direction or position of the field of view 40a. Therefore, as Figure 9A As shown, the virtual object of ghost 50 and the real object of indoor plant 51 are within the field of vision 40a of user 12a wearing AR terminal 10a. At this moment, user 12a is viewing ghost 50 and indoor plant 51, as... Figure 9B As shown on the left.

[0186] In contrast, user 12b did not change location, but relative to time t described in the first specific example... l The user 12b changed the direction of their field of view 40b. Therefore, as... Figure 9A As shown, only the real object with flower 52 is within the field of vision 40b of user 12b wearing AR terminal 10b. At this time, user 12b is watching something like... Figure 9BThe flower shown on the right is number 52.

[0187] When the keyframe update decision is executed at time t2, AR terminal 10a is based on Figure 9B The photographic image 60a shown on the left is used to determine updated destination candidate keyframes. Furthermore, the AR terminal 10b is based on... Figure 9B The image 60b shown on the right is used to obtain updated destination candidate keyframes. Since image 60a and image 60b are dissimilar, the updated destination candidate keyframes obtained based on each image are also dissimilar. Therefore, the keyframe update determination process in server 20 determines that AR terminal 10 will update the location determination keyframe. In addition, AR terminal 10a and AR terminal 10b will update the location determination keyframe.

[0188] (3) Third specific example

[0189] Next, we will refer to Figure 10A and Figure 10B This example describes how to update the position determination keyframe when the positional relationship between user 12a and user 12b has changed. Figure 10A This is a diagram illustrating the positional relationships among a plurality of users 12 in a third specific example of an implementation according to the present disclosure. Figure 10B This is a diagram showing views of each user 12 in a third specific example of an implementation according to this disclosure. Note that the time in the third specific example should be time t3.

[0190] From time t2 in the second specific example, user 12a did not change the direction or position of their field of view 40a. Therefore, as Figure 10A As shown, the virtual object of ghost 50 and the real object of indoor plant 51 are within the field of vision 40a of user 12a wearing AR terminal 10a. At this moment, user 12a is viewing... Figure 10B The ghost 50 and indoor plant 51 are shown on the left.

[0191] In contrast, relative to the direction at time t2 described in the second specific example, user 12b changed the direction of his field of vision 40b, a change occurring at time t2 relative to the first specific example. l The direction is the same as the direction at that location. Therefore, as... Figure 10A As shown, the virtual object of ghost 50 and the real object of indoor plant 51 are within the field of vision 40b of user 12b wearing AR terminal 10b. At this moment, user 12b is viewing... Figure 10B The ghost 50 and indoor plant 51 are shown on the right. However, relative to the position at time t2 described in the second specific example, user 12b changed position, such that the movement distance was the second threshold or greater.

[0192] When the keyframe update decision is executed at time t2, AR terminal 10a is based on Figure 10B The photographic image 60a shown on the left is used to determine updated destination candidate keyframes. Furthermore, the AR terminal 10b is based on... Figure 10B The photographic image 60b shown on the right is used to obtain updated destination candidate keyframes. Since photographic image 60a and photographic image 60b are similar, the updated destination candidate keyframes obtained based on each photographic image are also similar. However, since the user 12b's movement distance is a second threshold or greater, it is determined in the keyframe update determination process in server 20 that the location determination keyframe needs to be updated. In addition, AR terminals 10a and 10b will update the location determination keyframes.

[0193] (4) Fourth specific example

[0194] Finally, refer to Figure 11 The time-series changes in the keyframes are described in the first specific example to the third specific example above. Figure 11 This is a graph illustrating the time-series changes of keyframes in a fourth specific example of an implementation according to the present disclosure.

[0195] Figure 11 The diagram shows the temporal sequence changes of the location determination keyframes for AR terminal 10a and AR terminal 10b. Note that AR terminal 10a uses location determination keyframes 1 to 5, while AR terminal 10b uses location determination keyframes 6 to 10.

[0196] First, from time t0 and time t l During the duration, the location determination keyframes of AR terminal 10a change in the order of location determination keyframes 3, 5, 4, and 1. Similarly, the location determination keyframes of AR terminal 10b change in the order of location determination keyframes 8, 10, 9, and 7.

[0197] As described in the first specific example above, at time t l The keyframe update determination process determines that AR terminal 10 will not update the location determination keyframe. Based on this determination, during the duration from time t1 to time t2, location determination keyframe 1 continues to be used in AR terminal 10a, while location determination keyframe 7 continues to be used in AR terminal 10b.

[0198] As described in the second specific example above, the keyframe update determination process at time t2 has determined that AR terminal 10 will update the location determination keyframe. Based on this determination, during the duration from time t2 to time t3, the location determination keyframe of AR terminal 10a changes in the order of location determination keyframes 3, 1, 2, 1, and 4. Similarly, the location determination keyframe of AR terminal 10b changes in the order of location determination keyframes 6, 9, 6, 8, and 10.

[0199] As described in the third specific example above, the keyframe update determination process at time t3 determines that AR terminal 10 will update the location determination keyframe. Based on this determination, at time t3 or later, the location determination keyframe of AR terminal 10a changes in the order of location determination keyframes 4, 1, 3, ... Similarly, the location determination keyframe of AR terminal 10b changes in the order of location determination keyframes 10, 9, 10, ...

[0200] <1-5. Modified Examples>

[0201] In the following sections, modifications to embodiments of the present disclosure will be described. Note that the modifications described below can be applied individually or in combination to embodiments of the present disclosure. Furthermore, modifications can be applied instead of the configurations described in embodiments of the present disclosure, or modifications can be additionally applied to the configurations described in embodiments of the present disclosure.

[0202] (1) First Modified Example

[0203] The above embodiment is an example where the number of AR terminals 10 sharing the same virtual object is two (two users 12). However, there is no specific limitation on the number of AR terminals 10 sharing the same virtual object, and it can be any number. In the following, reference will be made to... Figure 12 An example describing three AR terminals 10 that share the same virtual object. Figure 12 This is a flowchart illustrating the keyframe update determination process in a first modified example of an embodiment of the present disclosure.

[0204] Server 20 receives updated destination candidate keyframes from all AR terminals 10 sharing the same virtual object and performs keyframe update decision processing, considering all combinations of the received updated destination candidate keyframes. For example, server 20 calculates a similarity score for all combinations of the received updated destination candidate keyframes and compares the calculated score with a maximum value. Furthermore, server 20 compares the movement distance of all AR terminals 10 with a second threshold.

[0205] With this configuration, the server 20 can perform keyframe update determination processing even when there are three or more AR terminals 10 sharing the same virtual object. Furthermore, when three or more display terminals display the same virtual object based on the result of the keyframe update determination processing, the difference in display position can be reduced for each of the three or more AR terminals 10.

[0206] Here, we will refer to Figure 12 Describe the keyframe update determination process in the first modified example. For example... Figure 12 As shown, server 20 first calculates all combinations of selecting two AR terminals 10 from N AR terminals 10. N The similarity score of the updated destination candidate keyframes in the C2 variant (S502). In this modified example, since the number of AR terminals 10 is three, there are 3C2 = 3 combinations.

[0207] In calculation N After C2 scored, server 20 confirmed. N Is the C2 score equal to or greater than the first threshold (S504)? In this modified example, it is confirmed whether the 3C2 score is equal to or greater than the first threshold.

[0208] When the score is not the first threshold or greater (S504 / “No”), the server 20 determines the key frame to be updated (S514) and ends the key frame update determination process.

[0209] When the score is at or above the first threshold (S504 / “Yes”), server 20 confirms. N If at least one of the C2 scores is the maximum value or greater (S506), the server 20 updates the maximum value of the score (S508). After updating the maximum value, the server 20 determines the update position to determine the keyframe (S514) and ends the keyframe update determination process.

[0210] When none of the scores are at or above the maximum value (S506 / “No”), the server 20 confirms whether the movement distance of at least one of the N AR terminals 10 since the last update of the location determination keyframe is at or above the second threshold (S510).

[0211] When the movement distance of at least one AR terminal 10 is the second threshold or greater (S510 / “Yes”), the server 20 updates the maximum score to 0 (S512). After updating the maximum score, the server 20 determines the update position keyframe (S514) and ends the keyframe update determination process.

[0212] When the movement distance of at least one AR terminal 10 is not a predetermined threshold or greater (S510 / “No”), the server 20 determines not to update the location determination key frame (S516) and ends the key frame update determination process.

[0213] (2) Second Amendment

[0214] Although the above implementation is an example of multiple AR terminals 10 sharing the same virtual object at the same point in time, the points in time when multiple AR terminals 10 share the same virtual object can be different from each other. In the following text, reference will be made to... Figure 13 This describes an example of multiple AR terminals 10 sharing the same virtual object at different times. Figure 13 This is a flowchart illustrating the keyframe update determination process in a second modified embodiment of the present disclosure.

[0215] Suppose that user 12a of AR terminal 10a views a virtual object at a predetermined location at a specific time t4. Suppose that user 12a views the same virtual object at the same predetermined location at a different time t5. In this case, the virtual object should be displayed on user 12b's AR terminal 10b in the same manner as when user 12a viewed the virtual object.

[0216] Specific examples of this scenario include AR puzzle games and AR escape games. For instance, suppose user 12a places a virtual object somewhere in a room and leaves. Then, suppose user 12b enters the same room and needs to determine the location of the virtual object placed by user 12a. If user 12b does not accurately determine the location of the virtual object placed by user 12a, the game will be affected. For example, suppose there are red and blue chairs in the room, and user 12a places the virtual object on the red chair. However, suppose that when user 12b views the virtual object, its location shifts, and the virtual object appears on the blue chair. In this case, user 12b cannot correctly determine the location of the virtual object placed by user 12a and cannot complete the game.

[0217] To address this issue, even across different time series, server 20 needs to control the display of virtual objects based on updated destination candidate keyframes from AR terminal 10a and AR terminal 10b. However, unfortunately, because user 12a's AR terminal 10a is not at the same location as user 12b's AR terminal 10b at time t5, server 20 cannot obtain the updated destination candidate keyframe for AR terminal 10a at that location.

[0218] To address this issue, the control unit 220 of server 20 calculates the similarity between the operating state of AR terminal 10a, which has a specific positional relationship with a predetermined location, and the operating state of AR terminal 10b, which has a specific positional relationship with a predetermined location in a time series different from the operating state of AR terminal 10a, and then performs keyframe update determination processing.

[0219] A specific positional relationship with a predetermined location is, for example, the positional relationship between the predetermined location and the AR terminal 10 when the user 12 views a virtual object displayed at the predetermined location. Therefore, in other words, the operating state of the AR terminal 10, which has a specific positional relationship with the predetermined location, is the operating state of the AR terminal 10 when the user 12 views a virtual object displayed at the predetermined location.

[0220] For example, server 20 acquires a location determination keyframe based on the operational state of AR terminal 10a of user 12a who first views a virtual object displayed at a predetermined location, as a reference keyframe (hereinafter also referred to as "keyframe X"). When user 12b views the virtual object displayed at the predetermined location at a different time than user 12a, server 20 acquires an updated destination candidate keyframe based on the operational state of AR terminal 10b of user 12b. Subsequently, server 20 calculates the similarity between keyframe X and the updated destination candidate keyframe of AR terminal 10b, and then performs keyframe update determination processing.

[0221] With this configuration, even when multiple users view the same virtual object at different times, the difference in display position when multiple display terminals display the same virtual object can be reduced.

[0222] Here, we will refer to Figure 13 The flow of the keyframe update determination process in the second modified example is described. Note that the keyframe for determining the position of AR terminal 10a of user 12a who first views the predetermined position is defined as keyframe X. Furthermore, it is assumed that the keyframe update determination process is performed when user 12b views the predetermined position at a time later than user 12a's viewing time.

[0223] like Figure 13 As shown, server 20 first calculates a score indicating the similarity between the location determination keyframe of AR terminal 10b and keyframe X (S602). After calculating the score, server 20 confirms whether the score is a first threshold or greater (S604).

[0224] When the score is not the first threshold or greater (S604 / “No”), the server 20 determines the key frame to be updated (S614) and ends the key frame update determination process.

[0225] When the score is at or above the first threshold (S604 / "Yes"), server 20 confirms whether the score is at or above the maximum value (S606). When the score is at or above the maximum value (S606 / "Yes"), server 20 updates the maximum score (S608). After updating the maximum value, server 20 determines the update position to determine the keyframe (S614) and ends the keyframe update determination process.

[0226] When the score is not the maximum value or greater (S606 / “No”), the server 20 confirms whether the movement distance of AR terminal 10a or AR terminal 10b since the last update of the location determination keyframe is the second threshold or greater (S610).

[0227] When the movement distance is the second threshold or greater (S610 / “Yes”), the server 20 updates the maximum score to 0 (S612). After updating the maximum score, the server 20 determines the update position to determine the keyframe (S614) and ends the keyframe update determination process.

[0228] When the moving distance is not the second threshold or greater (S610 / “No”), the server 20 determines not to update the position determination key frame (S616) and ends the key frame update determination process.

[0229] <<2. Second Implementation Method>>

[0230] The first embodiment has been described above. A second embodiment of the present disclosure will now be described. Although the above embodiment is an example of keyframe update determination processing performed by server 20, keyframe update processing can be performed by AR terminal 10.

[0231] In addition to the functions of the AR terminal 10 described in the first embodiment, the AR terminal 10 according to the second embodiment also includes the same functions as the control unit 220 of the server 20 described in the first embodiment. With this configuration, the AR terminal 10 according to the second embodiment can perform keyframe update determination processing independently of the server 20.

[0232] Note that when there are multiple AR terminals 10 capable of performing keyframe update determination processing, the AR terminal performing the keyframe update determination processing can be determined based on the processing capabilities of each AR terminal 10. For example, the AR terminal with the highest processing capability can be determined as the terminal performing the keyframe update determination processing.

[0233] <<3. Hardware Configuration Example>>

[0234] Finally, refer to Figure 14 An example of the hardware configuration of the information processing apparatus according to this embodiment is described. Figure 14This is a block diagram illustrating an example of the hardware configuration of the information processing apparatus according to this embodiment. Note that... Figure 14 The information processing device 900 shown in the figure can, for example, implement Figure 3 The server 20 shown in the figure. The processing of the server 20 according to this embodiment is achieved through the cooperative operation of software and hardware as described below.

[0235] like Figure 14 As shown, the information processing device 900 includes a central processing unit (CPU) 901, a read-only memory (ROM) 902, and a random access memory (RAM) 903. Furthermore, the information processing device 900 also includes a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, a driver 911, a connection port 912, and a communication device 913. Note that the hardware configuration shown here is an example, and some components may be omitted. Additionally, the hardware configuration may include components other than those shown here.

[0236] CPU 901 functions as, for example, an arithmetic processing device or a control device, and controls all or part of the operation of each component based on various programs recorded in ROM 902, RAM 903, or storage device 910. ROM 902 is a device that stores programs loaded by CPU 901, data for calculation, etc. RAM 903 temporarily or permanently stores, for example, programs loaded by CPU 901, various parameters that change appropriately during program execution, etc. These are interconnected via a host bus 904, including a CPU bus, etc. CPU 901, ROM 902, and RAM 903 can, for example, cooperate with software to implement references. Figure 3 The functions of the control unit 220 are described.

[0237] CPU 901, ROM 902, and RAM 903 are interconnected, for example, via a host bus 904 capable of high-speed data transfer. On the other hand, the host bus 904 is connected, for example, via a bridge 905 to an external bus 906 with a relatively low data transfer speed. Furthermore, the external bus 906 is connected to various components via an interface 907.

[0238] The input device 908 is implemented by a device through which a user inputs information, such as a mouse, keyboard, touch panel, button, microphone, switch, and joystick. Furthermore, the input device 908 may be, for example, a remote control device using infrared or other radio waves, or an external connection device such as a mobile phone or PDA that supports the operation of the information processing device 900. Additionally, the input device 908 may include, for example, an input control circuit that generates input signals based on information input by the user using the aforementioned input device and outputs these input signals to the CPU 901. By operating the input device 908, the user of the information processing device 900 can input various data into the information processing device 900 or give instructions regarding processing operations.

[0239] Additionally, the input device 908 can be formed by a device for detecting information about the user. For example, the input device 908 may include various sensors, such as image sensors (e.g., cameras), depth sensors (e.g., stereo cameras), accelerometers, gyroscopes, geomagnetic sensors, optical sensors, sound sensors, distance measurement sensors (e.g., time-of-flight (ToF) sensors), and force sensors. Furthermore, the sensor 908 can acquire information about the information processing device 900's own state, such as the attitude and speed of movement of the information processing device 900, and information about the surrounding environment of the information processing device 900, such as the brightness and noise around the information processing device 900. Furthermore, the input device 908 may include a Global Navigation Satellite System (GNSS) module that receives GNSS signals from GNSS satellites (e.g., GPS signals from Global Positioning System (GPS) satellites) and measures location information including the latitude, longitude, and altitude of the device. Moreover, regarding location information, the input device 908 can detect its location via Wi-Fi (registered trademark), transmission and reception using mobile phones, PHS, smartphones, etc., near-field communication, etc.

[0240] Output device 909 is formed by means of a device capable of visually or audibly notifying a user of acquired information. Examples of such devices include display devices (e.g., CRT display devices, liquid crystal display devices, plasma display devices, EL display devices, laser projectors, LED projectors, and lamps), voice output devices (e.g., speakers and headphones), and printer devices. Output device 909 outputs, for example, the results obtained through various processes performed by information processing device 900. Specifically, the display device displays the results obtained through various processes performed by information processing device 900 in various formats such as text, images, tables, and graphs. The audio output device converts audio signals, including reproduced audio data, acoustic data, etc., into analog signals and outputs the signals audibly.

[0241] Storage device 910 is an example data storage device formed as a storage unit in information processing device 900. Storage device 910 may be implemented, for example, as a magnetic storage device (e.g., HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, etc. Storage device 910 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded on the storage medium. This storage device 910 stores programs executed by CPU 901, various data, and various data acquired from external sources. Storage device 910 may, for example, implement a reference... Figure 3 The function of the storage unit 230 is described.

[0242] Drive 911 is a reader / writer for storage media and is either built into or externally connected to the information processing device 900. Drive 911 reads information recorded on removable storage media (e.g., an installed disk, optical disk, magneto-optical disk, or semiconductor memory) and outputs the read information to RAM 903. Drive 911 can also write information to removable storage media.

[0243] Connection port 912 is, for example, a port for connecting external connection devices such as a Universal Serial Bus (USB) port, an IEEE 1394 port, a Small Computer System Interface (SCSI) port, an RS-232C port, or an optical audio terminal.

[0244] The communication device 913 is, for example, a communication interface formed by a communication device for connecting to the network 920. The communication device 913 is, for example, a communication card for wired or wireless local area networks (LANs), Long Term Evolution (LTE), Bluetooth (registered trademark), Wireless USB (WUSB), etc. Furthermore, the communication device 913 can be a router for optical communication, an Asymmetric Digital Subscriber Line (ADSL) router, a modem for various communications, etc. The communication device 913 can send signals to the Internet and other communication devices and receive signals from the Internet and other communication devices according to predetermined protocols such as TCP / IP. The input device 913 can, for example, implement a reference... Figure 3 The functions of the communication unit 210 are described.

[0245] Network 920 is a wired or wireless transmission path for sending information from devices connected to network 920. For example, network 920 may include public networks such as the Internet, telephone networks, and satellite communication networks; various local area networks (LANs) including Ethernet (registered trademark); wide area networks (WANs); etc. Furthermore, network 920 may include private networks, such as Internet Protocol Virtual Private Networks (IP-VPNs).

[0246] Examples of hardware configurations capable of implementing the functions of the information processing apparatus 900 according to this embodiment have been described above. Each of the above-described components can be implemented using general-purpose components, or can be implemented using hardware devices dedicated to the functions of each component. Therefore, the hardware configuration to be used can be appropriately changed depending on the level of technology at the time of performing this embodiment.

[0247] <<4. Supplement>>

[0248] Preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings. However, the technical scope of the present disclosure is not limited to such examples. It will be apparent to those skilled in the art that various modifications and alterations can be conceived within the scope of the technical concept described in the claims, and such modifications and alterations naturally fall within the technical scope of the present disclosure.

[0249] Furthermore, the series of processes to be performed by the various devices described herein can be achieved using any of software, hardware, or a combination of software and hardware. The programs constituting the software are pre-stored in, for example, a recording medium (non-transitory medium) located internally or externally in each of the devices. Each of the programs is then read into RAM and executed by a processor such as a CPU when executed by a computer.

[0250] Furthermore, the processes described using the flowcharts and sequence diagrams in this specification do not necessarily have to be performed in the order shown. Some processing steps can be performed in parallel. Additionally, extra processing steps can be used, and some processing steps can be omitted.

[0251] Furthermore, the effects described in this specification are merely illustrative or exemplary and are not intended to be limiting. That is, in addition to or in lieu of the effects described above, the technology based on this disclosure may exhibit other effects that would be apparent to those skilled in the art based on the description herein.

[0252] Note that the following configurations also fall within the technical scope of this disclosure. (1)

[0254] An information processing device, comprising

[0255] The control unit controls the display terminal that arranges virtual objects in the real space using a coordinate system associated with the real space.

[0256] The control unit determines whether to update the first spatial information of the first display terminal and the second spatial information of the second display terminal used to determine the location of the virtual object, based on the first operating state of the first display terminal and the second operating state of the second display terminal that shares the virtual object with the first display terminal. (2)

[0258] According to the information processing apparatus described in (1), the control unit determines whether to update the first spatial information and the second spatial information based on whether the first operating state and the second operating state are similar. (3)

[0260] According to the information processing device described in (2),

[0261] Specifically, when the first operation state is similar to the second operation state, the control unit determines not to update the first spatial information and the second spatial information; and when the first operation state is not similar to the second operation state, the control unit determines to update the first spatial information and the second spatial information. (4)

[0263] According to the information processing apparatus described in (2) or (3), the control unit calculates the similarity between the first operation state and the second operation state, and determines whether the first operation state and the second operation state are similar based on the result of comparing the similarity with a first threshold. (5)

[0265] According to the information processing apparatus described in (4), the control unit calculates the similarity based on third spatial information as an updated destination candidate obtained by the first display terminal in the first operating state and fourth spatial information as an updated destination candidate obtained by the second display terminal in the second operating state. (6)

[0267] According to the information processing apparatus described in (5), the control unit performs matching processing between the feature points included in the third spatial information and the feature points included in the fourth spatial information, and calculates the similarity based on the number of successfully matched feature points. (7)

[0269] According to the information processing device described in (4), the control unit changes the first threshold according to the positional relationship between the first display terminal and the second display terminal. (8)

[0271] The information processing apparatus according to any one of (4) to (7),

[0272] The control unit further determines whether to update the first spatial information and the second spatial information based on whether the similarity exceeds a third threshold.

[0273] The third threshold is the maximum similarity calculated within a time period that meets predetermined conditions. (9)

[0275] According to the information processing device described in (8),

[0276] Specifically, when the similarity exceeds the third threshold, the control unit determines to update the first spatial information and the second spatial information, and

[0277] When the similarity does not exceed the third threshold, the control unit determines not to update the first spatial information and the second spatial information. (10)

[0279] The information processing apparatus according to any one of (2) to (9), wherein the control unit further determines whether to update the first spatial information and the second spatial information based on the moving distance of the first display terminal or the moving distance of the second display terminal after the last update of the first spatial information and the second spatial information. (11)

[0281] According to the information processing device described in (10),

[0282] Specifically, when the movement distance is a second threshold or greater, the control unit determines to update the first spatial information and the second spatial information, and

[0283] When the movement distance is not the second threshold or greater, the control unit determines not to update the first spatial information and the second spatial information. (12)

[0285] According to the information processing apparatus of (4), the control unit calculates the similarity between the first operating state having a specific positional relationship with a predetermined position and the second operating state having a specific relationship with the predetermined position in a time series different from the first operating state. (13)

[0287] An information processing device, comprising:

[0288] The output unit includes a first output device for presenting a virtual object to a first user; and

[0289] The control unit determines the position of the virtual object to be placed in a coordinate system associated with real space.

[0290] The control unit determines whether to update the first spatial information of the first output device and the second spatial information of the second output device for determining the location of the virtual object based on the first operating state of the first output device and the second operating state of the second output device. The second output device is located externally and shares the virtual object with the first output device. (14)

[0292] An information processing method to be executed by a processor, the method comprising:

[0293] Controlling a display terminal that arranges virtual objects in the real space using a coordinate system associated with the real space; and

[0294] Based on the first operating state of the first display terminal and the second operating state of the second display terminal that shares the virtual object with the first display terminal, it is determined whether to update the first spatial information of the first display terminal and the second spatial information of the second display terminal used to determine the location of the virtual object. (15)

[0296] A recording medium having a program recorded thereon that causes a computer to perform processing, said processing including:

[0297] Controlling a display terminal that arranges virtual objects in the real space using a coordinate system associated with the real space; and

[0298] Based on the first operating state of the first display terminal and the second operating state of the second display terminal that shares the virtual object with the first display terminal, it is determined whether to update the first spatial information of the first display terminal and the second spatial information of the second display terminal used to determine the location of the virtual object.

[0299] List of reference numerals

[0300] 10AR terminal

[0301] 12 users

[0302] 20 servers

[0303] 30 Network

[0304] 120 communication units

[0305] 130 sensor units

[0306] 140 control unit

[0307] 150 storage units

[0308] 160 output units

[0309] 210 Communication Unit

[0310] 220 control unit

[0311] 230 storage units

[0312] 1000 Information Processing System

[0313] 1402 Surrounding Conditions Information Acquisition Unit

[0314] 1404 Surrounding Area Information Management Unit

[0315] 1406 Spatial Information Acquisition Unit

[0316] 1408 Spatial Information Management Unit

[0317] 1410 Image Output Control Unit

[0318] 1412 Output Image Generation Unit

Claims

1. An information processing device, comprising: The control unit controls the display terminal that arranges virtual objects in the real space using a coordinate system associated with the real space. in, The control unit determines whether to update the first spatial information of the first display terminal and the second spatial information of the second display terminal used to determine the location of the virtual object, based on the first operating state of the first display terminal and the second operating state of the second display terminal that shares the virtual object with the first display terminal. This is done when the first display terminal presents the virtual object to a first user, and the second display terminal presents the virtual object to a second user. The control unit determines whether to update the first spatial information and the second spatial information based on whether the first operating state and the second operating state are similar. The similarity between the first operating state and the second operating state indicates the similarity between the viewing direction of the first user and the viewing direction of the second user.

2. The information processing device according to claim 1, in, When the first operating state is similar to the second operating state, the control unit determines not to update the first spatial information and the second spatial information, and When the first operating state is not similar to the second operating state, the control unit determines to update the first spatial information and the second spatial information.

3. The information processing apparatus according to claim 1, wherein, The control unit calculates the similarity between the first operation state and the second operation state, and determines whether the first operation state and the second operation state are similar based on the result of comparing the similarity with a first threshold.

4. The information processing apparatus according to claim 3, wherein, The control unit calculates the similarity based on third spatial information, which is the updated destination candidate obtained by the first display terminal in the first operating state, and fourth spatial information, which is the updated destination candidate obtained by the second display terminal in the second operating state.

5. The information processing apparatus according to claim 4, wherein, The control unit performs matching processing between the feature points included in the third spatial information and the feature points included in the fourth spatial information, and calculates the similarity based on the number of successfully matched feature points.

6. The information processing apparatus according to claim 3, wherein, The control unit changes the first threshold according to the positional relationship between the first display terminal and the second display terminal.

7. The information processing apparatus according to claim 3, in, The control unit further determines whether to update the first spatial information and the second spatial information based on whether the similarity exceeds a third threshold. The third threshold is the maximum similarity calculated within a time period that meets predetermined conditions.

8. The information processing apparatus according to claim 7, in, When the similarity exceeds the third threshold, the control unit determines to update the first spatial information and the second spatial information, and When the similarity does not exceed the third threshold, the control unit determines not to update the first spatial information and the second spatial information.

9. The information processing apparatus according to claim 1, wherein, The control unit further determines whether to update the first spatial information and the second spatial information based on the movement distance of the first display terminal or the movement distance of the second display terminal since the last update of the first spatial information and the second spatial information.

10. The information processing apparatus according to claim 9, in, When the travel distance is a second threshold or greater, the control unit determines to update the first spatial information and the second spatial information, and When the movement distance is not the second threshold or greater, the control unit determines not to update the first spatial information and the second spatial information.

11. The information processing apparatus according to claim 3, wherein, The control unit calculates the similarity between a first operating state that has a positional relationship with a predetermined location and a second operating state that has a relationship with the predetermined location in a time series different from the first operating state.

12. An information processing apparatus, comprising: An output unit includes a first output device for presenting a virtual object to a first user; as well as The control unit determines the position of the virtual object to be placed in a coordinate system associated with real space. The control unit determines whether to update the first spatial information of the first output device and the second spatial information of the second output device used to determine the location of the virtual object based on the first operating state of the first output device and the second operating state of the second output device. The second output device is externally located and shares the virtual object with the first output device. The second output device presents the virtual object to the second user. The control unit determines whether to update the first spatial information and the second spatial information based on whether the first operating state and the second operating state are similar. The similarity between the first operating state and the second operating state indicates the similarity between the viewing direction of the first user and the viewing direction of the second user.

13. An information processing method to be executed by a processor, the method comprising: Controls the display terminal that arranges virtual objects in the real space using a coordinate system associated with the real space; as well as Based on the first operating state of the first display terminal and the second operating state of the second display terminal that shares the virtual object with the first display terminal, it is determined whether to update the first spatial information of the first display terminal and the second spatial information of the second display terminal used to determine the location of the virtual object, wherein the first display terminal presents the virtual object to a first user, and the second display terminal presents the virtual object to a second user. Specifically, the decision to update the first spatial information and the second spatial information is based on whether the first operation state and the second operation state are similar. The similarity between the first operation state and the second operation state indicates the similarity between the viewing direction of the first user and the viewing direction of the second user.

14. A recording medium having a program recorded thereon that causes a computer to perform processing, said processing comprising: Controls the display terminal that arranges virtual objects in the real space using a coordinate system associated with the real space; as well as Based on the first operating state of the first display terminal and the second operating state of the second display terminal that shares the virtual object with the first display terminal, it is determined whether to update the first spatial information of the first display terminal and the second spatial information of the second display terminal used to determine the location of the virtual object, wherein the first display terminal presents the virtual object to a first user, and the second display terminal presents the virtual object to a second user. Specifically, the decision to update the first spatial information and the second spatial information is based on whether the first operation state and the second operation state are similar. The similarity between the first operation state and the second operation state indicates the similarity between the viewing direction of the first user and the viewing direction of the second user.

Images