Position estimation system, position estimation method, and position estimation program

Abstract

This position estimation system includes a shared device, a server that estimates the position / orientation of the shared device in a global coordinate system, and a non-shared device for which a relative position / orientation with respect to the shared device is set. The non-shared device includes: a reference unit that acquires, from a server, a position / orientation estimation value pertaining to the shared device; a display unit that displays an object in the position / orientation of the shared device; and a correction unit that calculates the position / orientation of the non-shared device in the global coordinate system on the basis of the position / orientation estimation value and the relative position / orientation.

Description

Position estimation system, position estimation method, and position estimation program

[0001] The present invention relates to a position estimation system, a position estimation method, and a position estimation program.

[0002] Spatial AR (Augmented Reality) is a type of AR technology that overlays virtual information onto a physical space. Spatial AR projects images and information onto a real space to provide users with an interactive experience. Spatial AR can be categorized into non-shared spatial AR, which provides each user with unique AR content, and shared spatial AR, which provides common AR content to multiple users.

[0003] A non-shared AR device for experiencing non-shared space AR cannot support shared space AR. To support shared space AR, the non-shared AR device needs to be equipped with a camera or LiDAR. In this case, the hardware cost increases and the device becomes heavy.

[0004] Therefore, it is possible to install multiple cameras in the location where the shared space AR is desired and utilize pose estimation technology, which makes it possible to estimate the three-dimensional position and orientation of objects and people captured by the cameras.

[0005] Masahiko Tsuji, Takahiro Matsumoto, Taiga Yoshida, and Yutaka Chiaki, "Proposal of a self-location estimation method using images with a camera and 3D-LiDAR," 26th Image Recognition and Understanding Symposium

[0006] To utilize posture estimation technology, it is necessary to physically install multiple cameras whose positions in the global coordinate system are known, which incurs installation and management costs for fixed installation.

[0007] The present invention has been made in light of the above-mentioned circumstances, and its purpose is to provide a position estimation system, a position estimation method, and a position estimation program that enable the provision of shared spatial AR to non-shared AR devices without the need to install multiple fixed cameras.

[0008] One aspect of the present invention is a position estimation system. The position estimation system includes a shared device, a server that estimates the position and orientation of the shared device in a global coordinate system, and a non-shared device whose relative position and orientation with respect to the shared device is set. The non-shared device includes a reference unit that acquires position and orientation estimates of the shared device from the server, a display unit that displays an object at the position and orientation of the shared device, and a correction unit that calculates the position and orientation of the non-shared device in the global coordinate system based on the position and orientation estimates and the relative position and orientation.

[0009] One aspect of the present invention is a position estimation method, which includes the steps of acquiring position and orientation estimates of a shared device in a global coordinate system, displaying an object at the position and orientation of the shared device, and calculating, in the global coordinate system, the position and orientation of a non-shared device whose relative position and orientation with respect to the shared device has been set, based on the position and orientation estimates and the relative position and orientation.

[0010] One aspect of the present invention is a location estimation program that causes a computer to execute at least some of the functions of the components of the location estimation system described above.

[0011] According to the present invention, a position estimation system, a position estimation method, and a position estimation program are provided that enable a shared space AR to be provided to a non-shared AR device without the need to install multiple fixed cameras.

[0012] FIG. 1 is a block diagram showing the functional configuration of a position estimation system according to an embodiment. FIG. 2 is a flowchart showing the first half of an operation example flow of a first example of a position estimation system according to an embodiment. FIG. 3 is a diagram showing a time series flow of an operation example of the first example of a position estimation system according to an embodiment. FIG. 4 is a diagram showing the relative positions and orientations of a shared AR device and a non-shared AR device of a position estimation system according to an embodiment. FIG. 5 is a diagram showing a state at the time when suspension of object display updates is released. FIG. 6 is a flowchart showing the first half of an operation example flow of a second example of a position estimation system according to an embodiment. FIG. 7 is a diagram showing a time series flow of an operation example of the second example of a position estimation system according to an embodiment. FIG. 8 is a block diagram showing an example of the hardware configuration of computers constituting a self-location estimation server, a shared AR device, and a non-shared AR device of a position estimation system according to an embodiment.

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, the background technology of the embodiments will be described.

[0014] (Spatial AR) Devices such as smartphones and AR glasses (hereinafter referred to as AR devices) position AR objects in three-dimensional space, providing users with an experience (hereinafter referred to as spatial AR) in which they can view the AR objects from various angles while moving through the space.

[0015] In spatial AR, the AR device uses an IMU (Inertial Measurement Unit) to measure the relative movement of the device in its own coordinate system (hereinafter referred to as the local coordinate system), thereby calculating the position / posture of the AR object and the device itself, which are positioned in the local coordinate system, and allowing the AR object to be viewed from various angles.

[0016] (Self-Localization Technology and Shared Space AR) Furthermore, there is "shared space AR" in which such space AR is shared by multiple people. In shared space AR, a space is shared by multiple AR devices, and an AR object is localized in the same location in the space, allowing multiple people to view the AR object from various angles.

[0017] In this type of shared space AR, sharing of space with other AR devices is achieved by combining self-position estimation technology that estimates the position / posture of the device in a coordinate system (global coordinate system) common to other AR devices.

[0018] To realize shared space AR, a self-location estimation technology is required to estimate one's own position and orientation in a global coordinate system. There are various self-location estimation technologies, such as VPS, LiDAR-SLAM, and UWB-based positioning technology.

[0019] In a shared space AR where a user visually recognizes an AR object, it is easy for the user to sense the magnitude of the estimation error, so it is desirable that the estimation error be small.

[0020] VPS and LiDAR-SLAM can estimate position and orientation with high accuracy as long as the environment remains unchanged, making them well suited to shared space AR. UWB requires the installation of a UWB transmitter in the space, but can estimate position and orientation with high accuracy, making them well suited to shared space AR.

[0021] To use this type of shared space AR, the AR device needs to be equipped with a camera, LiDAR, or UWB wireless chip, but equipping the AR device with either a camera, LiDAR, or UWB wireless chip increases the hardware cost and makes the device heavy.

[0022] (Position Estimation System) Hereinafter, a position estimation system according to an embodiment will be described. In the position estimation system according to the embodiment, an AR device that is not compatible with shared space AR and does not include hardware for self-position estimation, such as a camera, LIDAR, or UWB wireless chip (hereinafter referred to as a non-shared AR device) is linked with an AR device that is compatible with shared space AR (hereinafter referred to as a shared AR device) based on the above background, thereby estimating the position and orientation of the non-shared AR device in a global coordinate system, thereby enabling the non-shared AR device to be compatible with shared space AR.

[0023] (Functional Configuration) First, the functional configuration of a position estimation system 10 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the functional configuration of a position estimation system 10 according to an embodiment.

[0024] The position estimation system 10 includes a self-position estimation server 20 , a shared AR device 30 , and a non-shared AR device 40 .

[0025] The self-location estimation server 20 has already generated a feature space of the space and can perform position estimation using images. In response to a position estimation request including an image from the shared AR device 30, the self-location estimation server 20 estimates the position and orientation of the shared AR device 30 in a global coordinate system. The system of the self-location estimation server 20 may be VPS or LiDAR-SLAM.

[0026] The shared AR device 30 has a self-location estimation client function that is compatible with the self-location estimation method of the self-location estimation server 20, and can make a request for self-location estimation to the self-location estimation server 20 and obtain the estimation result from the self-location estimation server 20.

[0027] The shared AR device 30 and the non-shared AR device 40 are set in advance to have a predetermined relative positional relationship. For example, the shared AR device 30 is configured as a smartphone, and the non-shared AR device 40 is configured as AR glasses.

[0028] For example, the non-shared AR device 40 holds information about its relative positional relationship. The relative positional relationship is, for example, a relative position and orientation, and includes a relative position and a relative orientation. For example, in a three-dimensional Cartesian coordinate system, the relative position is a difference in position along three axes, and the relative orientation is a difference in angle around the three axes.

[0029] The non-shared AR device 40 can access the self-location estimation server 20 and refer to the position and orientation of the shared AR device 30 in the global coordinate system.

[0030] The self-location estimation server 20 includes a self-location estimation unit 21 , a location information DB (Database) 22 , and a DB reference receiving unit 23 .

[0031] The self-position estimation unit 21 receives a request for self-position estimation from the shared AR device 30, refers to the position information DB 22, estimates the self-position of the shared AR device 30, and records the estimation result together with the estimation time in the position information DB 22. The self-position estimation unit 21 also returns the estimation result to the shared AR device 30 as a request response.

[0032] The DB reference receiving unit 23 receives a request to reference the position information DB 22 from the unshared AR device 40 and permits the unshared AR device 40 to reference the position information DB 22 .

[0033] The shared AR device 30 has a self-location estimation client unit 31. The self-location estimation client unit 31 requests the self-location estimation server 20 for self-location estimation. The self-location estimation client unit 31 also acquires, from the self-location estimation server 20, estimated values ​​of the position and orientation of the shared AR device 30 in the global coordinate system, which are the estimation results.

[0034] The non-shared AR device 40 includes an AR object display unit 41 , a relative movement amount measurement unit 42 , an error correction unit 43 , and a position information DB reference unit 44 .

[0035] The position information DB reference unit 44 can refer to the position information DB 22 via the DB reference receiving unit 23 and acquire estimated values ​​of the position and orientation of the shared AR device 30 .

[0036] In detail, the location information DB reference unit 44 holds an ID that identifies the shared AR device 30, and by using the ID, it is possible to identify the shared AR device 30 and obtain estimated position and orientation values ​​of the shared AR device 30 from the location information DB 22.

[0037] Furthermore, by specifying a time, the position information DB reference unit 44 can acquire, from the position information DB 22, estimated values ​​of the position and orientation of the shared AR device 30 at the specified time.

[0038] The AR object display unit 41 displays an object at the position and orientation of the shared AR device 30. The AR object display unit 41 has a display and a control unit thereof. For example, when the non-shared AR device 40 is configured with AR glasses, the AR object display unit 41 has a transparent display placed in front of the user's eyes.

[0039] The relative movement amount measurement unit 42 measures the relative movement amount in the local coordinate system of the non-shared AR device 40. For example, the relative movement amount measurement unit 42 is configured by an IMU.

[0040] The error correction unit 43 stores the relative positional relationship (relative position and orientation) with the shared AR device 30 in advance, and can calculate the position and orientation of the non-shared AR device 40 in the global coordinate system based on the position and orientation estimates of the shared AR device 30 in the global coordinate system obtained from the position information DB 22 by the position information DB reference unit 44 and the relative position and orientation with respect to the shared AR device 30.

[0041] The error correction unit 43 also stops display updates of the AR object display unit 41, and releases the update stop of the AR object display unit 41 while the AR object is superimposed on the shared AR device 30, thereby correcting the position and orientation of the non-shared AR device 40 in the global coordinate system to the correct position and orientation based on the position and orientation estimated values ​​at the time the update stop was released.

[0042] Furthermore, the error correction unit 43 updates the position and orientation of the non-shared AR device 40 in the global coordinate system using the relative movement amount of the non-shared AR device 40 in the local coordinate system measured by the relative movement amount measurement unit 42 .

[0043] (Operation Example of First Embodiment) Next, an operation example of the first embodiment in the position estimation system 10 will be described with reference to Fig. 2 and Fig. 3. Fig. 2 is a flowchart showing the flow of the operation example of the first embodiment in the position estimation system 10. Fig. 3 is a diagram showing the time series flow of the operation example shown in Fig. 2.

[0044] An example of operation of the first embodiment is an example of manually correcting an AR object by superimposing it on a shared AR device.

[0045] In step S11, the shared AR device 30 requests the self-location estimation server 20 to estimate its own location. Upon receiving the request for self-location estimation, the self-location estimation unit 21 estimates the self-location of the shared AR device 30. The estimation results are estimated values ​​of the position and orientation of the shared AR device 30 in the global coordinate system. This makes it possible to determine the position and orientation of the shared AR device 30 in the global coordinate system. The self-location estimation unit 21 returns the estimation result to the shared AR device 30 as a request response. The self-location estimation unit 21 also records the estimation result together with the estimation time in the position information DB 22. Note that the request for self-location estimation is continuously made while the shared AR device 30 is operating.

[0046] In step S12, the non-shared AR device 40 acquires position and orientation estimates of the shared AR device 30 in global coordinates from the position information DB 22 of the self-position estimation server 20. Specifically, the position information DB reference unit 44 requests the DB reference reception unit 23 to access the position information DB 22. Upon receiving the access, the DB reference reception unit 23 grants the position information DB reference unit 44 access to the position information DB 22. The position information DB reference unit 44 accesses the position information DB 22 and acquires the position and orientation estimates of the shared AR device 30 from the position information DB 22 using an ID that identifies the shared AR device 30.

[0047] In step S13, the non-shared AR device 40 calculates the position and orientation of the non-shared AR device 40 in global coordinates based on the position and orientation estimates of the shared AR device 30 in global coordinates and information on the relative positional relationship with the shared AR device 30. The shared AR device 30 and the non-shared AR device 40 are set to a predetermined relative positional relationship in advance. For example, the error correction unit 43 holds information on this relative positional relationship in advance. The relative positional relationship is, for example, a relative position and orientation. The relative position and orientation represent the relative positional relationship and the relative orientation relationship. The error correction unit 43 calculates the position and orientation of the non-shared AR device 40 in global coordinates based on the position and orientation estimates of the shared AR device 30 in global coordinates and the relative position and orientation with the shared AR device 30. This determines the position and orientation of the non-shared AR device 40 in the global coordinate system. A method for calculating the position and orientation in the global coordinate system will be described later.

[0048] In step S14, the non-shared AR device 40 displays the AR object. More specifically, the AR object display unit 41 displays the AR object. The AR object display unit 41 is set in advance to display the AR object at the position and orientation of the shared AR device 30. Therefore, if the shared AR device 30 and the non-shared AR device 40 are in the correct relative position and orientation, the AR object is displayed in the same position and orientation as the shared AR device 30. On the other hand, if the shared AR device 30 and the non-shared AR device 40 are in an incorrect relative position and orientation, the AR object is displayed in a position and orientation different from that of the shared AR device 30.

[0049] Furthermore, the non-shared AR device 40 prompts the user to check whether there is a discrepancy in the position and orientation between the shared AR device 30 and the AR object. To this end, the error correction unit 43 displays information prompting the user to check whether there is a discrepancy in the position and orientation between the shared AR device 30 and the AR object via the AR object display unit 41, and accepts the user's check result. For example, the error correction unit 43 displays a question about whether the AR object overlaps the shared AR device 30 via the AR object display unit 41 to check whether there is a discrepancy in the position and orientation between the shared AR device 30 and the AR object, displays an icon as an answer option for whether the AR object overlaps the shared AR device 30, and receives the user's check result by detecting a user operation on the icon.

[0050] In step S15, if the result of checking whether there is a discrepancy in the position and orientation between the shared AR device 30 and the AR object is that the AR object overlaps the shared AR device 30, that is, if there is a discrepancy in the position and orientation between the shared AR device 30 and the AR object (NO in step S15), the non-shared AR device 40 determines that the AR object will be displayed in the same position and orientation as the shared AR device 30, and terminates the processing without performing any further correction processing.

[0051] On the other hand, if the result of checking whether there is a misalignment between the position and orientation of the shared AR device 30 and the AR object in step S15 is that the AR object does not overlap the shared AR device 30, that is, if the result is that there is a misalignment between the position and orientation of the shared AR device 30 and the AR object (YES in step S15), then in step S16 the non-shared AR device 40 instructs the user to perform an operation. In detail, the error correction unit 43 displays, via the AR object display unit 41, information instructing the user to stop updating the display of the AR object, correct the misalignment between the position and orientation of the shared AR device 30 and the AR object in the update-stopped state, and cancel the stop of updating the display of the AR object in the misalignment-corrected state.

[0052] In this embodiment, relative movement between the shared AR device 30 and the non-shared AR device 40 is specified to correct the deviation in position and orientation between the shared AR device and the AR object.

[0053] Upon receiving the instruction, the user performs operations to stop updating the display of the AR object, move the shared AR device 30 and the non-shared AR device 40 relative to each other, and release the stoppage of updating the display of the AR object.

[0054] The operation to stop updating and release the update stop of the display of the AR object may be, for example, an operation on an icon for selecting update stop and update stop that is displayed by the AR object display unit 41. The error correction unit 43 receives an instruction from the user to stop updating and release the update stop by detecting the operation on the icon for stopping updating and release the update stop.

[0055] Alternatively, the operation of stopping and canceling the update stop of the display of the AR object may be, for example, an operation on an icon of options for stopping and canceling the update stop displayed on the touch display of the shared AR device 30.

[0056] In step S17, the non-shared AR device 40 stops updating the display of the AR object and displays the AR object in a fixed state. Specifically, when the error correction unit 43 receives an instruction from the user to stop updating the display of the AR object via the AR object display unit 41, the error correction unit 43 stops updating the display of the AR object on the AR object display unit 41 and displays the AR object in a fixed state on the AR object display unit 41. Thereafter, the error correction unit 43 waits for the user to cancel the stop of updating the display of the AR object.

[0057] In step S18, the user follows the instructions and performs relative movement between the shared AR device 30 and the non-shared AR device 40 so that the AR object overlaps the shared AR device 30. The relative movement between the shared AR device 30 and the non-shared AR device 40 may be performed by moving either the shared AR device 30 or the non-shared AR device 40, or by moving both. Furthermore, when the AR object overlaps the shared AR device 30, the user performs an operation to cancel the stop of updating the display of the AR object. In this state, the shared AR device 30 and the non-shared AR device 40 are in the correct relative position and orientation.

[0058] In step S19, the non-shared AR device 40 cancels the stop of updating the display of the AR objects and resumes updating the display of the AR objects. In detail, when the error correction unit 43 receives an instruction from the user to cancel the stop of updating the display of the AR objects via the AR object display unit 41, the error correction unit 43 cancels the stop of updating the display of the AR objects on the AR object display unit 41 and resumes updating the display of the AR objects on the AR object display unit 41, and records the update stop cancellation time Tq. The update stop cancellation time Tq is the time T when the deviation correction is completed.

[0059] In step S20, the non-shared AR device 40 acquires the position and orientation estimates of the shared AR device 30 in global coordinates at time T when the deviation correction is completed. That is, the position information DB reference unit 44 acquires the position and orientation estimates of the shared AR device 30 in global coordinates at time T from the position information DB 22.

[0060] In step S21, the non-shared AR device 40 calculates its own position and orientation in global coordinates at time T based on the position and orientation estimated values ​​of the shared AR device 30 and information on the relative positional relationship. That is, the error correction unit 43 calculates the position and orientation of the non-shared AR device 40 in global coordinates at time T based on the position and orientation estimated values ​​of the shared AR device 30 and the relative position and orientation with respect to the shared AR device 30. A method for calculating the position and orientation in global coordinates at time T will be described later.

[0061] Through the above processing, the position and orientation of the non-shared AR device 40 in the global coordinate system are correctly estimated.

[0062] (Method for calculating position and orientation of non-shared AR device 40 based on position and orientation estimated values ​​and relative position and orientation in step S13) Next, a method for calculating the position and orientation of the non-shared AR device 40 based on the position and orientation estimated values ​​and relative position and orientation will be described with reference to Fig. 4. As shown in Fig. 4, the position and orientation of the shared AR device 30 are Va and Ra, the position and orientation of the non-shared AR device 40 are Vb and Rb, and the relative position and orientation between the shared AR device 30 and the non-shared AR device 40 are Vrelative and Rrelative, respectively.

[0063] The position Va and the orientation Ra of the shared AR device 30 are respectively expressed by the following formula (1).

[0064]

[0065] The position Vb and the orientation Rb of the non-shared AR device 40 are respectively expressed by the following equation (2).

[0066]

[0067] The non-shared AR device 40 is set to be located at a position moved by a position Vrelative relative to the shared AR device 30 and rotated by a relative orientation Rrelative. That is, the relative position Vrelative and relative orientation Rrelative between the shared AR device 30 and the non-shared AR device 40 are respectively expressed by the following equation (3), where the rotation angle is expressed in Euler angles.

[0068]

[0069] In this case, the position Vb of the non-shared AR device 40 can be calculated by the following equation (4).

[0070]

[0071] Furthermore, the orientation Rb of the non-shared AR device 40 can be calculated by the following equation (5).

[0072]

[0073] (Method for calculating the position and orientation of the non-shared AR device 40 in global coordinates at time T when correction is completed in step S21) Next, a method for calculating the position and orientation of the non-shared AR device 40 in global coordinates at time T when correction is completed will be described with reference to Fig. 5. As shown in Fig. 5, the position and orientation of the shared AR device 30 in global coordinates at time T when correction is completed is set to VaT and RaT, and the position and orientation of the non-shared AR device 40 in local coordinates is set to VbT and RbT, respectively.

[0074] The position VbT and orientation RbT of the non-shared AR device 40 in global coordinates at time T can be calculated using the following equation (6) from the pre-set relative position Vrelative and relative orientation Rrelative and the position VaT and orientation RaT of the shared AR device 30 in global coordinates.

[0075]

[0076] (Operation Example of Second Example) Next, an operation example of a second example in the position estimation system 10 according to the embodiment will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a flowchart showing the flow of the operation example of the second example in the position estimation system 10 according to the embodiment. Fig. 7 is a diagram showing the time series flow of the operation example shown in Fig. 6.

[0077] An operation example of the second embodiment is an example in which an AR object is superimposed on a display image of the shared AR device 30 and manually corrected.

[0078] The processing from step S31 to step S35 in the operation example of the second embodiment is the same as the processing from step S11 to step S15 in the operation example of the first embodiment, and therefore will only be briefly described below.

[0079] In step S31, the shared AR device 30 requests the self-location estimation server 20 to estimate its own location, and obtains an estimation result from the self-location estimation server 20. As a result, the position and orientation of the shared AR device 30 in the global coordinate system are known.

[0080] In step S32, the unshared AR device 40 acquires estimated values ​​of the position and orientation of the shared AR device 30 in global coordinates from the position information DB 22 of the self-position estimation server 20.

[0081] In step S33, the non-shared AR device 40 calculates its own position and orientation in global coordinates based on the position and orientation estimates of the shared AR device 30 in global coordinates and information on the relative positional relationship with the shared AR device 30. This determines the position and orientation of the non-shared AR device 40 in the global coordinate system. The method of calculating the position and orientation is the same as in the first embodiment.

[0082] In step S34, the non-shared AR device 40 displays the AR object. The AR object display unit 41 is set in advance to display the AR object at the position and orientation of the display image of the shared AR device 30. Therefore, if the shared AR device 30 and the non-shared AR device 40 are in the correct relative position and orientation, the AR object is displayed in the same position and orientation as the display image of the shared AR device 30. On the other hand, if the shared AR device 30 and the non-shared AR device 40 are in an incorrect relative position and orientation, the AR object is displayed in a position and orientation different from the display image of the shared AR device 30.

[0083] Furthermore, the non-shared AR device 40 prompts the user to check whether there is a discrepancy in the position and orientation between the shared AR device 30 and the AR object. To this end, the error correction unit 43 displays information prompting the user to check whether there is a discrepancy in the position and orientation between the shared AR device 30 and the AR object via the AR object display unit 41, and accepts the user's confirmation result. For example, the error correction unit 43 displays a question about whether the AR object overlaps the display image of the shared AR device 30 via the AR object display unit 41 to check whether there is a discrepancy in the position and orientation between the shared AR device 30 and the AR object, displays icons as answer options for whether the AR object overlaps the display image of the shared AR device 30, and receives the user's confirmation result by detecting a user operation on the icon.

[0084] In step S35, if the result of checking whether there is a discrepancy in the position and orientation between the shared AR device 30 and the AR object is that the AR object overlaps the display image of the shared AR device 30, that is, if there is a discrepancy in the position and orientation between the shared AR device 30 and the AR object (NO in step S35), the non-shared AR device 40 determines that the AR object will be displayed in the same position and orientation as the shared AR device 30, and terminates the processing without performing any further correction processing.

[0085] On the other hand, if the result of checking whether there is a misalignment between the position and orientation of the shared AR device 30 and the AR object in step S35 is that the AR object does not overlap the display image of the shared AR device 30, that is, if the result is that there is a misalignment between the position and orientation of the shared AR device 30 and the AR object (YES in step S35), then in step S36 the non-shared AR device 40 instructs the user to perform an operation. In detail, the error correction unit 43 displays, via the AR object display unit 41, information instructing the user to stop updating the display of the AR object, correct the misalignment between the position and orientation of the shared AR device 30 and the AR object in the update-stopped state, and cancel the stop of updating the display of the AR object in the misalignment-corrected state.

[0086] In this embodiment, to correct the position / orientation deviation between the shared AR device and the AR object, movement of the shared AR device 30 that superimposes the AR object on the display image of the shared AR device 30 is specified.

[0087] Upon receiving the instruction, the user performs operations to stop updating the display of the AR object, move the shared AR device 30, and release the stoppage of updating the display of the AR object.

[0088] In step S37, the non-shared AR device 40 stops updating the display of the AR object and displays the AR object in a fixed state. More specifically, when the error correction unit 43 receives an instruction from the user to stop updating the display of the AR object via the AR object display unit 41, the error correction unit 43 stops updating the display of the AR object on the AR object display unit 41 and displays the AR object in a fixed state on the AR object display unit 41. Thereafter, the error correction unit 43 waits for the user to cancel the stop of updating the display of the AR object.

[0089] In step S38, the user follows the instruction to move the shared AR device 30 so that the AR object overlaps the display image of the shared AR device 30. Furthermore, the user performs an operation to cancel the stop of updating the display of the AR object when the AR object overlaps the display image of the shared AR device 30. In this state, the shared AR device 30 and the non-shared AR device 40 are in the correct relative positions and orientations.

[0090] In step S39, the non-shared AR device 40 cancels the stop of updating the display of the AR objects and resumes updating the display of the AR objects. In particular, when the error correction unit 43 receives an instruction from the user to cancel the stop of updating the display of the AR objects via the AR object display unit 41, the error correction unit 43 cancels the stop of updating the display of the AR objects on the AR object display unit 41 and resumes updating the display of the AR objects on the AR object display unit 41, and records the update stop cancellation time Tq. The update stop cancellation time Tq is the time T when the correction is completed.

[0091] In step S40, the non-shared AR device 40 calculates its own position and orientation in a state in which the misalignment correction is complete. That is, the position information DB reference unit 44 acquires, from the position information DB 22, position and orientation estimates of the shared AR device 30 in global coordinates at time T when the misalignment correction is complete. Furthermore, the error correction unit 43 calculates the position and orientation of the non-shared AR device 40 in global coordinates at time T based on the position and orientation estimates of the shared AR device 30 and the relative position and orientation with respect to the shared AR device 30. The method of calculating the position and orientation in global coordinates at time T is the same as in the first embodiment.

[0092] In step S41, the non-shared AR device 40 assumes that its own position and orientation in the correction-completed state calculated in step S40 is the position and orientation of its own calculated in step S33. In detail, the error correction unit 43 corrects the position and orientation of the non-shared AR device 40 so that the position and orientation of the non-shared AR device 40 calculated in step S33 become the position and orientation of the non-shared AR device 40 in the correction-completed state calculated in step S40.

[0093] Through the above processing, the position and orientation of the non-shared AR device 40 in the global coordinate system are correctly estimated.

[0094] (Effects) As described above, according to the embodiment, a technology is provided for estimating the self-position of a non-shared AR device without installing multiple fixed cameras. This makes it possible to provide a shared spatial AR to a non-shared AR device that does not have the hardware necessary for self-position estimation, such as a camera or LiDAR. The relative positional relationship between the shared AR device and the non-shared AR device only needs to be correct to a certain extent, and no jig is required to fix the relative positional relationship.

[0095] (Hardware Configuration) Next, a description will be given of the hardware configurations of the self-location estimation server 20, the shared AR device 30, and the non-shared AR device 40 of the position estimation system 10. For example, the self-location estimation server 20, the shared AR device 30, and the non-shared AR device 40 are each configured as a computer. Examples of computers include a personal computer, a server computer, a smartphone, an AR device, etc.

[0096] 8 is a block diagram showing an example of the hardware configuration of each of the self-location estimation server 20, the shared AR device 30, and the non-shared AR device 40 according to the embodiment, in other words, the hardware configuration of a computer 100 that constitutes each of the self-location estimation server 20, the shared AR device 30, and the non-shared AR device 40. The computer 100 has a control device 120, an input device 140, and an output device 150.

[0097] The control device 120 controls the entire computer 100. The control device 120 includes a processor 121, a read only memory (ROM) 122, a random access memory (RAM) 123, and an auxiliary storage device .

[0098] The processor 121, ROM 122, RAM 123, auxiliary storage device 124, input device 140, and output device 150 are electrically connected to one another via a bus 130, and can transmit and receive data.

[0099] The processor 121 is configured by a general-purpose hardware processor including, for example, a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), etc. The processor 121 executes various functions of the computer 100 by executing programs deployed in the RAM 123.

[0100] The ROM 122 is a non-volatile memory that constitutes part of the main storage device. The ROM 122 non-temporarily stores a startup program required to start up the computer 100. The processor 121 loads the startup program in the ROM 122 into the RAM 123 and executes it to start up the computer 100. The ROM 122 is configured, for example, from an erasable programmable read-only memory (EPROM), and is capable of storing various startup settings in addition to the startup program.

[0101] The RAM 123 is a volatile memory that constitutes part of the main storage device. The RAM 123 temporarily stores programs required for processing by the processor 121 and data required for executing the programs. In other words, the RAM 123 functions as a work area for the processor 121.

[0102] The auxiliary storage device 124 is configured with non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD). The auxiliary storage device 124 can non-temporarily store various programs executed by the processor 121 and data required to execute the programs. The processor 121 executes various functions of the computer 100 by loading the programs in the auxiliary storage device 124 into the RAM 123 and executing them.

[0103] The input device 140 is a device for a user to input information and instructions, and accepts input of information and instructions. The input device 140 includes a keyboard, a pointing device, etc. The pointing device includes a mouse, a trackpad, a touch screen, etc.

[0104] The output device 150 is a device that outputs information to provide information to a user. The output device 150 is, for example, a display device that displays characters, images, etc. on a screen. For example, the output device 150 is a liquid crystal display, an organic EL display, a plasma display, etc.

[0105] The output device 150 and the input device 140 may be configured as an input / output device having the functions of both. Such an input / output device may be configured as, for example, a touch panel display.

[0106] The input device 140 may also include a device that inputs information or data from an external device. For example, the input device 140 may include a wired or wireless interface or receiving device. The input device 140 may also include a camera.

[0107] The output device 150 may also include a device that outputs information or data to the outside. For example, the output device 150 may include a wired or wireless interface or transmission device.

[0108] The input device 140 may also include a device that reads data from a computer-readable recording medium 160 that non-temporarily records data such as a program. For example, the recording medium 160 includes disks such as flexible disks, optical disks (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), magneto-optical disks (MO, etc.), semiconductor memories, etc. The input device 140 includes drives, readers, etc. for these.

[0109] The program stored in the auxiliary storage device 124 is provided to the computer 100, for example, via the recording medium 160. Alternatively, the program may be stored on a server on a network and provided to the computer 100 by downloading it.

[0110] For example, when the computer 100 starts up, the processor 121 executes a startup program in the ROM 122 to start up the operating system (OS). Under the control of the OS, the processor 121 monitors input instructions, connections to external devices, etc. Also, under the control of the OS, the processor 121 sets up a program area and a data area in the RAM 123.

[0111] In response to an instruction to start a program, processor 121 reads the program from auxiliary storage device 124 into the program area of ​​RAM 123, and also reads data necessary for executing the program from auxiliary storage device 124 into the data area of ​​RAM 123. Processor 121 calculates the data in the data area in accordance with the program and writes the calculation results into the data area.

[0112] Through these operations, the processor 121, RAM 123, and auxiliary storage device 124 work together to execute at least a portion of the functions of the control device 120. Furthermore, the control device 120, input device 140, and output device 150 work together to execute at least a portion of the functions of the computer 100.

[0113] The programs non-temporarily stored in the auxiliary storage device 124 include a program that causes the processor 121 to execute at least some of the functions of the control device 120. In other words, the processor 121 executes this program to execute at least some of the functions of the control device 120.

[0114] As a result, the control device 120 cooperates with the input device 140 and the output device 150 to execute at least part of the functions of the self-location estimation server 20 , the shared AR device 30 , or the non-shared AR device 40 .

[0115] (Other) Embodiments of the present invention have been described above with reference to the drawings. However, the above embodiment is merely an example of a configuration that embodies the present invention. In other words, it is clear that the present invention is not limited to the above embodiment. Therefore, additions, omissions, substitutions, and other modifications of components may be made within the scope of the technical concept of the present invention.

[0116] In short, the present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0117] 10... Position estimation system 20... Self-position estimation server 21... Self-position estimation unit 22... Position information database 23... DB reference reception unit 30... Shared AR device 31... Self-position estimation client unit 40... Non-shared AR device 41... AR object display unit 42... Relative movement amount measurement unit 43... Error correction unit 44... Position information DB reference unit 100... Computer 120... Control device 121... Processor 122... ROM 123... RAM 124... Auxiliary storage device 130... Bus 140... Input device 150... Output device 160... Recording medium

Claims

1. A position estimation system comprising: a shared device; a server that estimates the position and orientation of the shared device in a global coordinate system; and a non-shared device whose relative position and orientation with respect to the shared device is set, wherein the non-shared device comprises: a reference unit that acquires estimated values ​​of the position and orientation of the shared device from the server; a display unit that displays an object at the position and orientation of the shared device; and a correction unit that calculates the position and orientation of the non-shared device in the global coordinate system based on the estimated values ​​of the position and orientation and the relative position and orientation.

2. The position estimation system according to claim 1, wherein the correction unit displays information via the display unit prompting the user to check whether there is a misalignment in the position and orientation of the shared device and the object, and accepts the user's confirmation results.

3. The position estimation system of claim 2, wherein the correction unit, in response to the confirmation result that there is a deviation, displays information instructing the user to stop updating the display of the object, correct the deviation between the position and orientation of the shared device and the object when the updates are stopped, and release the stoppage of updating the display of the object when the deviation correction is complete.

4. The position estimation system according to claim 3, wherein the correction unit displays information instructing the user to perform an operation of relative movement of the shared device and the non-shared device to overlap the shared device on the object as the operation of correcting the deviation.

5. The position estimation system according to claim 4, wherein the correction unit calculates the position and orientation of the non-shared device based on the position and orientation estimated values ​​at the time when the update stop is released and the relative position and orientation.

6. The position estimation system of claim 3, wherein the display unit displays information instructing the user to move the shared device to superimpose the object on the display image of the shared device as the misalignment correction operation, and the correction unit corrects the position and orientation of the non-shared device so that the position and orientation of the non-shared device at the time of calculating the position and orientation become the position and orientation of the non-shared device at the time the update stop is released.

7. A position estimation method comprising the steps of: acquiring estimated values ​​of a position and orientation of a shared device in a global coordinate system; displaying an object at the position and orientation of the shared device; and calculating, based on the estimated values ​​of position and orientation and the relative position and orientation, the position and orientation of a non-shared device in the global coordinate system, the relative position and orientation of which has been set with respect to the shared device.

8. A location estimation program that causes a computer to execute at least a portion of the functions of each component of the location estimation system according to claim 1.

Images