Information processing system, information processing device, terminal device and control method of information processing system

Abstract

To suitably estimate a position or attitude of a terminal device in which processing performance and storage capacity are restricted.SOLUTION: An information processing system comprises an information processing device and a terminal device, and estimates a position or attitude of the terminal device. The information processing device comprises: first capturing means; first acquisition means which acquires information on a position or attitude of the information processing device by using captured images by the first capturing means; generation means which generates map information on the basis of the information on the position or attitude of the information processing device, and a key frame image which is the captured image captured with the position or attitude; extraction means which extracts a part of the map information; and communication means which transmits extraction map information extracted by the extraction means to the terminal device. The terminal device comprises: second capturing means; and second acquisition means which acquires the information on the position or attitude of the terminal device, by using the captured image by the second capturing means and the extraction map information transmitted from the information processing device.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing device, a terminal device, and a method for controlling an information processing system. [Background technology]

[0002] In recent years, technologies for estimating self-position and creating environmental maps by mounting an imaging device on a terminal device and using SLAM (Simultaneous Localization and Mapping) have been used for various purposes. For example, Patent Document 1 discloses an automated guided vehicle that is equipped with a map update function using SLAM and is capable of autonomous driving, and that uses coordinate information of landmark images projected onto the driving path to improve the accuracy of autonomous driving. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-093887 [Non-patent literature]

[0004] [Non-Patent Document 1] Rainer Kummerle, et al., “g2o: A General Framework for Graph Optimization”, [online], May 9, 2011, 2011 IEEE International Conference on Robotics and Automation, Shanghai International Conference Center, [Retrieved June 27, 2022], Internet<URL:http: / / ais.informatik.uni-freiburg.de / publications / papers / kuemmerle11icra.pdf> [Non-Patent Document 2] R. Mur-Artal, et al., “ORB-SLAM: A Versatile and Accurate Monocular SLAM System”, [online], October 5, 2015, IEEE Transactions on Robotics, vol. 31, no. 5, pp. 1147-1163, [Searched on July 5, Reiwa 4], Internet <URL:https: / / ieeexplore.ieee.org / document / <7219438>

Summary of the Invention

Problems to be Solved by the Invention

[0005] A controller (terminal device) such as an HMD (head-mounted display) or a game console can accurately estimate its position and orientation by referring to a map created by SLAM. However, due to limitations in processing power and memory capacity to reduce weight and size, the controller has limitations in estimating position and orientation by SLAM. The controller can estimate its position and orientation (position and orientation) by odometry (autonomous navigation), but errors accumulate. Therefore, it is difficult for the controller to reduce the processing load and estimate the position and orientation with high accuracy.

[0006] An object of the present invention is to provide a technique that enables suitable estimation of position and orientation in a terminal device with limited processing power and memory capacity.

Means for Solving the Problems

[0007] In order to achieve the above object, an information processing system of the present invention includes an information processing device and a terminal device, and is an information processing system for estimating a position and orientation of the terminal device, wherein the information processing device includes a first imaging means and a first acquisition means for acquiring information on the position and orientation of the information processing device using an image captured by the first imaging means, a generation means for generating map information based on the information on the position and orientation of the information processing device and a key frame image which is an image captured at the position and orientation, an extraction means for extracting a part of the map information, and and a communication means for transmitting the extracted extraction map information to the terminal device, the terminal device having a second imaging means and a second acquisition means for acquiring information on the position and orientation of the terminal device using an image captured by the second imaging means and the extraction map information transmitted from the information processing device. Effect of the Invention

[0008] According to the present invention, it is possible to estimate a suitable position and orientation on a terminal device with limited processing power and storage capacity. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram illustrating the configuration of an HMD and a controller. [Diagram 2] FIG. 1 is a diagram illustrating an information processing system according to a first embodiment. [Diagram 3] FIG. 4 is a diagram illustrating key frame information. [Figure 4] FIG. 13 is a diagram illustrating a first extraction method for the extraction map. [Diagram 5] FIG. 13 is a diagram illustrating a second extraction method for the extraction map. [Figure 6] 13 is a flowchart showing a first example of a third extraction method for the extraction map. [Figure 7] FIG. 13 is a diagram illustrating a first example of a third extraction method for the extraction map. [Figure 8] 13 is a flowchart showing a second example of a third extraction method for the extraction map. [Figure 9] FIG. 11 is a diagram illustrating an information processing system according to a second embodiment. [Figure 10] FIG. 11 is a diagram illustrating an information processing system according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment described below is one example of a method for realizing the present invention, and may be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions. In addition, each embodiment may be appropriately combined.

[0011] <Embodiment 1> 1 is a block diagram illustrating a configuration of an information processing system according to an embodiment of the present invention. The information processing system includes a head mounted display (HMD) 100 as an information processing device and a controller 200 as a terminal device.

[0012] The HMD 100 is a video see-through type HMD that displays an image of the outside world (real space) by synthesizing graphics (e.g., virtual objects) as necessary. The controller 200 can operate, for example, virtual objects displayed on the HMD 100.

[0013] 1A is a block diagram illustrating the configuration of an HMD 100. A CPU 101 reads out a control program for each block of the HMD 100 from a ROM 102, and develops and executes the program in a RAM 103. In this way, the CPU 101 controls the operation of each block of the HMD 100. Some of the processing executed by the CPU 101 may be executed by a hardware circuit.

[0014] The ROM 102 is a non-volatile memory that can be electrically erased and recorded. The ROM 102 stores operation programs for each block of the HMD 100, as well as parameters used for the operation of each block.

[0015] The RAM 103 is a rewritable volatile memory and is used, for example, to expand programs executed by the CPU 101 and the like, and to temporarily store data generated by the operation of each block of the HMD 100.

[0016] The imaging unit 104 is a camera having an optical system (lens), an imaging element such as a CCD or CMOS sensor, an A / D converter, etc. The imaging unit 104 photoelectrically converts an optical image formed on an imaging surface by the optical system, and outputs an analog image signal. The analog image signal is converted into digital image data by the A / D converter and temporarily stored in the RAM 103.

[0017] The display unit 105 controls the display of images captured by the imaging unit 104 and other visual objects. The input unit 106 accepts operations from a user. The storage unit 107 stores images captured by the imaging unit 104, application programs, various data generated by the application programs, and the like.

[0018] The communication unit 108 is an interface for communicating with an external device by wire or wirelessly. The communication unit 108 is capable of wireless communication with other devices such as the controller 200 by using a communication standard such as Wi-Fi and / or BLE (Bluetooth (registered trademark) Low Energy).

[0019] The CPU 101 has, as functional blocks, a position and orientation acquisition unit 109, a map generation unit 110, and a map extraction unit 111. The position and orientation acquisition unit 109 acquires information on the position and orientation of the HMD 100. The position and orientation acquisition unit 109 can acquire the information on the position and orientation from an image captured by the imaging unit 104 using SLAM technology.

[0020] The map generation unit 110 generates map information of SLAM used by the position and orientation acquisition unit 109 to acquire information on the position and orientation of the HMD 100. For example, the map generation unit 110 associates a periodically captured key frame image with information on feature points on the image and information on the position and orientation of the HMD 100 at the time of capture, and records the map information (map information) in the storage unit 107. As disclosed in Non-Patent Document 2, the position and orientation acquisition unit 109 can estimate information on the position and orientation of the HMD 100 by optimizing the position and orientation of the HMD 100 using feature points of map information, which is a collection of key frame information, while tracking feature points detected from the captured image and creating and optimizing key frames as appropriate.

[0021] The map extraction unit 111 extracts map information used to estimate the position designation of the controller 200 from the map information of the HMD 100 generated by the map generation unit 110. The extracted map information is also referred to as extracted map information. The extracted map information is transmitted to the controller 200. Since the extracted map information is generated by extracting a portion of the map information of the HMD 100, the amount of data transmitted to the controller 200 is suppressed, and the processing load of the controller 200 is reduced.

[0022] 1(B) is a block diagram illustrating an example of the configuration of the controller 200. The CPU 201 reads out a control program for each block of the controller 200 from the ROM 202, and loads and executes the program in the RAM 203. In this way, the CPU 201 controls the operation of each block of the controller 200. Some of the processing executed by the CPU 201 may be executed by a hardware circuit.

[0023] The ROM 202 is a non-volatile memory that is electrically erasable and recordable. The ROM 202 stores operation programs for each block of the controller 200, as well as parameters used for the operation of each block.

[0024] The RAM 203 is a rewritable volatile memory and is used, for example, to expand programs executed by the CPU 201 and the like, and to temporarily store data generated by the operation of each block included in the controller 200.

[0025] The imaging unit 204 is a camera having an optical system, an imaging element such as a CCD or CMOS sensor, an A / D converter, etc., similar to the imaging unit 104 of the HMD 100. The imaging unit 204 may include multiple cameras. The imaging unit 204 is, for example, a monochrome monocular camera.

[0026] The input unit 206 accepts operations from a user. The storage unit 207 stores images captured by the imaging unit 204, application programs, various data generated by the application programs, etc. The communication unit 208 is an interface for communicating with external devices by wire or wirelessly, and enables wireless communication with other devices such as the HMD 100 using communication standards such as Wi-Fi and / or BLE.

[0027] The CPU 201 has a position and orientation acquisition unit 209 as a functional block. The position and orientation acquisition unit 209 acquires information on the position and orientation of the controller 200. The position and orientation acquisition unit 209 acquires information on the position and orientation of the controller 200 from an image captured by the imaging unit 204. The controller 200 also has an IMU (Inertial Measurement Unit) (not shown), and the position and orientation acquisition unit 209 may estimate the position and orientation of the controller 200 based on the acceleration and angular velocity measured by the IMU.

[0028] The position and orientation acquisition unit 209 may acquire information on the position and orientation of the controller 200 by an image captured by the imaging unit 204 or odometry (autonomous navigation) using an IMU, and may acquire the information on the position and orientation by using the extraction map information at a predetermined timing. That is, the position and orientation acquisition unit 209 acquires information on the position and orientation of the controller 200 by using an image captured by the imaging unit 204 and the extraction map information extracted by the map extraction unit 111 at a predetermined timing.

[0029] The predetermined timing is, for example, a regular or fixed timing. The frequency of acquiring the position and orientation information of the controller 200 using the extraction map information may be determined according to the processing capacity of the controller 200, and the lower the processing capacity, the lower the frequency may be. In addition, the predetermined timing may be a timing when the accuracy of the position and orientation information by the odometry is reduced due to an increase in the moving speed of the controller 200, an irregular movement, or the like.

[0030] When acquiring position and orientation information by odometry, errors accumulate, but the position and orientation acquisition unit 209 can eliminate the accumulated errors and acquire position and orientation information of the controller 200 with high accuracy by using the captured image by the imaging unit 204 and the extraction map information.

[0031] FIG. 2 is a diagram for explaining an information processing system according to the first embodiment. The information processing system 1 includes an HMD 100 and a controller 200. A position and orientation acquisition unit 109 of the HMD 100 acquires information on the position and orientation of the HMD 100 using an image captured by an imaging unit 104. The HMD 100 generates a map 120 (map information) based on the information on the position and orientation of the HMD 100 and a key frame image, which is an image captured in the position and orientation. The map 120 also includes information on feature points detected from the key frame image. The HMD 100 extracts a part of the map 120 to generate an extraction map 121 (extraction map information). The generated extraction map 121 is transmitted to the controller 200.

[0032] A position and orientation acquisition unit 209 of the controller 200 acquires information about the position and orientation of the controller 200, using an image captured by the imaging unit 204. A position and orientation correction unit 210 acquires information about the position and orientation of the controller 200, using an image captured by the imaging unit 204 and an extraction map 121 received from the HMD 100, at a predetermined timing.

[0033] FIG. 3 is a diagram for explaining key frame information. The example in FIG. 3 shows the distribution of key frames KF1 to KF6 on the trajectory of the HMD 100. Each key frame is The map 120 includes key frame information that associates information on the position and orientation of D100 with key frame images captured at the corresponding position and orientation. The map 120 includes a plurality of pieces of key frame information, and for simplification, only some of the key frames are shown in Fig. 3. In addition, although the positions of the key frames are illustrated in two dimensions, they actually correspond to positions in a three-dimensional space.

[0034] The key frame information is generated, for example, for each predetermined time or for each predetermined moving distance. The predetermined time and the predetermined moving distance can be determined, for example, according to the moving speed and processing power of the HMD 100. The generated key frame information is stored in the storage unit 107 as the map 120.

[0035] The key frame information includes a key frame image, position and orientation information, key frame tree structure information, and a plurality of pieces of feature point information. The key frame image is an image captured by the imaging unit 104 when the key frame information is created. The key frame image may include information on the creation date and time of the key frame image as meta information. The creation date and time of the key frame image may be held as the key frame information.

[0036] The position and orientation information is information on the position and orientation of the HMD 100 when a key frame image is captured, and is expressed in, for example, an XYZ coordinate system. The key frame tree structure information is information indicating the relationship between key frames, such as the shooting order and positional relationship. The feature point information is information on the positions of feature points captured in the key frame image.

[0037] A method in which the map extraction unit 111 of the HMD 100 extracts key frame information from the map 120 and generates the extraction map 121 will be described with reference to FIGS.

[0038] (First extraction method) 4 is a diagram illustrating a first extraction method for the extraction map 121. The map extraction unit 111 extracts key frame information created in the peripheral area of ​​the controller 200 from the map 120. In the example of FIG. 4, when the peripheral area of ​​the controller 200 is a rectangle 401, the key frame information of the key frames KF1 and KF2 is extracted as the extraction map 121 and transmitted to the controller 200.

[0039] When controller 200 moves in the direction of arrow 410, map extraction unit 111 extracts key frame information of key frames KF2 to KF4 included in rectangle 402 to generate extraction map 121. In response to the movement of controller 200, the surrounding area is further changed to rectangle 403 and rectangle 404, and key frame information included in each rectangular area is extracted as extraction map 121. Map extraction unit 111 generates extraction map 121, for example, when controller 200 moves for a predetermined time or a predetermined distance. The generated extraction map 121 is transmitted to controller 200 by communication unit 108.

[0040] The surrounding area of ​​the controller 200 may be, for example, a rectangular parallelepiped or spherical area centered on the position of the controller 200. The position of the controller 200 is acquired from the controller 200, for example, when the surrounding area is determined. The size of the surrounding area may be determined based on, for example, the processing power of the CPU 201, the storage capacity of the storage unit 207, and the number of pieces of key frame information within the surrounding area.

[0041] The map extraction unit 111 of the HMD 100 can acquire the position of the controller 200 and determine the surrounding area by, for example, detecting the controller 200 from an image captured by the imaging unit 104. The map extraction unit 111 may also determine the surrounding area by using position information received from the controller 200.

[0042] The transmission ratio of the key frame information extracted from the map 120 by the map extraction unit 111 and transmitted to the controller 200 may be changed based on the processing capability and storage capacity of the controller 200. In other words, the map extraction unit 111 changes the extraction amount or extraction ratio of the key frame information extracted from the map 120 based on at least one of the processing capability and storage capacity of the controller 200.

[0043] Furthermore, the map extraction unit 111 may change the extraction amount or extraction ratio of key frame information extracted from the map 120 based on the movement speed of the controller 200. For example, when it is expected that the movement speed will increase based on information such as the type of application executed by the controller 200 and the history of the movement speed during past use, it is preferable that the HMD 100 increases the extraction amount or extraction ratio of the extraction map 121. In this case, the extraction amount or extraction ratio of the extraction map 121 may be set in advance according to the type of application, etc.

[0044] Furthermore, when the controller 200 comes to a standstill, the HMD 100 may increase the extraction amount of the extraction map 121 in preparation for the next movement of the controller 200.

[0045] Furthermore, the map extraction unit 111 may change the amount or ratio of key frame information extracted from the map 120, based on the acquisition status of the position and orientation information of the controller 200. For example, when the controller 200 is in a lost state in which the position and orientation information cannot be acquired due to blurring of the image captured by the imaging unit 204, the map extraction unit 111 may increase the amount or ratio of extraction.

[0046] (Second Extraction Method) 5 is a diagram illustrating a second extraction method for the extraction map 121. The map extraction unit 111 generates the extraction map 121 by extracting representative key frame information for each divided area obtained by dividing the entire area (experiential area) in which key frames (positions where key frame images are captured) are distributed.

[0047] 5, the map extraction unit 111 extracts key frame information by selecting key frame KF1, key frame KF2, key frame KF4, and key frame KF6 from divided areas 501 to 504, respectively. The map extraction unit 111 can reduce the amount of extraction by extracting representative key frame information for each divided area.

[0048] The map extraction unit 111 can select representative keyframe information from each divided region based on the quality of the keyframe information. The quality of the keyframe information is, for example, the amount of correction of the keyframe position generated when performing keyframe optimization processing (see Non-Patent Document 1), the number of feature points reflected in the keyframe image (the number of remaining feature points if there is a processing to later remove feature points), the proportion of blur in the keyframe image, the number of moving objects in the keyframe image, or the generation time of the keyframe information. The map extraction unit 111 can select, as representative, keyframe information with a smaller amount of correction of the keyframe position, keyframe information with a larger number of feature points associated with the keyframe image, keyframe information with less blur, keyframe information with fewer moving objects, or keyframe information with a newer creation date and time, from among the keyframe information in the divided region.

[0049] The number of divided regions and the size of the divided regions may be changed based on the size of the entire region (experiential region) in which key frames are distributed. Since the amount of extraction increases as the experiential region becomes larger and the number of divided regions increases, the map extraction unit 111 may reduce the number of divisions by increasing the size of the divided regions. When the number of divided regions and the size of the divided regions are changed, the map extraction unit 111 generates an extraction map 121 and transmits the generated extraction map 121 via the communication unit 108. The packet 121 may be transmitted to the controller 200 .

[0050] (Third Extraction Method) 6 to 8, a third extraction method for the extraction map 121 will be described. The map extraction unit 111 extracts key frame information based on the variance of positions of feature points captured in key frame images or on the reprojection errors of feature points.

[0051] 6 and 7 are diagrams for explaining a first example of the third extraction method for extracting keyframe information based on the variance of the positions of feature points captured in a keyframe image. A keyframe image includes a plurality of feature points, and in the first example, the map extraction unit 111 calculates the variance of the positions of the plurality of feature points captured in the keyframe image. The map extraction unit 111 can use, for example, the average value of the variance of each feature point as an index value for extracting keyframe information. The map extraction unit 111 preferentially extracts a keyframe image having a smaller average variance of the positions of each feature point included in the keyframe image. FIG. 6 is a flowchart for explaining a first example of a calculation method for an index value for measuring the quality of each keyframe in order to extract keyframe information.

[0052] The map extraction unit 111 executes a process L1 to calculate the average variance of the positions of feature points in all key frame images i (i=1, ..., N, N is a natural number in the example of Figure 6) included in the map 120.

[0053] In step S101 of process L1, the map extraction unit 111 acquires information on feature points captured in key frame image i. The information on feature points at this time is information before optimization processing as shown in Non-Patent Document 1 is performed, and includes information on depth values ​​observed from key frames and coordinates in the image. Even for the same feature point, the three-dimensional position differs slightly depending on the observed key frame.

[0054] The map extraction unit 111 extracts all feature points j (j=1, . . . , M in the example of FIG. 6) captured in the key frame image i. i , M i where j is a natural number that differs for each keyframe image i), a process L2 is executed to calculate the variance of the position of feature point j.

[0055] In step S102 of process L2, the map extraction unit 111 calculates the variance of the positions of feature points j. The map extraction unit 111 calculates the variance of the positions of all feature points j (j=1, ..., M i Once the variance of the positions of (1, 2, 3) is calculated, the process proceeds to step S103.

[0056] In step S103 of process L1, the map extraction unit 111 calculates, for the key frame image i, the average of the variances of the positions of each feature point j obtained in step S102 as an index value. Note that it is sufficient to obtain one index value from the variances of the positions of each feature point j, and the method of calculating the index value is not necessarily limited to the method of calculating the average. In addition to the simple arithmetic average, various methods such as the median and weighted average can be used to calculate the index value.

[0057] After calculating the average variance of feature point positions for all key frame images i (i=1, ..., N) included in the map 120, the map extraction unit 111 proceeds to step S104. In step S104, the map extraction unit 111 extracts key frame information for which the average variance of feature point positions calculated in step S103 is equal to or smaller than a threshold value. The extracted key frame information is transmitted to the controller 200 by the communication unit 108.

[0058] 7, the map extraction unit 111 extracts key frame information of key frames KF1 and KF4 in which the average variance of the feature point positions calculated as index values ​​is less than 0.40. In this manner, the map extraction unit 111 sets a threshold value and extracts key frame information of key frames KF1 and KF4 in which the average variance of the feature point positions calculated as index values ​​is less than the threshold value. Key frame information may be extracted, or a predetermined percentage of key frame information may be extracted from the area with the lowest variance.

[0059] Fig. 8 is a diagram for explaining a second example of the third extraction method for extracting keyframe information based on the reprojection error of feature points captured in a keyframe image. The map extraction unit 111 preferentially extracts a keyframe image having a smaller reprojection error of the feature points. Fig. 8 is a flowchart for explaining a second example of a calculation method for an index value that measures the quality of each keyframe in order to extract keyframe information.

[0060] The map extraction unit 111 executes process L3 to calculate the average of reprojection errors of feature points in all key frame images i (i=1, ..., N, N is a natural number in the example of Fig. 8) included in the map 120. The process of step S201 of process L3 is the same as step S101 in Fig. 6.

[0061] The map extraction unit 111 extracts all feature points j (in the example of FIG. 8, j=1, . . . , M) captured in the key frame image i. i , M i where i is a natural number that differs for each keyframe image i), a process L4 is executed to calculate the reprojection error of feature point j.

[0062] In step S202 of process L4, the map extraction unit 111 calculates the reprojection error of feature point j. When a projection plane is set, for example, 1 m ahead based on the position and orientation of the HMD 100 acquired by the position and orientation acquisition unit 109, and each feature point included in the map is reprojected, the reprojection error is the difference between the image coordinates of the feature point visible in the current position and orientation and the two-dimensional coordinates of the feature point reprojected onto the projection plane. The map extraction unit 111 calculates the reprojection error of all feature points j (j=1, ..., M iOnce the reprojection error of , is calculated, the process proceeds to step S203.

[0063] In step S203 of process L3, the map extraction unit 111 calculates the average of the reprojection errors of each feature point j obtained in step S202 for the key frame image i as an index value. Note that it is sufficient to obtain one index value, and the method of calculating the index value is not necessarily limited to the method of calculating the average. In addition to the simple arithmetic average, various methods such as the median and weighted average can be used to calculate the index value.

[0064] After calculating the average of the reprojection errors of the feature points for all keyframe images i (i=1, ..., N) included in the map 120, the map extraction unit 111 proceeds to step S204. In step S204, the map extraction unit 111 extracts keyframe information for which the average of the reprojection errors of the feature points calculated in step S203 is equal to or smaller than a threshold value. The extracted keyframe information is transmitted to the controller 200 by the communication unit 108.

[0065] The map extraction unit 111 can extract keyframe information based on various index values ​​indicating the quality of the keyframe, in addition to the variance of the positions of feature points captured in the keyframe image and the reprojection error of the feature points.

[0066] (Other extraction methods) In addition to the above first to third extraction methods, an extraction map 121 may be generated from map 120 based on the quality of the keyframe information, as in the case where representative keyframe information is selected from a divided region in the second extraction method. The quality of the keyframe information may be, for example, the amount of correction of the keyframe position generated when performing keyframe optimization processing (see Non-Patent Document 1), the number of feature points appearing in the keyframe image (the number of remaining feature points if there is a process of removing feature points later), the proportion of blur in the keyframe image, the number of moving objects in the keyframe image, or the time of generation of the keyframe information.

[0067] The map extraction unit 111 may extract, from among the key frame information of the map 120, a predetermined number (a predetermined ratio) of key frame information in the order of the smallest correction amount of the key frame position as a result of the optimization process of the key frame, or key frame information in which the correction amount of the key frame position is equal to or greater than a threshold value. The map extraction unit 111 may also extract, from among the key frame information in the map 120, a predetermined number (a predetermined ratio) of key frame information in the order of the largest number of feature points reflected in the key frame image, or key frame information in which the number of feature points associated with the key frame image is equal to or greater than a threshold value. The map extraction unit 111 may also extract key frame information in which the ratio of the blurred area in the key frame image is equal to or less than a threshold value. The map extraction unit 111 may also extract key frame information in which the number of moving objects detected in the key frame image is equal to or less than a predetermined number, or the ratio of the area occupied by the moving objects in the key frame image is equal to or less than a threshold value. The map extraction unit 111 may also extract a predetermined number (a predetermined ratio) of key frame information in the order of the most recent creation date and time of the key frame information.

[0068] The map extraction unit 111 may also extract key frame information including a marker indicating a specific position. Furthermore, the map extraction unit 111 may extract key frame information from the map 120 based on the imaging direction of the imaging unit 204 of the controller 200. In order to preferentially extract key frame information in a direction in which the imaging unit 204 is often oriented, for example, weights are assigned in advance to the positive and negative directions of the three axes. A larger weight is set for a direction in which the imaging unit 204 is often oriented. The map extraction unit 111 can generate the extraction map 121 by extracting a number of key frame information proportional to the weight of each direction from each direction. The map extraction unit 111 may extract key frame information by combining a plurality of conditions selected from the above conditions.

[0069] The extraction map 121 is transmitted to the controller 200 multiple times depending on the extraction method, and the controller 200 may sequentially delete the extraction maps 121 received in the past, starting from the oldest extraction map 121, depending on the storage capacity of the controller 200.

[0070] According to the above-described first embodiment, the controller 200 can accurately estimate the position and orientation of the controller 200 by acquiring information on the position and orientation of the controller 200 using the extraction map 121 generated by the HMD 100. Furthermore, since the extraction map 121, which is an extraction of a part of the map 120, is received from the HMD 100, even the controller 200, which has a smaller processing power and storage capacity than the HMD 100, can preferably estimate the position and orientation.

[0071] The above first to third extraction methods and other extraction methods can be applied in appropriate combination. Furthermore, the HMD 100 may generate an extraction map 121 for each of a plurality of controllers 200, not limited to one controller 200, and transmit the corresponding extraction map 121 to the plurality of controllers 200.

[0072] <Embodiment 2> In the first embodiment, the controller 200 receives the extraction map 121 from the HMD 100, and acquires information on the position and orientation of the controller 200 based on the image captured by the imaging unit 204 and the received extraction map 121. In contrast, in the second embodiment, the controller 200 transmits the image captured by the imaging unit 204 to the HMD 100, and causes the HMD 100 to acquire information on the position and orientation of the controller 200.

[0073] The configuration of the HMD 100 according to the second embodiment is the same as that shown in Fig. 1(A), but the processing of the communication unit 108 differs from that of the first embodiment. The configuration of the controller 200 according to the second embodiment is the same as that shown in Fig. 1(B), but the processing of the communication unit 208 and the position and orientation acquisition unit 209 differs from that of the first embodiment. The processing that differs from that of the first embodiment will be described below.

[0074] 9 is a diagram for explaining an information processing system according to the second embodiment. The communication unit 208 of the controller 200 transmits an image captured by the imaging unit 204 to the HMD 100. The HMD 100 acquires information on the position and orientation of the controller 200 based on the image captured by the imaging unit 204 received from the controller 200 and the map 120 generated by the map generation unit 110. The map 120 is map information generated based on the information on the position and orientation of the HMD 100 and a key frame image captured at the position and orientation, as in the first embodiment. The communication unit 108 of the HMD 100 transmits the acquired information on the position and orientation of the controller 200 to the controller 200.

[0075] The controller 200 transmits captured images to the HMD 100 at predetermined time intervals, and receives information on the position and orientation of the controller 200 acquired by the HMD 100. After receiving the information on the position and orientation of the controller 200 from the HMD 100, the controller 200 may acquire information on the position and orientation of the controller 200 by odometry until it next transmits a captured image to the HMD 100.

[0076] According to the above-mentioned second embodiment, the controller 200 receives information on the position and orientation of the controller 200 acquired in the HMD 100 using the map 120, and therefore can estimate the position and orientation of the controller 200 with high accuracy. Furthermore, the controller 200 does not execute processes for generating the map 120 and acquiring information on the position and orientation of the controller 200, and therefore the processing load is reduced. Furthermore, since map information is not held, the storage capacity may be smaller than that of the controller 200 of the first embodiment.

[0077] <Embodiment 3> In the second embodiment, the controller 200 transmits an image captured by the imaging unit 204 to the HMD 100, and causes the HMD 100 to acquire information on the position and orientation of the controller 200. In contrast, in the third embodiment, the controller 200 transmits an image captured by the imaging unit 204 to the HMD 100, and causes the HMD 100 to generate a map for the controller 200.

[0078] The configuration of the HMD 100 according to the third embodiment is the same as the configuration in Fig. 1(A) except for the map extraction unit 111, but the processing of the communication unit 108, the position and orientation acquisition unit 109, and the map generation unit 110 is different from that of the first embodiment. The configuration of the controller 200 according to the second embodiment is the same as that in Fig. 1(B), but the processing of the communication unit 208 and the position and orientation acquisition unit 209 is different from that of the first embodiment. The processing that differs from that of the first embodiment will be described below.

[0079] 10 is a diagram for explaining an information processing system according to the third embodiment. The communication unit 208 of the controller 200 transmits an image captured by the imaging unit 204 to the HMD 100. The position and orientation acquisition unit 109 of the HMD 100 acquires information on the position and orientation of the controller 200 using the captured image received from the controller 200. The map generation unit 110 generates a controller map 130 based on the information on the position and orientation of the controller 200 and a key frame image that is a captured image captured at the position and orientation. The communication unit 108 transmits the generated controller map 130 to the controller 200.

[0080] The controller 200 acquires information about the position and orientation of the controller 200 using the image captured by the imaging unit 204 and the controller map 130 received from the HMD 100. The controller 200 can acquire information about the position and orientation of the controller 200 with high accuracy by using the controller map 130 generated by the HMD 100, which has a higher processing capacity than the controller 200. Note that the HMD 100 may generate the controller map 130 in non-real time using the image captured by the imaging unit 204 received from the controller 200, and transmit the controller map 130 to the controller 200.

[0081] Furthermore, the HMD 100 may generate a new controller map 130 by combining the map 120 for the HMD 100 generated in the first embodiment and the controller map 130. The HMD 100 can generate a new controller map 130 by, for example, extracting and combining keyframe information in which various indices indicating quality satisfy predetermined conditions from the map 120 and the controller map 130. Furthermore, the HMD 100 may integrate the map 120 and the controller map 130 by a map integration process using known SLAM.

[0082] Furthermore, the position and orientation acquisition unit 109 of the HMD 100 may acquire information on the position and orientation of the controller 200 by detecting the HMD 100 from the captured image received from the controller 200. Specifically, the HMD 100 acquires information on the position and orientation of the HMD 100 using the map 120, and acquires the relative position and orientation of the controller 200 from the detection result of the HMD 100 in the captured image received from the controller 200. The HMD 100 can acquire information on the position and orientation of the controller 200 based on the position and orientation of the HMD 100 and the relative position and orientation of the controller 200.

[0083] According to the above-described third embodiment, the controller 200 acquires information on the position and orientation of the controller 200 using the map for controller 130 generated by the HMD 100, and therefore the position and orientation of the controller 200 can be estimated with high accuracy.

[0084] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0085] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An information processing system having an information processing device and a terminal device, for estimating a position and orientation of the terminal device, The information processing device includes: A first imaging means; a first acquisition means for acquiring information on a position and orientation of the information processing device using an image captured by the first imaging means; a generation means for generating map information based on information on a position and orientation of the information processing device and a key frame image which is an image captured at the position and orientation; an extraction means for extracting a portion of the map information; a communication means for transmitting extraction map information extracted by the extraction means to the terminal device; having The terminal device A second imaging means; a second acquisition means for acquiring information on a position and orientation of the terminal device using an image captured by the second imaging means and the extraction map information transmitted from the information processing device; An information processing system comprising: (Configuration 2) The second acquisition means acquires information on the position and orientation of the terminal device using an image captured by a second imaging means, and acquires information on the position and orientation of the terminal device at a predetermined timing using the image captured by the second imaging means and the extraction map information transmitted from the information processing device. 2. The information processing system according to configuration 1. (Configuration 3) The predetermined timing is a regular or fixed timing. 3. The information processing system according to configuration 2. (Configuration 4) The map information includes a plurality of pieces of key frame information that associate information on the position and orientation of the information processing device with the key frame images captured at the corresponding positions and orientations. 4. The information processing system according to any one of configurations 1 to 3. (Configuration 5) The extraction means extracts the key frame information created in the peripheral area of ​​the terminal device from the map information. 5. The information processing system according to configuration 4. (Configuration 6) The size of the surrounding area is determined based on at least one of the processing capability of the terminal device, the storage capacity of a storage unit of the terminal device, and the number of pieces of key frame information within the surrounding area. 6. The information processing system according to configuration 5. (Configuration 7) The extraction means extracts representative key frame information for each of a plurality of divided regions obtained by dividing an area in which the positions at which the key frame images are captured are distributed. 5. The information processing system according to configuration 4. (Configuration 8) The extraction means changes the number of the divided regions or the size of the divided regions based on the size of an area in which the positions at which the key frame images are captured are distributed. 8. The information processing system according to configuration 7. (Configuration 9) The communication means transmits the extraction map information to the terminal device when the number of the divided regions or the size of the divided regions is changed. 9. The information processing system according to configuration 7 or 8. (Configuration 10) The extraction means extracts the key frame information based on a variance of positions of feature points captured in the key frame images or a reprojection error of the feature points. 5. The information processing system according to configuration 4. (Configuration 11) The extraction means extracts the key frame information from the map information based on a proportion of blur in the key frame image, a number of moving objects in the key frame image, or a generation time of the key frame information. 11. The information processing system according to any one of configurations 4 to 10. (Configuration 12) The extraction means extracts the key frame information from the map information based on an imaging direction of the terminal device. 12. The information processing system according to any one of configurations 4 to 11. (Configuration 13) The terminal device further includes an inertial measurement means, The second acquisition means acquires information on the position and orientation of the terminal device by the inertial measurement means, and acquires information on the position and orientation of the terminal device at a predetermined timing using an image captured by the second imaging means and the extraction map information transmitted from the information processing device. 2. The information processing system according to configuration 1. (Configuration 14) The extraction means is based on at least one of the processing capability and the storage capacity of the terminal device. and changing the extraction amount or extraction ratio to be extracted from the map information. 14. The information processing system according to any one of configurations 1 to 13. (Configuration 15) The extraction means changes an amount or a ratio of extraction to be extracted from the map information based on an acquisition status of information on a moving speed of the terminal device or a position and orientation of the terminal device. 15. The information processing system according to any one of configurations 1 to 14. (Configuration 16) The extraction means generates the extraction map information for each of the plurality of terminal devices, The communication means transmits the extraction map information corresponding to each of the plurality of terminal devices. 16. The information processing system according to any one of configurations 1 to 15. (Configuration 17) An information processing system having an information processing device and a terminal device, for estimating a position and orientation of the terminal device, The terminal device A second imaging means; a transmitting means for transmitting an image captured by the second imaging means to the information processing device; Have The information processing device includes: A first imaging means; a first acquisition means for acquiring information on a position and orientation of the information processing device using an image captured by the first imaging means; a generation means for generating map information based on information on a position and orientation of the information processing device and a key frame image which is an image captured at the position and orientation; an acquisition means for acquiring information on a position and orientation of the terminal device based on an image captured by the second imaging means and the map information; A communication means for transmitting information on the position and orientation of the terminal device to the terminal device; An information processing system comprising: (Configuration 18) An information processing system having an information processing device and a terminal device, for estimating a position and orientation of the terminal device, The information processing device includes: a first acquisition means for acquiring information on a position and orientation of the terminal device using a captured image received from the terminal device; a generation means for generating map information based on information on the position and orientation of the terminal device and a key frame image that is an image captured at the position and orientation; A communication means for transmitting the map information to the terminal device; having The terminal device A second imaging means; a second acquisition means for acquiring information on a position and orientation of the terminal device using an image captured by the second imaging means and the map information transmitted from the information processing device; An information processing system comprising: (Configuration 19) the first acquisition means acquires information about a position and orientation of the information processing device using an image captured by the information processing device; The generating means generates first map information based on information on the position and orientation of the terminal device and a key frame image which is a captured image captured at the position and orientation, and second map information based on information on the position and orientation of the information processing device and a key frame image which is a captured image captured at the position and orientation. The two pieces of map information are combined to generate the map information. 20. The information processing system according to configuration 18, (Configuration 20) The first acquisition means acquires information about a position and orientation of the terminal device by detecting the information processing device from an image captured by the terminal device. 20. The information processing system according to configuration 18 or 19. (Configuration 21) An information processing device that generates map information for a terminal device to estimate a position and orientation, Imaging means and an acquisition means for acquiring information on a position and orientation of the information processing device using an image captured by the imaging means; a generation means for generating the map information based on information on a position and orientation of the information processing device and a key frame image which is an image captured at the position and orientation; an extraction means for extracting a portion of the map information and generating extracted map information used to acquire attitude information of the terminal device; a communication means for transmitting the extraction map information to the terminal device; 13. An information processing device comprising: (Configuration 22) A terminal device that estimates a position and orientation using map information received from an information processing device, An imaging means; a communication means for receiving, from the information processing device, extracted map information obtained by extracting a part of map information generated based on information on a position and orientation of the information processing device and a key frame image which is an image captured at the position and orientation; an acquisition means for acquiring information on the position and orientation of the terminal device using an image captured by the imaging means and the extraction map information transmitted from the information processing device; A terminal device comprising: (method) A control method for an information processing system having an information processing device and a terminal device, the control method comprising the steps of: a first acquisition step of acquiring information on a position and orientation of the information processing device using an image captured by a first imaging means of the information processing device; a generating step of generating map information based on information on the position and orientation of the information processing device and a key frame image which is an image captured at the position and orientation; an extraction step of extracting a portion of the map information; a communication step of transmitting extraction map information extracted in the extraction step to the terminal device; a second acquisition step of acquiring information on the position and orientation of the terminal device using an image captured by a second imaging means of the terminal device and the extraction map information transmitted from the information processing device; A control method comprising the steps of: (program) 24. A program for causing a computer to execute each step of the control method according to claim 23. [Explanation of symbols]

[0086] 100: HMD (information processing device), 101: CPU, 104: imaging unit, 108: communication unit, 109: position and orientation acquisition unit, 110: map generation unit, 111: map extraction unit, 120: map, 121: extracted map 200: controller (terminal device), 204: imaging unit, 209: position and orientation acquisition unit

Claims

1. An information processing system having a head-mounted display and a controller, for estimating the position and orientation of the controller, comprising: The head-mounted display includes: first imaging means; first acquisition means for acquiring information on the position and orientation of the head-mounted display using an imaging image captured by the first imaging means; generation means for generating map information based on the information on the position and orientation of the head-mounted display and a keyframe image which is an imaging image captured in the position and orientation; extraction means for extracting a part of the map information; communication means for transmitting the extracted map information extracted by the extraction means to the controller and having; The controller includes: second imaging means; second acquisition means for acquiring information on the position and orientation of the controller using the imaging image captured by the second imaging means and the extracted map information transmitted from the head-mounted display characterized in that it has an information processing system.

2. The second acquisition means acquires information on the position and orientation of the controller using the imaging image captured by the second imaging means, and at a predetermined timing, acquires information on the position and orientation of the controller using the imaging image by the second imaging means and the extracted map information transmitted from the head-mounted display The information processing system according to claim 1, characterized in that.

3. The predetermined timing is a periodic or regular timing The information processing system according to claim 2, characterized in that.

4. The map information includes a plurality of keyframe information in which information on the position and orientation of the head-mounted display and the keyframe image captured in the position and orientation are associated with each other The information processing system according to any one of claims 1 to 3, characterized in that.

5. The extraction means extracts the keyframe information created in the peripheral area of the controller from the map information The information processing system according to claim 4, characterized in that.

6. The size of the peripheral area is determined based on at least any one of the processing capacity of the controller, the storage capacity of the storage unit of the controller, and the number of the keyframe information in the peripheral area The information processing system according to claim 5, characterized in that.

7. The extraction means extracts the representative key frame information for each divided region obtained by dividing the region where the positions of the captured key frame images are distributed into a plurality of regions. The information processing system according to claim 4, characterized in that.

8. The extraction means changes the number or size of the divided regions based on the size of the region where the positions of the captured key frame images are distributed. The information processing system according to claim 7, characterized in that.

9. When the number or size of the divided regions is changed, the communication means transmits the extraction map information to the controller. The information processing system according to claim 7, characterized in that.

10. The extraction means extracts the key frame information based on the variance of the positions of the feature points captured in the key frame image or the reprojection error of the feature points. The information processing system according to claim 4, characterized in that.

11. The extraction means extracts the key frame information from the map information based on the blur ratio in the key frame image, the number of moving objects in the key frame image, or the generation timing of the key frame information. The information processing system according to claim 4, characterized in that.

12. The extraction means extracts the key frame information from the map information based on the imaging direction of the controller. The information processing system according to claim 4, characterized in that.

13. The controller further includes inertial measurement means. The second acquisition means acquires information on the position and orientation of the controller by the inertial measurement means, and acquires information on the position and orientation of the controller using the captured image by the second imaging means and the extraction map information transmitted from the head-mounted display at a predetermined timing. The information processing system according to claim 1, characterized in that.

14. The extraction means changes the extraction amount or extraction ratio extracted from the map information based on at least one of the processing capacity and storage capacity of the controller. The information processing system according to any one of claims 1 to 3, characterized in that.

15. The extraction means changes the extraction amount or extraction ratio extracted from the map information based on the acquisition status of the movement speed of the controller or the information on the position and orientation of the controller. The information processing system according to any one of claims 1 to 3.

16. The extraction means generates the extraction map information for each of the plurality of controllers. The communication means transmits the extraction map information corresponding to each of the plurality of controllers to each of them. The information processing system according to any one of claims 1 to 3.

17. A head-mounted display that generates map information for estimating the position and orientation of a controller, imaging means; acquisition means for acquiring information on the position and orientation of the head-mounted display using the captured image by the imaging means; generation means for generating the map information based on the information on the position and orientation of the head-mounted display and a keyframe image which is a captured image captured in the position and orientation; extraction means for extracting a part of the map information and generating extraction map information used for acquiring the orientation information of the controller; communication means for transmitting the extraction map information to the controller A head-mounted display characterized by comprising.

18. A controller that estimates the position and orientation using the map information received from the head-mounted display, imaging means; communication means for receiving, from the head-mounted display, extraction map information obtained by extracting a part of the map information generated based on the information on the position and orientation of the head-mounted display and a keyframe image which is a captured image captured in the position and orientation; acquisition means for acquiring information on the position and orientation of the controller using the captured image by the imaging means and the extraction map information transmitted from the head-mounted display A controller characterized by comprising.

19. A control method for an information processing system having a head-mounted display and a controller, for estimating the position and orientation of the controller, a first acquisition step of acquiring information on the position and orientation of the head-mounted display using a captured image captured by a first imaging means of the head-mounted display; A generation step of generating map information based on information on the position and orientation of the head-mounted display and a keyframe image which is an imaging image captured in the position and orientation; An extraction step of extracting a part of the map information; A communication step of transmitting the extracted map information extracted in the extraction step to the controller; A second acquisition step of acquiring information on the position and orientation of the controller using the imaging image captured by the second imaging means of the controller and the extracted map information transmitted from the head-mounted display; A control method characterized by comprising the above.

Images