Information processing device and information processing method

Abstract

The present invention provides an information processing device that can reliably acquire position and orientation even if there is an abnormality in a portion of the captured image used to acquire position and orientation. [Solution] The information processing device includes receiving means for receiving an image from the device, and acquiring means for acquiring the position or orientation of the device based on a second region of the image excluding a first region containing data errors or missing data. The acquiring means identifies the second region using the position information of the first region, and acquires the position or orientation of the device based on the identified second region.

Description

Technical Field

[0001] It relates to an information processing apparatus and an information processing method.

Background Art

[0002] In recent years, as a technology for seamlessly and real-time integrating the real world and the virtual world, a composite reality technology, so-called MR (Mixed Reality) technology, is known. For example, a user can experience a composite reality space by using a video see-through type HMD (Head Mounted Display). The HMD captures a subject that substantially coincides with the subject observed from the position of the user's pupil by a video camera or the like, and displays an image in which CG (Computer Graphics) is superimposed on the captured image.

[0003] The video see-through type HMD captures a subject by a charge-coupled device such as a CCD to obtain digital image data of the subject, and displays an MR image (composite reality image) in which a CG image is superimposed on a user (wearer) via a display device such as a liquid crystal or an organic EL. The process of superimposing a CG image on the captured image may be executed by an external device having higher processing performance than the HMD. In this case, the HMD transmits the captured image captured by the HMD to the external device. The external device acquires the position and orientation of the HMD using the captured image received from the HMD. The external device generates a composite reality image in which a CG image is superimposed on the captured image based on the acquired position and orientation of the HMD, and transmits the generated composite reality image to the HMD. The HMD displays the composite reality image received from the external device. Thereby, a user wearing the HMD can experience an MR space.

[0004] Patent Document 1 proposes a method of not using an image in which an abnormality is detected for learning in an image processing apparatus that detects an object shown in a moving image while performing dictionary learning.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2016-001397 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] If an abnormality is found in a portion of the captured image used to acquire the HMD's position and orientation, and the captured image is not used for position and orientation acquisition, and if abnormalities occur repeatedly, it may become difficult for the external device to stably acquire the HMD's position and orientation.

[0007] Therefore, the present invention aims to provide an information processing device that can stably acquire position and orientation even if there is an abnormality in a part of the captured image used to acquire position and orientation. [Means for solving the problem]

[0008] The information processing apparatus according to the present invention is characterized by comprising: receiving means for receiving an image from a device; and acquiring means for acquiring the position or orientation of the device based on a second region of the image excluding a first region containing data errors or missing data. [Effects of the Invention]

[0009] According to the present invention, even if there is an abnormality in a part of the captured image used to acquire position and orientation, it is possible to stably acquire position and orientation. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram illustrating the configuration of the system according to the present invention. [Figure 2] This is a block diagram of the HMD and information processing device according to Embodiment 1. [Figure 3] This figure illustrates an image captured according to Embodiment 1. [Figure 4] This is a flowchart of the position and orientation acquisition process according to Embodiment 1. [Figure 5] This is another example of a block diagram of the HMD and information processing device according to Embodiment 1. [Figure 6] This is a block diagram of the HMD and information processing device according to Embodiment 2. [Figure 7] This is a diagram illustrating the image processing according to Embodiment 2. [Figure 8] This is a flowchart of the position and orientation acquisition process according to Embodiment 2. [Figure 9] This is a block diagram of the HMD and information processing device according to Embodiment 3. [Figure 10] This is a diagram illustrating the image processing according to Embodiment 3. [Figure 11] This is a flowchart of the position and orientation acquisition process according to Embodiment 3. [Figure 12] This figure illustrates time-series images captured by multiple imaging units. [Figure 13] This is a flowchart of the position and orientation acquisition process according to Embodiment 4. [Figure 14] This is a flowchart for acquiring position and orientation using images from a single imaging unit. [Modes for carrying out the invention]

[0011] <Embodiment 1> Embodiments of the present invention will be described below with reference to the drawings. Figure 1 is a diagram illustrating the configuration of a system according to the present invention. The system shown in Figure 1 is a head-mounted display device (Head Includes a Mounted Display (HMD) 100 and an information processing device 200. The information processing device 200 is connected to the display unit 211 and the operation unit 212. The present invention applies when the information processing device 200 acquires the position and orientation of the HMD 100 in order to generate an image to be displayed on the HMD 100. The system shown in Figure 1 below will be described assuming it is an MR system, but it may also be a VR (Virtual Reality) system or an AR (Augmented Reality) system.

[0012] The HMD 100 is worn on the user's head. The HMD 100 includes a control unit that controls components such as a display unit and a communication unit included in the HMD 100. The HMD 100 receives an image generated by the information processing device 200 via the communication unit. The HMD 100 displays the received image on the display unit. The display unit of the HMD 100 includes an optical system disposed in front of each of the user's left and right eyes.

[0013] The HMD 100 can communicate with the information processing device 只要你需要,我可以继续为你翻译剩余内容。你是在处理专利相关的翻译工作吗?如果还有其他文本需要翻译,随时告诉我。200 via a small-scale network such as a WLAN (Wireless Local Area Network) or a WPAN (Wireless Personal Area Network). Communication between the HMD 100 and the information processing device 200 is not limited to a wireless communication method, and a wired communication method may also be used.

[0014] The information processing device 200 includes a control unit that controls components such as a playback unit, a storage unit, and a communication unit included in the information processing device 200. The control unit can implement the processing of each part of the information processing device 200 by expanding and executing a program stored in the storage unit in a memory. The playback unit generates an image to be played back by the HMD 100. The storage unit stores the image generated by the playback unit. The information processing device 200 communicates with the HMD 100 via the communication unit. For example, the information processing device 200 transmits an image stored in the storage unit to the HMD 100 via the communication unit.

[0015] The operation unit 212 connected to the information processing device 200 is an input device such as a keyboard. The user can input data, instructions, etc. using the operation unit 212. The display unit 211 displays the data input by the user or the result of processing based on instructions from the user.

[0016] FIG. 2 is an example of a block diagram of the HMD 100 and the information processing device 200 according to Embodiment 1. The HMD 100 includes a communication unit 101, a left imaging unit 102L, a right imaging unit 102R, a left display unit 103L, and a right display unit 103R.

[0017] The communication unit 101 is a communication interface for communicating with external devices such as the information processing device 200. The communication unit 101 transmits and receives various data, such as images, and various control signals, to and from the information processing device 200, for example. The left imaging unit 102L and the right imaging unit 102R capture images of the outside world from approximately the same position as the user's eyes. The left imaging unit 102L and the right imaging unit 102R are collectively referred to as the imaging unit 102.

[0018] The left display unit 103L and the right display unit 103R display images (stereo images) to the user wearing the HMD100. The left display unit 103L displays an image for the user's left eye, and the right display unit 103R displays an image for the user's right eye. The left display unit 103L and the right display unit 103R are collectively referred to as display unit 103.

[0019] The information processing device 200 is an external device different from the HMD 100, such as a personal computer (PC), workstation (WS), or cloud server via a public network. The information processing device 200 includes a communication unit 201, a detection unit 202, a region determination unit 203, an HMD position and orientation acquisition unit 204, a CG rendering unit 205, and a content DB 206.

[0020] The communication unit 201 is a communication interface for communicating with external devices such as the HMD 100. The communication unit 201 transmits and receives various data, such as images, and various control signals, to and from the HMD 100, for example.

[0021] The detection unit 202 determines the location of errors from the captured image received from the HMD 100. An error location is a location in the received captured image data where a data error such as a packet error or data loss such as a packet loss has occurred. The region determination unit 203 determines the region on the captured image corresponding to the location where the data error or data loss occurred.

[0022] The HMD position and orientation acquisition unit 204 acquires the position and orientation of the HMD 100, that is, the position and orientation of the HMD 100. The HMD position and orientation acquisition unit 204 may also acquire the position or orientation of the HMD 100. The content DB 206 stores information about the CG content of the virtual image. The CG drawing unit 205 draws the CG image based on the CG content information stored in the content DB 206 and the position and orientation of the HMD 100.

[0023] The information processing device 200 is used for the purpose of generating a composite image by superimposing a CG image onto the real-world image received from the HMD 100, and for the purpose of acquiring the position and orientation of the HMD 100. In the following description, the image used for generating the composite image and the image used for acquiring the position and orientation of the HMD 100 are assumed to be images captured by the same imaging unit 102, but are not limited to this. The image used for generating the composite image and the image used for acquiring the position and orientation of the HMD 100 may be images captured by different imaging devices.

[0024] The imaging unit 102 of the HMD100 captures images of the outside world. The HMD100 transmits the captured images to the information processing device 200 via the communication unit 101. Since communication between the HMD100 and the information processing device 200 is wireless, packet errors and packet loss are likely to occur when transmitting captured images. If a packet error or packet loss occurs, retransmitting the captured image will cause a delay. In systems such as MR and VR, if a delay occurs, the user may feel uncomfortable or lose their sense of immersion. Therefore, the HMD1 Even if a packet error or packet loss occurs in the image transmitted to the information processing device 200, 00 will not retransmit the image.

[0025] The information processing device 200 inputs the captured image received from the HMD 100 via the communication unit 201 to the detection unit 202. The detection unit 202 detects packet errors and packet losses that occurred during wireless communication. The detection unit 202 outputs information about the locations (packets) where packet errors or packet losses occurred in the captured image, as well as the captured image itself, to the region determination unit 203.

[0026] The region determination unit 203 determines the region (first region) in the captured image that contains a packet error or packet loss, based on the information of the packet that has resulted in a packet error or packet loss.

[0027] The HMD position and orientation acquisition unit 204 acquires the position and orientation of the HMD 100 based on the region (second region) obtained by excluding the region containing packet errors or packet losses determined by the region determination unit 203 in the captured image.

[0028] The CG drawing unit 205 draws CG for the left eye and CG for the right eye based on the position and orientation information of the HMD 100 acquired by the HMD position and orientation acquisition unit 204. The CG drawing unit 205 generates a composite image by superimposing the drawn CG onto the captured image received from the HMD 100. The information processing device 200 transmits the composite image generated by the CG drawing unit 205 to the HMD 100 via the communication unit 201. The HMD 100 displays the received composite image on the display unit 103.

[0029] By wearing the HMD100, the user can view a composite image of the computer graphics (CG) drawn by the information processing device 200 and the captured image captured by the imaging unit 102, according to the user's position and orientation. The information processing device 200 transmits the composite image generated by the CG drawing unit 205 superimposing the CG onto the captured image to the HMD100, but is not limited to this. The information processing device 200 may transmit the CG drawn by the CG drawing unit 205 to the HMD100, and the HMD100 may superimpose the CG onto the captured image and display it on the display unit 103.

[0030] Figure 3 is a diagram illustrating an image captured according to Embodiment 1. The information processing device 200 receives the image captured by the imaging unit 102 of the HMD 100 from the HMD 100 via wireless communication. Figure 3(A) shows the image captured by the information processing device 200, indicating a state where there were no packet errors or packet losses. The image in Figure 3(A) is a normal image captured without any packet errors or packet losses occurring during wireless communication.

[0031] Figures 3(B) and 3(C) show images captured when packet errors or packet loss occur in wireless communication. Regions 321-323 in Figure 3(B) and regions 331-334 in Figure 3(C) are the regions where packet errors or packet loss occurred. The area of ​​region 331-334 in Figure 3(C) is larger than the area of ​​region 321-323 in Figure 3(B).

[0032] In areas where packet loss occurs, no image data is displayed because the image data was not received due to the data loss. On the other hand, in areas where a packet error occurs, the correct image is not displayed due to errors in the image data.

[0033] One known method for calculating the position and orientation of devices such as the HMD100 using captured images is called SLAM (Simultainius Localization and Mapping). SLAM is a technique that detects feature points from captured images and uses these detected feature points to estimate the device's own position and orientation.

[0034] A common method for detecting feature points involves detecting high-frequency components in an image and then detecting angles from these high-frequency components. When detecting feature points, as shown in Figure 3(B), if there are areas in the captured image where packet errors or packet loss have occurred, incorrect feature points may be detected within those areas. In SLAM, the position and orientation of the HMD 100 are estimated using the feature points detected in the captured image. Therefore, if incorrect feature points are detected, the position and orientation of the HMD 100 may not be estimated correctly. In other words, if feature points are not correctly detected from the captured image, the position and orientation of the HMD 100 acquired by the HMD position and orientation acquisition unit 204 will be unstable.

[0035] In Embodiment 1, the information processing device 200 does not use areas in the captured image that contain packet errors or packet losses to acquire the position and orientation of the HMD 100. When transmitting the captured image wirelessly, the information processing device 200 divides the captured image into packets and transmits them. Since the packet size and transmission order are known, the information processing device 200 can acquire information such as which packet received had an error and which packets were not received. Using the information of the packets that have resulted in packet errors or packet losses, the information processing device 200 can determine which areas of the captured image contain packet errors or packet losses.

[0036] The detection unit 202 in Figure 2 detects packet errors or lost packets in the captured image received wirelessly from the HMD 100. The region determination unit 203 determines the region in the captured image that contains packet errors / packet losses based on the packet error / packet loss information detected by the detection unit 202. The region determination unit 203 outputs the information of the region containing packet errors / packet losses to the HMD position and attitude acquisition unit 204. The HMD position and attitude acquisition unit 204 acquires the position and attitude of the HMD 100 from the captured image excluding the region determined by the region determination unit 203.

[0037] Figure 4 is a flowchart illustrating the position and orientation acquisition process according to Embodiment 1. The process shown in Figure 4 is realized, for example, by the processor of the information processing device 200 loading a program for realizing the functions of each part of the information processing device 200 shown in Figure 2 into memory and executing it.

[0038] In step S401, the communication unit 201 receives an image from the HMD 100 to acquire the position and orientation of the HMD 100. In step S402, the detection unit 202 determines whether or not it detected a packet error or packet loss when receiving the image. If a packet error or packet loss is detected, the process proceeds to step S403. If no packet error or packet loss is detected, the process proceeds to step S406.

[0039] In step S403, the region determination unit 203 determines the region in the captured image that contains a packet error or packet loss, based on the information of the packet that has resulted in a packet error or packet loss. The region in the captured image that contains a packet error or packet loss (the first region) is also referred to as the excluded region in the following description.

[0040] In step S404, the region determination unit 203 determines whether the area of ​​the excluded region determined in step S403 is smaller than a predetermined threshold. The predetermined threshold is set in advance, for example, based on the area of ​​the captured image. For example, the predetermined threshold can be the area of ​​the captured image multiplied by a predetermined coefficient (for example, 0.2). If the area of ​​the excluded region is smaller than the predetermined threshold, the process proceeds to step S405. If the area of ​​the excluded region is greater than or equal to the predetermined threshold, the process shown in Figure 4 ends.

[0041] In step S405, the region determination unit 203 notifies the HMD position and orientation acquisition unit 204 of the location information of the excluded region. In step S406, the HMD position and orientation acquisition unit 204 uses the location information of the excluded region to identify the region (second region) remaining after removing the excluded region from the captured image, and calculates the position and orientation of the HMD 100 based on the identified region.

[0042] In step S404, the information processing device 200 uses the captured image to obtain the position and orientation of the HMD 100 if the area of ​​the excluded region in the captured image is smaller than a predetermined threshold. On the other hand, if the area of ​​the excluded region in the captured image is greater than or equal to a predetermined threshold, the information processing device 200 does not use the captured image to obtain the position and orientation of the HMD 100. For example, as shown in Figure 3(C), if the area of ​​the region where a packet error or packet loss occurred is greater than or equal to a predetermined threshold, sufficient feature points for obtaining the position and orientation cannot be detected, making it difficult to estimate a stable position and orientation.

[0043] According to the above embodiment 1, the information processing device 200 can reduce the impact on the acquisition process of the position and orientation of the HMD 100 even if a packet error or packet loss occurs when receiving an image via wireless communication.

[0044] The images transmitted and received between the HMD100 and the information processing device 200 may be compressed before transmission and decompressed after transmission. Figure 5 is an example of a block diagram of the HMD100 and the information processing device 200 when images are compressed and decompressed. In Figure 5, the HMD100 has a compression unit 501 and a decompression unit 502 in addition to the configuration in Figure 2. The information processing device 200 has a decompression unit 503 and a compression unit 504 in addition to the configuration in Figure 2.

[0045] The compression and decompression of the captured image transmitted from the HMD 100 to the information processing device 200 will be described. The compression unit 501 of the HMD 100 compresses the captured image captured by the imaging unit 102. The communication unit 101 transmits the compressed captured image to the information processing device 200. The communication unit 201 of the information processing device 200 receives the captured image in its compressed state. The decompression unit 503 decompresses the compressed captured image. The HMD position and orientation acquisition unit 204 acquires the position and orientation of the HMD 100 based on the region (second region) obtained by excluding the excluded region from the decompressed captured image.

[0046] The compression and decompression of the display image transmitted from the information processing device 200 to the HMD 100 will be described. The display image is, for example, a composite image of CG and an captured image, and is displayed on the display unit 103 of the HMD 100. The compression unit 504 of the information processing device 200 compresses the display image generated by the CG drawing unit 205. The communication unit 201 transmits the compressed display image to the HMD 100. The communication unit 101 of the HMD 100 receives the display image in its compressed state. The decompression unit 502 decompresses the compressed display image. The display unit 103 displays the decompressed display image.

[0047] Furthermore, the HMD position and orientation acquisition unit 204 acquires the position and orientation of the HMD 100 based on the region obtained by excluding the excluded region from the captured image, but the excluded region may further include the region surrounding the region where a packet error or packet loss occurred.

[0048] <Embodiment 2> In Embodiment 1, the information processing device 200 uses location information of areas in the received captured image that contain packet errors or packet losses (first area, excluded area) to identify the area in the captured image excluding the excluded area (second area). Based on the area identified in the captured image, the information processing device 200 acquires the position and orientation of the HMD 100. In contrast, in Embodiment 2, before acquiring the position and orientation of the HMD 100, the information processing device 200 ensures that the excluded area of ​​the captured image is not used to acquire the position and orientation of the HMD 100. To achieve this, image processing is performed on the excluded area. Specifically, the information processing device 200 replaces the excluded area with a monochrome image.

[0049] Figure 6 is an example of a block diagram of the HMD 100 and information processing device 200 according to Embodiment 2. In Figure 6, the information processing device 200 has an image replacement unit 601 in addition to the configuration in Figure 2. The image replacement unit 601 performs image processing on the excluded region determined by the region determination unit 203 so that it is not used to obtain the position and orientation of the HMD 100. That is, the image replacement unit 601 performs image processing on the excluded region so that the position and orientation of the HMD 100 cannot be obtained from the excluded region. For example, the image replacement unit 601 may replace the image of the excluded region with an image in which no feature points are detected.

[0050] The image processing according to Embodiment 2 will be described with reference to Figures 7(A) to 7(C). Figure 7(A) shows an example of an image captured when a packet error or packet loss occurs. Regions 321 to 323 are regions where a packet error or packet loss occurred.

[0051] The image replacement unit 601 performs image processing on regions 321 to 323 where packet errors or packet loss have occurred. As an example of image processing, the image replacement unit 601 replaces regions 321 to 323 with black images 701 to 703, as shown in Figure 7(B). The image replacement unit 601 is not limited to black images, and may replace regions 321 to 323 with a single-color image of any color. Alternatively, as shown in Figure 7(C), the image replacement unit 601 may apply a gradient 711 to 713 to the region including the boundary of the black images 701 to 703 in Figure 7(B).

[0052] As shown in Figure 7(C), the advantages of applying a gradient 711-713 are explained below. When detecting feature points for self-localization estimation by SLAM, high-frequency components of the image are detected. In the image processing exemplified in Figure 7(B), the boundary between the black images 701-703 and the real-world image becomes a high-frequency component and may be detected as an incorrect feature point. As shown in Figure 7(C), the image replacement unit 601 can prevent the detection of high-frequency components by applying a gradient to the boundary portion (the region including the boundary) of the black images 701-703. The image replacement unit 601 should apply a gradient to the boundary portion of the black images 701-703 so that the inner part becomes closer to black.

[0053] Figure 8 is a flowchart illustrating the position and orientation acquisition process according to Embodiment 2. The position and orientation acquisition process according to Embodiment 2 includes the process of step S801 in place of the process of step S405 in the position and orientation acquisition process according to Embodiment 1 in Figure 4. Steps that are the same as those in Embodiment 1 are denoted by the same reference numerals and their descriptions are omitted.

[0054] In step S801, the image replacement unit 601 performs image processing on the excluded region determined by the region determination unit 203 so that it is not used to acquire the position and orientation of the HMD 100. Specifically, the image replacement unit 601 replaces the excluded region with a monochrome image. The image replacement unit 601 may also apply a gradient to the boundary portion.

[0055] According to Embodiment 2 described above, the HMD position and orientation acquisition unit 204 can acquire the position and orientation of the HMD 100 based on an image captured by image processing the excluded area, rather than directly using the position information of the excluded area. Similar to Embodiment 1, the information processing device 200 can reduce the impact on the acquisition process of the position and orientation of the HMD 100 even if a packet error or packet loss occurs when receiving the captured image via wireless communication.

[0056] <Embodiment 3> In Embodiment 2, the information processing device 200 uses the region (exclusion region) in the captured image that contains packet errors or packet losses to obtain the position and orientation of the HMD 100. To prevent this, the excluded area is replaced with a monochrome image. In contrast, in Embodiment 3, the information processing device 200 performs image processing to remove high-frequency components from the excluded area.

[0057] Figure 9 is an example of a block diagram of the HMD 100 and information processing device 200 according to Embodiment 3. In Figure 9, the information processing device 200 has an image processing unit 901 in addition to the configuration in Figure 2. The image processing unit 901 performs image processing to prevent the exclusion region determined by the region determination unit 203 from being used to acquire the position and orientation of the HMD 100. For example, the image processing unit 901 performs image processing to remove high-frequency components from the exclusion region. The image processing unit 901 outputs the processed image to the CG rendering unit 205. The CG rendering unit 205 combines the image processed by the image processing unit 901 with a CG image.

[0058] Referring to Figures 10(A) and 10(B), the image processing according to Embodiment 3 will be described. Figure 10(A) shows an example of an image captured when a packet error or packet loss occurs. Regions 321 to 323 are regions where a packet error or packet loss occurred.

[0059] The image processing unit 901 performs image processing on regions 321 to 323 where packet errors or packet loss have occurred. As an example of image processing, the image processing unit 901 removes high-frequency components in regions 321 to 323, as shown in Figure 10(B). Regions 1001 to 1003 in Figure 10(B) show the regions from which high-frequency components have been removed in regions 321 to 323.

[0060] When detecting feature points for self-localization using SLAM, high-frequency components are detected in the image. The image processing unit 901 can prevent the detection of high-frequency components by removing them in areas where packet errors or packet losses have occurred. The image processing unit 901 may also perform the process of removing high-frequency components not only in areas where packet errors or packet losses have occurred, but also in the boundary areas.

[0061] Figure 11 is a flowchart illustrating the position and orientation acquisition process according to Embodiment 3. The position and orientation acquisition process according to Embodiment 3 includes the process of step S1101 in place of the process of step S405 in the position and orientation acquisition process according to Embodiment 1 in Figure 4. Steps that are the same as those in Embodiment 1 are denoted by the same reference numerals and their descriptions are omitted.

[0062] In step S1101, the image processing unit 901 performs image processing on the exclusion region determined by the region determination unit 203 so that it is not used to acquire the position and orientation of the HMD 100. Specifically, the image processing unit 901 removes high-frequency components from the exclusion region. The image processing unit 901 may also perform image processing to remove high-frequency components from the boundary portion of the exclusion region.

[0063] According to Embodiment 3 described above, the HMD position and orientation acquisition unit 204 can acquire the position and orientation of the HMD 100 using an image captured by image processing the excluded area, rather than directly using position information of the excluded area, similar to Embodiment 2. The information processing device 200 can reduce the impact on the acquisition process of the position and orientation of the HMD 100 even if a packet error or packet loss occurs when receiving the captured image via wireless communication, similar to Embodiment 1.

[0064] <Embodiment 4> In Embodiment 4, the information processing device 200 receives multiple images captured by multiple imaging devices (imaging units) and, depending on the situation in which a packet error or packet loss occurs, uses at least one of the images to obtain the position and orientation of the HMD 100.

[0065] A block diagram of the HMD100 and information processing device 200 according to Embodiment 4 is shown in Figure 2. Since it is the same as the block diagram of Embodiment 1, the explanation will be omitted. The configuration of the HMD100 and information processing device 200 in Embodiment 4 may be the same as that of Embodiment 2 shown in Figure 6 when Embodiment 2 is applied to Embodiment 4, and may be the same as that of Embodiment 3 shown in Figure 9 when Embodiment 3 is applied to Embodiment 4.

[0066] Figure 12 illustrates a time-series image (frame) captured by multiple imaging units. The image shown in Figure 12 is an image received by the information processing device 200 from the HMD 100, and represents the left image captured by the left imaging unit 102L and the right image captured by the right imaging unit 102R at times T=0 to 4. The left image captured at times T=1, 2, and 3 includes areas where packet errors or packet loss occurred. The right image was received normally.

[0067] If there are multiple imaging units that capture images to acquire the position and orientation of the HMD100, and the image from one imaging unit is degraded due to packet errors or packet loss, the information processing unit 200 may acquire the position and orientation of the HMD100 using the image from another imaging unit. In the example shown in Figure 12, if the area of ​​the excluded region in the left image captured by the left imaging unit 102L is greater than a predetermined threshold, and the area of ​​the excluded region in the right image captured by the right imaging unit 102R is less than a predetermined threshold, the information processing unit 200 will acquire the position and orientation using the right image. That is, if the area of ​​the excluded region in the left image is greater than a predetermined threshold at time T=1 to 3, the information processing unit 200 will acquire the position and orientation of the HMD100 using the right image.

[0068] Furthermore, if the image from one imaging unit is degraded due to packet errors or packet loss, the information processing device 200 may use the image from another imaging unit to acquire the position and orientation of the HMD 100. In the example shown in Figure 12, the information processing device 200 may determine at time T=3 that packet errors or packet loss have occurred consecutively in the left image captured from time T=1 to T=3, and then use the right image captured from time T=3 to acquire the position and orientation.

[0069] Furthermore, by using images captured by multiple imaging units, the information processing device 200 can align feature points based on their three-dimensional positions, thereby accurately acquiring the position and orientation of the HMD 100. In the example shown in Figure 12, at times T=0 and T=4, when both the left and right images have been successfully received, the information processing device 200 can acquire the position and orientation of the HMD 100 using the two images. The information processing device 200 may normally acquire the position and orientation using an image from one imaging unit, and may acquire the position and orientation of the HMD 100 using images from multiple imaging units once every few frames.

[0070] Figure 13 is a flowchart illustrating the position and orientation acquisition process according to Embodiment 4. Figure 13 shows an example of a process in which the position and orientation of the HMD 100 is normally acquired using the image from one imaging unit, and then acquired using the images from multiple imaging units once every few frames. The position and orientation acquisition process shown in Figure 13 includes the processes of steps S1301 to S1304 in place of the process of step S406 of the position and orientation acquisition process according to Embodiment 1 in Figure 4. Steps that are the same as those in Embodiment 1 are denoted by the same reference numerals and their descriptions are omitted.

[0071] In step S1301, the HMD position and orientation acquisition unit 204 determines whether it is the timing to acquire the position and orientation from multiple images captured by multiple imaging units (once every few frames). If it is the timing to acquire the position and orientation from multiple images, the process proceeds to step S1303. If it is not the timing to acquire the position and orientation from multiple images, the process proceeds to step 1302.

[0072] In step S1302, the HMD position and orientation acquisition unit 204 acquires the position and orientation of the HMD 100 using an image from one imaging unit. Details of the process in step S1302 will be described later with reference to Figure 14.

[0073] In step S1303, the HMD position and orientation acquisition unit 204 determines whether there are multiple images among the images captured by the multiple imaging units in which the area of ​​the excluded region is smaller than a predetermined threshold. The HMD position and orientation acquisition unit 204 only needs to determine whether the area of ​​the region where a packet error or packet loss occurred is smaller than a threshold in the images captured by at least two imaging units. If there are multiple images in which the area of ​​the excluded region is smaller than the predetermined threshold, the process proceeds to step S1304. If there are no multiple images in which the area of ​​the excluded region is smaller than the predetermined threshold, the process proceeds to step S1302 because there is a possibility that feature points may not be correctly detected from multiple images.

[0074] In step S1304, the HMD position and orientation acquisition unit 204 acquires the position and orientation of the HMD 100 using two or more captured images in which the area of ​​the excluded region is smaller than a predetermined threshold. The HMD position and orientation acquisition unit 204 acquires the position and orientation of the HMD 100 based on the region excluding each of the two or more captured images in which the area of ​​the excluded region is smaller than a predetermined threshold.

[0075] In step S404 of Figure 13, if the area of ​​the excluded region in the captured image is greater than or equal to a predetermined threshold, the information processing device 200 may proceed to step S1302 instead of terminating the process shown in Figure 13. In this case, if the area of ​​the excluded region in the image captured by another imaging unit is less than the predetermined threshold, the information processing device 200 can use the image captured by the other imaging unit to obtain the position and orientation of the HMD 100.

[0076] Figure 14 is a flowchart of the position and orientation acquisition process using images from one imaging unit. The process in Figure 14 illustrates the detailed process of step S1302 in Figure 13. In step S1401, the HMD position and orientation acquisition unit 204 determines whether frames containing areas with packet errors or packet losses (exclusion areas) are consecutive among the captured images (frames) of the left imaging unit 102L (first imaging device). The HMD position and orientation acquisition unit 204 may also determine whether frames in which the area of ​​the exclusion area is greater than or equal to a predetermined threshold are consecutive.

[0077] The HMD position and orientation acquisition unit 204 can determine, for example, that frames containing the exclusion region are consecutive if they continue for a predetermined number of frames (e.g., 3 frames). If the frames containing the exclusion region are consecutive, the process proceeds to step S1402. If the frames containing the exclusion region are not consecutive, the process proceeds to step S1404.

[0078] In step S1402, the HMD position and orientation acquisition unit 204 determines whether frames containing areas with packet errors or packet losses (exclusion areas) are consecutive among the captured images (frames) of the right imaging unit 102R (second imaging device). Similar to the left imaging unit 102L in step S1401, the HMD position and orientation acquisition unit 204 can determine whether frames containing exclusion areas are consecutive among the captured images of the right imaging unit 102R. If frames containing exclusion areas are consecutive, the process proceeds to step S1403. If frames containing exclusion areas are not consecutive, the process proceeds to step S1405.

[0079] In step S1403, the HMD position and attitude acquisition unit 204 determines whether the image captured by the left imaging unit 102L was received more stably than the image captured by the right imaging unit 102R. The HMD position and attitude acquisition unit 204 can determine, for example, that the image from the imaging unit with fewer consecutive frames of packet errors or packet losses was received more stably. The HMD position and attitude acquisition unit 204 also determines the area of ​​the exclusion region containing packet errors or packet losses (in the case of multiple frames, the sum or average of the areas of the exclusion regions of each frame). The image with the smaller value may be determined to have been received more stably. The HMD position and attitude acquisition unit 204 may combine these conditions to determine which image was received more stably.

[0080] If the image captured by the left imaging unit 102L is received more stably than the image captured by the right imaging unit 102R, the process proceeds to step S1404. If the image captured by the left imaging unit 102L is not received more stably than the image captured by the right imaging unit 102R, the process proceeds to step S1405.

[0081] In step S1404, the HMD position and attitude acquisition unit 204 acquires the position and attitude of the HMD 100 using the image captured by the left imaging unit 102L. In step S1405, the HMD position and attitude acquisition unit 204 acquires the position and attitude of the HMD 100 using the image captured by the right imaging unit 102R.

[0082] In Figure 14, an example is shown in which the position and orientation of the HMD 100 is obtained using images captured by two imaging devices. However, the information processing device 200 may also obtain the position and orientation of the HMD 100 using images captured by three or more imaging devices. The information processing device 200 only needs to obtain the position and orientation of the HMD 100 using the image that is received most stably among the images captured by the three or more imaging devices.

[0083] According to Embodiment 4 described above, even if a packet error or packet loss occurs when receiving an image via wireless communication, the impact on the acquisition process of the HMD100's position and orientation can be reduced by switching the image used.

[0084] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0085] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0086] Furthermore, although embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0087] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0088] This embodiment includes the following configurations, methods, and programs. (Composition 1) A receiving means for receiving images from the device, In the aforementioned image, an acquisition means for acquiring the position or orientation of the device based on a second region excluding a first region containing data errors or missing data, and An information processing device characterized by having the following features. (Configuration 2) The acquisition means identifies the second region using the position information of the first region, and acquires the position or orientation of the device based on the identified second region. The information processing device according to configuration 1, characterized by the above. (Composition 3) The system further includes processing means for performing image processing on the first region of the image so that the position and orientation of the device cannot be obtained from the first region of the image. The acquisition means acquires the position or orientation of the device using the image that has undergone the image processing. The information processing device according to configuration 1, characterized by the above. (Composition 4) The processing means, as part of the image processing, replaces the first region with a monochrome image. The information processing apparatus according to configuration 3, characterized by the features described herein. (Composition 5) The processing means further applies a gradient to the region including the boundary of the first region. The information processing apparatus according to configuration 4, characterized by the features described above. (Composition 6) The processing means performs the image processing, which involves removing high-frequency components from the first region of the image. The information processing apparatus according to configuration 3, characterized by the features described herein. (Composition 7) The first region further includes the region surrounding the region where the data error or data loss occurred. An information processing device according to any one of configurations 1 to 6, characterized by the above. (Composition 8) The acquisition means acquires the position or orientation of the device based on the second region of the image when the area of ​​the first region is smaller than a predetermined threshold. An information processing device according to any one of configurations 1 to 7, characterized by the above. (Composition 9) The receiving means receives multiple images captured by multiple imaging devices, The acquisition means, among the plurality of images, acquires the position or orientation of the device using the second image if the area of ​​the first region of the first image captured by the first imaging device is greater than a predetermined threshold, and the area of ​​the first region of the second image captured by the second imaging device is smaller than the predetermined threshold. An information processing device according to any one of configurations 1 to 8, characterized by the above. (Composition 10) The receiving means receives multiple images captured by multiple imaging devices, The acquisition means acquires the position or orientation of the device based on the second region of each of two or more images from the plurality of images in which the area of ​​the first region is smaller than a predetermined threshold. An information processing device according to any one of configurations 1 to 8, characterized by the above. (Composition 11) The receiving means receives multiple images captured by multiple imaging devices, The acquisition means selects the second region of the image that is most stably received among the plurality of images. The position or orientation of the device is obtained based on the region. An information processing device according to any one of configurations 1 to 8, characterized by the above. (Composition 12) The acquisition means determines, when receiving the multiple images in succession, that the image with fewer consecutive frames in which data errors or data loss occurred is the one that was received more stably. The information processing apparatus according to configuration 11, characterized by the features described above. (Composition 13) The acquisition means determines that the image with a smaller area of ​​the first region was received more stably. An information processing device according to configuration 11 or 12, characterized by the above. (Composition 14) The receiving means receives the image in a compressed state, The acquisition means decompresses the compressed image and acquires the position or orientation of the device based on a second region obtained by removing the first region from the decompressed image. An information processing device according to any one of configurations 1 to 13, characterized by the above. (method) The steps include receiving an image from the device, The steps include: obtaining the position or orientation of the device based on a second region of the image, excluding a first region containing data errors or missing data; An information processing method characterized by having the following features. (program) A program for causing a computer to function as one of the means of the information processing device described in any of configurations 1 to 14. [Explanation of Symbols]

[0089] 100: HMD, 200: Information processing unit, 201: Communication unit, 202: Detection unit, 203: Area determination unit, 204: HMD position and orientation acquisition unit

Claims

1. A receiving means for receiving images from the device, In the aforementioned image, an acquisition means for acquiring the position or orientation of the device based on a second region excluding a first region containing data errors or missing data, and An information processing device characterized by having the following features.

2. The acquisition means identifies the second region using the position information of the first region, and acquires the position or orientation of the device based on the identified second region. The information processing apparatus according to feature 1.

3. The system further includes processing means for performing image processing on the first region of the image so that the position and orientation of the device cannot be obtained from the first region of the image. The acquisition means acquires the position or orientation of the device using the image that has undergone the image processing. The information processing apparatus according to feature 1.

4. The processing means, as part of the image processing, replaces the first region with a monochrome image. The information processing apparatus according to feature 3.

5. The processing means further applies a gradient to the region including the boundary of the first region. The information processing apparatus according to feature 4.

6. The processing means performs the image processing, which involves removing high-frequency components from the first region of the image. The information processing apparatus according to claim 3.

7. The first region further includes the region surrounding the region where the data error or data loss occurred. The information processing apparatus according to feature 1.

8. The acquisition means acquires the position or orientation of the device based on the second region of the image when the area of ​​the first region is smaller than a predetermined threshold. The information processing apparatus according to feature 1.

9. The receiving means receives multiple images captured by multiple imaging devices, The acquisition means acquires the position or orientation of the device using the second image if, among the plurality of images, the area of ​​the first region of the first image captured by the first imaging device is greater than a predetermined threshold, and the area of ​​the first region of the second image captured by the second imaging device is smaller than the predetermined threshold. The information processing apparatus according to feature 1.

10. The receiving means receives multiple images captured by multiple imaging devices, The acquisition means acquires the position or orientation of the device based on the second region of each of two or more images from the plurality of images in which the area of ​​the first region is smaller than a predetermined threshold. The information processing apparatus according to feature 1.

11. The receiving means receives multiple images captured by multiple imaging devices, The acquisition means selects the second region of the image that is most stably received from among the plurality of images. The position or orientation of the device is obtained based on the region. The information processing apparatus according to feature 1.

12. The acquisition means determines, when receiving the multiple images in succession, that the image with fewer consecutive frames in which data errors or data loss occurred is the one that was received more stably. The information processing apparatus according to feature 11.

13. The acquisition means determines that the image with a smaller area of ​​the first region was received more stably. The information processing apparatus according to feature 11.

14. The receiving means receives the image in a compressed state, The acquisition means decompresses the compressed image and acquires the position or orientation of the device based on a second region obtained by removing the first region from the decompressed image. The information processing apparatus according to feature 1.

15. The steps include receiving an image from the device, The steps include: obtaining the position or orientation of the device based on a second region of the image, excluding a first region containing data errors or missing data; An information processing method characterized by having the following features.

16. A program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 14.

Images