Information processing apparatus, information processing method, and program

Abstract

To reduce latency.SOLUTION: An information processing apparatus has detection means and region determination means. The detection means detects characteristics from an image of a real space captured by an imaging apparatus. The region determination means predicts, based on the detected characteristics, a region having characteristics with respect to an image captured next by the imaging apparatus, and determines only the predicted region having characteristics, in the image captured next by the imaging apparatus, as a region to be transmitted to the detection means.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to information processing technology for realizing a composite reality space or a virtual space.

Background Art

[0002] As a head-mounted image display device (HMD) used in a mixed reality (MR) system, for example, an HMD of a video see-through type is known. The video see-through type HMD displays a composite image in which a CG image (an image of a virtual object, character information, etc. drawn by computer graphics) generated according to the position and orientation of the camera is superimposed on an image of the real space captured by a built-in camera (imaging device).

[0003] In an MR system, since it is required to correctly align the positions of the real space captured by the built-in camera and the virtual object generated according to the position and orientation of the camera, it is necessary to accurately and real-time estimate the position and orientation of the camera. As a method for estimating the position and orientation of the camera, for example, a method of detecting feature points fixed in the real space (that is, the three-dimensional coordinates do not change) from an image of the real space and estimating the position and orientation based on the detected feature points is known. Note that such a position and orientation estimation technology can be used not only when generating a composite reality space but also in a VR system that provides virtual reality (VR).

[0004] In an MR system, an observer can virtually operate a virtual object displayed on an HMD, and furthermore, an interaction with such an operation is realized. When realizing such an interaction, it is necessary to display a real object such as the hand of the observer who virtually operates the virtual object, that is, a real object that moves within the real space and whose appearance also changes, in front of (foreground) the virtual object. This is because if a real object that should be in front of the virtual object is hidden by the virtual object, the sense of distance and reality with the virtual object will be disrupted. In addition, in order to correctly represent the positional relationship between such a real object and the virtual object, detection and distance measurement of the real object in the image of the real space are required. However, it is not easy to detect in real time a real object that moves and deforms within the real space, such as the hand of an observer. As a method for detecting a moving and deforming real object, a method has been proposed in which the color information of the real object to be detected is registered in advance, and the real object is detected by extracting an image region of the same color as the registered color from the captured image.

[0005] As described above, in an MR system, since detection of feature points, detection of an image region of the registered color of a real object to be detected, etc. are performed using an image captured by a camera, a plurality of cameras suitable for acquiring feature points, colors, etc. from the image may be mounted on the HMD. Furthermore, the HMD may be equipped with a camera for distance measurement by stereo measurement using images of two cameras having a so-called stereo camera configuration.

[0006] In general MR systems, processes such as detecting real objects from captured images, detecting feature points, and image synthesis of the real space and virtual objects are performed by information processing devices such as PCs (Personal Computers) and WSs (WorkStations). Note that the HMD and the information processing device are connected by wire or wirelessly. However, as described above, when a plurality of cameras are mounted on the HMD, the information processing device has to perform processing using the image data captured by those plurality of cameras, which may increase the latency. That is, the time from capturing the real space until the synthesized image generated by the information processing device is displayed on the HMD, or the time until the image of the virtual object generated by the information processing device according to the movement of the observer's head or hand, etc., is displayed on the HMD becomes long.

[0007] Note that Patent Document 1 discloses a technique that can reduce latency by using an event-driven sensor to determine whether the detection target in the region of interest has moved, and disabling the tracking process when the detection target has not moved to reduce the processing time.

Prior Art Documents

Patent Documents

[0008]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0009] However, the technique of Patent Document 1 is a technique for reducing latency when tracking the movement of a user who is a player of a game, etc., and it is difficult to apply it for reducing latency in a system including an HMD equipped with a camera and an information processing device.

[0010] Therefore, an object of the present invention is to enable reduction of latency.

Means for Solving the Problems

[0011] The information processing apparatus of the present invention includes: a detection unit that detects features from an image captured by an imaging device in a real space; and an area determination unit that predicts an area having the features in an image to be captured next by the imaging device based on the detected features, and determines, as an area to be transmitted to the detection unit, only the area having the predicted features among the images captured next by the imaging device.

Effect of the Invention

[0012] According to the present invention, it is possible to reduce latency.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all combinations of the features described in this embodiment are essential for the solution means of the present invention. The configuration of the embodiment can be appropriately modified or changed according to the specifications of the apparatus to which the present invention is applied and various conditions (usage conditions, usage environment, etc.). Also, a configuration may be formed by appropriately combining a part of the following embodiments and modification examples. In the following embodiments, the same configurations and processes will be described with the same reference numerals respectively.

[0015] <First Embodiment> First, as a first embodiment, a configuration example of an information processing system that can reduce latency by reducing the transmission amount when transmitting image data from an HMD (head-mounted display device) to an information processing apparatus will be described. In the following description, unless otherwise specified, the image data handled in the HMD and the information processing apparatus will be simply described as an image only.

[0016] FIG. 1 is a diagram showing a configuration example of an information processing system according to the first embodiment. The information processing system according to the first embodiment includes an HMD 100 and an information processing apparatus 110. The HMD 100 and the information processing apparatus 110 are connected so that data communication can be performed between them by wire, wirelessly, or both wire and wireless.

[0017] The HMD100 is worn on the head of a user who is an observer. The observer can experience MR (Mixed Reality) or VR (Virtual Reality, Augmented Reality) by observing the display image of the HMD100. Note that the MR system can also be used, for example, to evaluate the ease of assembly and maintainability using design (shape and design) data created by a CAD (Computer-Aided Design) system. That is, by using the MR system for evaluating the ease of assembly and maintainability, it becomes possible to shorten the evaluation period and reduce costs using prototypes in the design and manufacturing fields. In this embodiment, a video see-through type HMD is described as an example of the HMD100, but other types of head-mounted image display devices such as an optical see-through type HMD may be used. The optical see-through type HMD displays a CG image generated according to the position and posture of the observer's viewpoint on an optical see-through type display worn on the observer's head. Also, the device used by the observer is not limited to a head-mounted image display device, and any display device that can allow the observer to observe a mixed reality space or a virtual space may be used. For example, other types of display devices such as a hand-held type display device may be used.

[0018] The display unit 102 of the HMD100 displays an image of a mixed reality space or a virtual space generated by the information processing device 110 and transmitted thereto. The display unit 102 may be configured to include two displays respectively arranged corresponding to the left and right eyes of the observer. In this case, an image of a mixed reality space or a virtual space for the left eye is displayed on the display corresponding to the left eye of the observer, and an image of a mixed reality space or a virtual space for the right eye is displayed on the display corresponding to the right eye of the observer.

[0019] The imaging unit 101 of the HMD 100 is an imaging device (built-in camera) that captures moving images of the real space, and includes a plurality of imaging units as described below. In the case of the first embodiment, it is assumed that the imaging unit 101 of the HMD 100 includes at least a right imaging unit, a left imaging unit, and a position and orientation imaging unit. Note that the imaging unit 101 may include other imaging units in addition to the right imaging unit, the left imaging unit, and the position and orientation imaging unit, but examples of those other imaging units will be described later.

[0020] The right imaging unit is an imaging unit that captures a color moving image for the right eye of the observer, and the left imaging unit is an imaging unit that captures a color moving image for the left eye of the observer. Images of each frame constituting the moving images captured by the right imaging unit and the left imaging unit (imaging images of the real space) are sequentially transmitted from the transmission unit 103 to the information processing device 110. In the information processing system of this embodiment, for example, when providing a composite sense of reality to the observer, the information processing device 110 generates a composite image obtained by synthesizing the moving images of the real space captured by the right imaging unit and the left imaging unit and the image of the virtual space, and sends it to the HMD 100. On the other hand, in the information processing system of this embodiment, when providing the virtual space to the observer, only the image of the virtual space is displayed on the display unit 102 of the HMD 100. Therefore, the right imaging unit and the left imaging unit of the imaging unit 101 do not need to capture moving images of the real space, and the transmission unit 103 does not need to send those moving images to the information processing device 110.

[0021] The imaging unit for position and orientation is an imaging unit for capturing an image used for calculating the position and orientation of the imaging unit 101 (the position and orientation of the HMD 100). That is, the image captured by the imaging unit for position and orientation is transmitted to the information processing device 110 described later and is used when the detection unit 112 detects features in the real space and calculates the position and orientation of the imaging unit 101. The imaging unit for position and orientation is a wide-angle imaging unit capable of imaging a wider range than the right imaging unit and the left imaging unit so as to detect many features in the real space, and is an imaging unit capable of imaging at a higher frame rate than the right imaging unit and the left imaging unit. Also, the imaging unit for position and orientation is an imaging unit capable of setting parameters such as the shutter speed and ISO sensitivity during imaging. In the case of this embodiment, it is assumed that the imaging unit for position and orientation is an imaging unit that captures grayscale images, but it may be an imaging unit capable of capturing color images. The image captured by the imaging unit for position and orientation is transmitted from the transmission unit 103 to the information processing device 110, that is, transmitted to the detection unit 112 of the information processing device 110.

[0022] Here, the transmission unit 103 of the HMD 100 of this embodiment also has a function of transmitting only a part of the image area determined by the area determination unit 113 of the information processing device 110 described later based on the result of the prediction process among the images captured by the imaging unit for position and orientation. That is, the transmission unit 103 also has a function of transmitting only a part of the captured image of the imaging unit for position and orientation, which is determined by the area determination unit 113 based on the result of the prediction process, as an image to be used when the detection unit 112 of the information processing device 110 detects features.

[0023] The information processing device 110 is composed of a computer device such as a PC or a portable device such as a smartphone or a tablet terminal. FIG. 2 is a diagram showing the basic configuration of a computer applicable to the information processing device 110 according to this embodiment. Note that the configuration of FIG. 2 is also applicable as the configuration of the information processing device 1020 in the second embodiment described later and the first information processing device 1210 and the second information processing device 1220 in the third embodiment described later.

[0024] In the hardware configuration of FIG. 2, the processor 201 is, for example, a CPU, which controls the operation of the entire computer. The memory 202 is, for example, a RAM, which temporarily stores an operating system (OS), various programs including the information processing program according to the first embodiment, various data including image data, and the like. The mass storage device 203 includes a computer-readable storage medium and is composed of, for example, an HDD, an SSD, a disk drive, and the like. The mass storage device 203 stores various programs including the OS and the information processing program, various data including image data, and the like in a long-term manner. In the case of this embodiment, the information processing program according to this embodiment stored in the mass storage device 203 is expanded onto the memory 202. Then, when the processor 201 operates according to the program on the memory 202, the functions of each functional unit in the information processing apparatus of this embodiment are realized. Note that the information processing program according to the second embodiment described later and the information processing program according to the third embodiment described later are similarly stored in the mass storage device 203, expanded onto the memory 202, and executed by the processor 201.

[0025] The input I / F 205 converts a signal sent from an operating device such as a mouse or a touch panel or an external device such as an HMD into a form processable by the information processing apparatus 110 and inputs it. The output I / F 206 converts a signal processed or generated internally into a form processable by an external device such as an HMD and outputs it. That is, the communication with the HMD 100 according to this embodiment is performed via the input I / F 205 and the output I / F. Note that the communication between the information processing apparatus 1210 and the HMD 1200 in the second embodiment described later is also performed via the input I / F 205 and the output I / F. Further, the communication between the HMD 1200 and the first information processing apparatus 1210 and the communication between the first information processing apparatus 1210 and the second information processing apparatus 1220 in the third embodiment described later are also performed via the input I / F 205 and the output I / F. Since the processing and technology related to the communication between these HMDs and information processing apparatuses and the communication between information processing apparatuses are well known, the description thereof is omitted.

[0026] Hereinafter, each functional unit and its operation of the HMD 100 and the information processing apparatus 110 according to the first embodiment shown in FIG. 1 will be described with reference to the flowchart of FIG. 3. FIG. 3(A) is a flowchart of the HMD 100 according to the first embodiment, and FIG. 3(B) is a flowchart of the information processing apparatus 110. In the description of each flowchart hereinafter, the symbol S represents a processing step (processing step).

[0027] Here, the information processing apparatus 110 according to the first embodiment detects features from the image captured by the imaging unit 101 of the HMD 100 and transmitted, and predicts where the features will be located in the next frame. Then, the information processing apparatus 110 determines an image region corresponding to the predicted position of the feature, and when the transmission unit 103 of the HMD 100 transmits the image of the next frame, requests that only the determined image region be transmitted. The transmission unit 103 of the HMD 100 transmits only a part of the image region determined by the information processing apparatus 110 among the images captured by the imaging unit 101 to the information processing apparatus 110. As a result, in the information processing system according to the first embodiment, the amount of image data transmitted from the HMD 100 to the information processing apparatus 110 is reduced, and as a result, the latency can be reduced. Note that each functional unit in the information processing apparatus 110 shown in FIG. 1 shows only the functional units for realizing the information processing according to this embodiment, and the illustration of other functional units is omitted. Similarly, for each functional unit of the HMD 100 shown in FIG. 1, only the functional units related to the realization of this embodiment are shown, and the illustration of other functional units is omitted. This is assumed to be the same in the second and third embodiments described later.

[0028] The description will be made from the flowchart of FIG. 3(A) showing the operation of the HMD 100. In S301, the imaging unit 101 captures an image. Next, in S302, the transmission unit 103 transmits the image captured by the imaging unit 101 to the information processing apparatus 110.

[0029] Here, in the information processing system of the present embodiment, when providing a virtual space to an observer, the transmission unit 103 does not transmit the images of the right imaging unit and the left imaging unit of the imaging unit 101, but only transmits the image captured by the imaging unit for position and orientation. Also, when transmitting the image captured by the imaging unit for position and orientation, the transmission unit 103 transmits only the image area determined based on the result of the prediction process in the area determination unit 113 (to be described later) of the information processing device 110 to the information processing device 110. That is, in the case of the first embodiment, the information processing device 110 transmits a transmission request for only a part of the image area of the imaging image of the imaging unit for position and orientation of the imaging unit 101 to the HMD 100 based on the result of the prediction process in the area determination unit 113 (to be described later). The reception unit 104 of the HMD 100 sends the transmission request to the transmission unit 103. The transmission unit 103 transmits only a part of the image area requested by the area determination unit 113 of the information processing device 110 based on the prediction process from the images of the imaging unit for position and orientation to the information processing device 110. Thereby, the transmission amount of the image data from the HMD 100 to the information processing device 110 is reduced, and the reduction of latency becomes possible.

[0030] Also, for example, when providing a composite reality space to an observer, the transmission unit 103 transmits the images of the right imaging unit and the left imaging unit and the image of the imaging unit for position and orientation. At this time, for the images of the right imaging unit and the left imaging unit, the transmission unit 103 transmits the entire image because it is necessary for generating the image of the composite reality space. On the other hand, for the image of the imaging unit for position and orientation, the transmission unit 103 transmits only a part of the image area requested by the area determination unit 113 of the information processing device 110 based on the prediction process. Thereby, the transmission amount of the image data from the HMD 100 to the information processing device 110 is reduced, and the reduction of latency becomes possible.

[0031] Next, in S303, the receiving unit 104 of the HMD 100 receives the image of the composite reality space or the image of the virtual space generated in the information processing apparatus 110. Then, the display unit 102 displays the received image. That is, when, for example, an image of the composite reality space is transmitted from the information processing apparatus 110, an observer wearing the HMD 100 can experience the composite reality space. Also, for example, when an image of the virtual space is transmitted from the information processing apparatus 110, the observer can experience the virtual space.

[0032] Thereafter, in S304, the HMD 100 determines whether an instruction to end the process has been input from the user. If an instruction to end the process has been input, the process of the flowchart in Fig. 3(A) ends. On the other hand, if an instruction to end the process has not been input, the process of the HMD 100 returns to S301, and the processes after S301 are performed.

[0033] Subsequently, the flowchart in Fig. 3(B) showing the operation of the information processing apparatus 110 will be described. In S311, the receiving unit 111 of the information processing apparatus 110 receives the image transmitted from the HMD 100. This received image is sent to the detection unit 112 via, for example, the calculation storage area formed in the memory 202 of Fig. 2. In the case of the present embodiment, only a part of the image area, which has been transmitted from the transmission unit 103 in response to the transmission request based on the prediction process in the area determination unit 113, among the images captured by the position and orientation imaging unit of the HMD 100, is input to the detection unit 112.

[0034] Next, in S312, the detection unit 112 detects the image coordinates of the features in the image from the image area transmitted from the transmission unit 103 of the HMD 100. In the case of the present embodiment, since the detection unit 112 does not need to detect features from the entire captured image of the position and orientation imaging unit, but only from a part of the image area, the features can be detected in a short processing time. Note that the detection unit 112 can also detect the image coordinates of the features from the images of the right imaging unit and the left imaging unit of the HMD 100.

[0035] Here, as a process for detecting features from an image, for example, a process of detecting, as features, points where the luminance gradient between adjacent pixels is equal to or greater than a threshold can be cited. The luminance gradient is the amount of change in the density of adjacent pixels on the image. The luminance gradient can be obtained, for example, by applying a Sobel operator or a Prewitt operator in the horizontal and vertical directions of the image. Then, the detection unit 112 detects points where the luminance gradient is equal to or greater than the threshold as feature points, and sends the detection results of the feature points to the estimation unit 114.

[0036] In addition, the detection unit 112 may, for example, detect feature points (natural feature points) each having a different texture feature. In this case, the detection unit 112 performs template matching using a template image having each feature held in advance as known information on the image, thereby detecting feature points from the image. As another example, the detection unit 112 may detect features having identifiers and distinguishable features, such as SIFT features. The features detected from the image are not limited to the examples described above, and any feature may be used as long as it is a feature whose position is fixed in the real space and can be detected from the image.

[0037] Next, in S313, the estimation unit 114 estimates the position and orientation of the HMD 100, that is, the position and orientation of the imaging unit 101, using the image coordinates of the features detected by the detection unit 112. Here, the estimation unit 114 holds in advance the relative positional relationship between the right imaging unit and the left imaging unit of the imaging unit 101 of the HMD 100 and the imaging unit for position and orientation. The estimation unit 114 estimates the position and orientation of the right imaging unit and the left imaging unit of the imaging unit 101 based on the held relative positional relationship and the image coordinates of the features detected from the image captured by the imaging unit for position and orientation.

[0038] For example, based on the features detected by the detection unit 112, the estimation unit 114 performs Visual SLAM (Simultaneous Localization and Mapping) processing to estimate the position and orientation of the imaging unit 101. Here, SLAM is a technique that obtains the position and orientation of the imaging unit in the real space from the images captured by the imaging unit, and at the same time, extracts keyframe information from the images as needed and constructs a map that accumulates the information of the keyframes. The information of the keyframes includes the images, the position and orientation of the imaging unit that captured the images, and the information of the features detected from the images. Note that since the technique of estimating the position and orientation of the imaging unit using SLAM processing is a known technique, the description thereof is omitted. However, as described above, in SLAM, the information of the entire image may be required as a keyframe. In that case, the estimation unit 114 requests the transmission unit 103 of the HMD 100 to transmit the remaining image regions that were not transmitted from the transmission unit 103 in response to the transmission request based on the prediction processing in the region determination unit 113, so as to receive the remaining image regions.

[0039] In this embodiment, the position and orientation are estimated based on the features detected from the images captured by the imaging unit for position and orientation of the imaging unit 101, but it is not limited to this example. For example, the HMD 100 may be provided with one or more sensors such as an attitude sensor, an optical sensor system, a magnetic sensor system, and an ultrasonic sensor system. The estimation unit 114 may acquire the sensor information detected by the sensors of the HMD 100 and estimate the position and orientation of the imaging unit 101 based on the sensor information. Also, for example, the estimation unit 114 may combine the position and orientation of the imaging unit 101 estimated based on the above-described SLAM processing and the position and orientation of the imaging unit 101 estimated based on the sensor information to estimate a higher-precision position and orientation. Since the techniques of estimating the position and orientation of the imaging unit by combining SLAM processing and the sensor information of each sensor are known, the description thereof is omitted.

[0040] Next, in S314, the region determination unit 113 predicts where one or more features detected from the image by the detection unit 112 will be located in the image of the frame (next frame) to be sent from the imaging unit 101 next. For example, the region determination unit 113 performs a linear prediction process using the image coordinates of the features detected from past frames in the image, the imaging time of the image at that time, and the predicted imaging time when the imaging unit 101 captures the next frame. In addition, the region determination unit 113 may, for example, approximate the transition of the position of the feature and the imaging time at that time with a polynomial and make a prediction from the polynomial, or may make a prediction using a known technique such as a Kalman filter.

[0041] Note that when the HMD 100 is equipped with at least any one of the sensors such as the posture sensor described above and can acquire sensor information at a higher frequency than the frame rate of the imaging unit 101, the region determination unit 113 may predict the position of the feature in the next frame based on the sensor information. In the case of this example, the region determination unit 113 may calculate the movement and rotation directions of the imaging unit 101 using the sensor information, and predict the movement direction and position of the feature in the next frame based on the calculation result.

[0042] Furthermore, based on the result of the prediction process as described above in S314, that is, based on the predicted position of the feature for the image of the next frame, the region determination unit 113 determines the image region to be transmitted as the region centered on the predicted position of the feature. Then, the region determination unit 113 requests the transmission unit 103 of the HMD 100 to transmit only the image of the image region determined to be transmitted. The image region centered on the predicted position of the feature may be a rectangular region, or may be only the divided region that includes the predicted feature among the plurality of divided regions divided in advance as shown in FIG. 4, for example.

[0043] FIG. 4(A) is a diagram showing an example of an image of the N-th frame 400 captured by the imaging unit 101 and a feature 401 detected by the detection unit 112 from the image of this frame 400. FIG. 4(B) is a diagram showing a feature 402 at a position predicted by the region determination unit 113 for the (N + 1)-th frame 410, which is the next frame with respect to the N-th frame. Further, FIG. 4(B) shows an example in which the image of the (N + 1)-th frame 410 is divided into a plurality of regions of 4 vertical blocks and 4 horizontal blocks. FIG. 4(C) shows, among the divided regions of the (N + 1)-th frame 410, a divided region 404 including the feature at the position predicted by the region determination unit 113 as a white region, while a divided region 403 not including the feature at the predicted position is shown as a hatched region. That is, the region determination unit 113 determines, for the (N + 1)-th frame 410, the image region to be transmitted as only the divided region 404 represented by the white region including the feature at the predicted position, and requests the transmission unit 103 of the HMD 100 to transmit only those determined image regions.

[0044] Also, the region determination unit 113 can appropriately change the size of the image region transmitted by the transmission unit 103. For example, if the difference between the position of the feature detected from the image of the N-th frame and the position of the feature predicted for the (N + 1)-th frame is equal to or greater than a predetermined difference threshold, the region determination unit 113 may increase the image region, while if it is less than the predetermined difference threshold, the region determination unit 113 may decrease the image region.

[0045] Next, in S315, the generation unit 115 constructs a virtual space in which each virtual object is arranged, using the information of the virtual object and the position and orientation of the virtual object acquired from the mass storage device 203. Alternatively, when generating an image of the virtual space, the generation unit 115 may be configured to allow a virtual object placed by the observer to be set. Then, the generation unit 115 generates an image of the virtual space as seen from the position and orientation of the imaging unit 101 calculated by the estimation unit 114 in S313. At this time, the generation unit 115 utilizes the relative relationships previously obtained for each imaging unit of the imaging unit 101 in order to generate images of the virtual space corresponding to the right eye and the left eye of the observer respectively. Note that since known techniques can be used for the technology of generating an image of the virtual space from the position and orientation of the imaging unit, the description thereof is omitted.

[0046] Also, when providing a composite reality space in the information processing system of the present embodiment, the generation unit 115 synthesizes the image of the virtual space generated as described above and the images of the real space captured by the right imaging unit and the left imaging unit of the imaging unit 101 to generate an image of the composite reality space. In the synthesis process at this time, a process of superimposing the image of the virtual space on the image of the real space is performed. That is, the composite image is an image in which the pixels of the real space are displayed for the pixels other than the area of the virtual object. Note that when only the virtual space is provided in the information processing apparatus of the present embodiment, the generation unit 115 does not perform the synthesis of the image of the real space with the image of the virtual space generated as described above, and outputs only the image of the virtual space.

[0047] Next, in S316, the transmission unit 116 transmits the image of the composite reality space or the image of the virtual space generated by the generation unit 115 as described above to the HMD 100. The reception unit 104 of the HMD 100 sends the image of the composite reality space or the image of the virtual space transmitted from the information processing apparatus 110 to the display unit 102. Thereby, when the image of the composite reality space is transmitted from the information processing apparatus 110, the observer can experience the composite reality space. Also, when only the image of the virtual space is transmitted from the information processing apparatus 110, the observer can experience the virtual space.

[0048] Next, in S317, the information processing apparatus 110 determines whether an instruction to end the process has been input from the user. If an instruction to end the process has been input, the process of the flowchart in FIG. 3(B) is ended. On the other hand, if an instruction to end the process has not been input, the information processing apparatus 110 returns the process to S311, and thereby the process after S311 is performed in the information processing apparatus 110.

[0049] As described above, in the first embodiment, the information processing apparatus 110 detects features from the image captured by the imaging unit 101 of the HMD 100, and requests the HMD 100 to transmit only the image area including the positions of the features predicted in the next frame. Then, when transmitting the image of the next frame to the information processing apparatus 110, the HMD 100 transmits only the image area including the positions of the features predicted in the information processing apparatus 110. According to the first embodiment, therefore, the transmission amount when transmitting image data from the HMD 100 to the information processing apparatus 110 can be reduced, and as a result, the latency can be reduced.

[0050] <Modification Example 1 of the First Embodiment> Next, as a modification example 1 of the first embodiment, an example of detecting a marker having a known size as a feature will be described. Hereinafter, the description of the configuration and process common to the first embodiment will be omitted as appropriate, and the differences from the first embodiment will be mainly described. In the case of the information processing system according to modification example 1, the operations of the detection unit 112, the estimation unit 114, and the region determination unit 113 of the information processing apparatus 110 are different from those of the first embodiment.

[0051] The detection unit 112 extracts, as a feature, a marker having a known size from the image captured by the position and orientation imaging unit of the HMD 100. First, the detection unit 112 detects the marker 501 from the image of the Nth frame 500 shown in FIG. 5(A), for example, and sends the feature that is the detection result to the estimation unit 114. The estimation unit 114 estimates the position and orientation of the imaging unit 101 based on the feature of the marker detected by the detection unit 112.

[0052] For example, when the marker is a rectangle, the detection unit 112 may detect the four corners of the rectangular marker as feature points and send them to the region determination unit 113, or may send the entire region of the marker as a feature to the region determination unit 113. In this case, similar to the first embodiment, for the (N + 1)-th frame 510 shown in FIG. 5(B), the region determination unit 113 predicts the four corners of the marker as feature points 502. Alternatively, as shown in FIG. 5(C), the region determination unit 113 may predict the marker region 503 in the (N + 1)-th frame 510 as a feature. FIG. 5(D) shows, among the divided regions of the (N + 1)-th frame 510, the divided region 505 including the position of the feature predicted by the region determination unit 113 as a white region, while the divided region 504 not including the position of the predicted feature is shown as a hatched region. That is, in the case of Modification 1, the region determination unit 113 requests the transmission unit 103 of the HMD 100 to transmit only the image region of the divided region 505 including the position of the predicted feature for the (N + 1)-th frame 510 as shown in FIG. 5(D).

[0053] As described above, in the case of Modification 1, the detection unit 112 detects a marker as a feature for estimating the position and orientation of the imaging unit 101, and the information processing apparatus 110 requests the transmission unit 103 to transmit only the image region where the marker is predicted to be present in the next frame. Thereby, also in Modification 1, the transmission amount can be reduced and the latency can be reduced.

[0054] <Modification 2 of the First Embodiment> Next, as Modification 2 of the first embodiment, an example of detecting a specific real object as a feature will be described. Hereinafter, descriptions of the configurations and processes common to the first embodiment will be omitted as appropriate, and the differences will be mainly described. In the case of the information processing system according to Modification 2, the operations of the reception unit 111, the detection unit 112, and the region determination unit 113 of the information processing apparatus 110 are different from those of the first embodiment. Also, in the case of Modification 2, the configuration of the imaging unit 101 and the operation of the transmission unit 103 of the HMD 100 are different from those of the first embodiment.

[0055] In the case of Modification Example 2, it is assumed that the imaging unit 101 includes, in addition to the right imaging unit and the left imaging unit described above and the imaging unit for position and orientation, an imaging unit for real objects. The imaging unit for real objects is an imaging unit that captures a moving image of the real space for the detection unit 112 of the information processing apparatus 110 to detect a specific real object as a feature. The imaging unit for real objects may have a stereo camera configuration capable of obtaining the depth of the real object by stereo measurement, and may also be able to set shooting parameters such as the shutter speed and ISO sensitivity during imaging in order to capture the real object more clearly. When the imaging unit 101 also includes an imaging unit for real objects as in this Modification Example 2, the information processing apparatus 110 pre-holds the relative positional relationship between the imaging unit for real objects and the right imaging unit and the left imaging unit. Further, when the imaging unit for real objects has a stereo camera configuration, the information processing apparatus 110 also pre-holds the relative positional relationship between the imaging units for real objects of the stereo camera configuration.

[0056] In the case of Modification Example 2 in which the imaging unit 101 includes an imaging unit for real objects, the transmission unit 103 also transmits the image captured by the imaging unit for real objects to the information processing apparatus 110. The reception unit 111 of the information processing apparatus 110 sends the image captured by the imaging unit for real objects to the detection unit 112, and the detection unit 112 detects the real object as a feature from the image and sends it to the region determination unit 113.

[0057] The region determination unit 113 in Modification 2 predicts the position of the feature in the next frame, that is, the position of the real object in the next frame, based on the feature (the feature of the real object) detected by the detection unit 112 from the image captured by the real object imaging unit. Then, when the region determination unit 113 in Modification 2 can predict the position of the feature in the next frame, that is, the position of the real object in the next frame, it requests the transmission unit 103 to transmit the image region including the predicted feature. Thus, in the case of Modification 2, the transmission unit 103 also transmits only the image region corresponding to the request from the region determination unit 113 for the image of the real object imaging unit. That is, in the case of Modification 2, the detection unit 112 detects a specific real object as a feature using the image region transmitted from the transmission unit 103 among the images captured by the real object imaging unit. Note that also in the case of Modification 2, the transmission unit 103, similar to the first embodiment, also transmits only the image region predicted by the region determination unit 113 to have a feature in the next frame for the image of the position and orientation imaging unit.

[0058] As described above, in Modification 2, an example in which the imaging unit 101 includes a real object imaging unit is given, but the feature of a specific real object is not limited to being detected from the image captured by the real object imaging unit. That is, in Modification 2, the imaging unit 101 does not necessarily have to include a real object imaging unit. In that case, the detection unit 112 may detect a real object as a feature using, for example, the image region transmitted from the transmission unit 103 among the images captured by the position and orientation imaging unit.

[0059] Here, in Modification 2, examples of the specific real object that the detection unit 112 detects features of include, for example, the hand of the observer when the observer virtually operates a virtual object in the composite reality space, or an object held by the observer. Hereinafter, these hands of the observer and the specific real objects held in the hands are referred to as real objects to be detected.

[0060] As a method for detecting the hand of an observer as a real object to be detected, the method disclosed in the reference (Japanese Patent Application Laid-Open No. 2013-134706) can be used. When using the method of the reference, since it is necessary to register the color information of the hand in advance, it is assumed that the information processing apparatus 110 according to the second modification also registers the color information of the hand in advance. Further, when using the method of Reference 1, it is desirable that the imaging unit for real objects has a stereo camera configuration, and depth information can be obtained by stereo measurement using the image captured by the imaging unit for real objects having the stereo camera configuration.

[0061] Further, when the observer virtually operates a virtual object in the composite reality space, it is necessary to detect the real object (hand or object held in the hand) to be detected from the captured image of the real space, and further measure the distance in the depth direction of the detected real object. That is, it is necessary to obtain the positional relationship in the depth direction between the detected real object and the virtual object, and perform an interference determination between the detected real object and the virtual object. For this reason, when the detection unit 112 of the second modification detects the real object to be detected from the image of the real space captured by the imaging unit for real objects, it calculates the distance in the depth direction of the detected real object, and further performs an interference determination between the detected real object and the virtual object. As a method for detecting a virtual operation of the observer on the virtual object, for example, a method of recognizing the movement of the observer's hand or the object held in the hand, that is, gesture recognition, and detecting a virtual operation on the virtual object according to the result of the gesture recognition can be mentioned.

[0062] In addition, as a method for detecting the real object to be detected by using its characteristics, for example, a method of tracking the real object may be used. In this example, the information processing apparatus 110 holds in advance the information used for tracking. The information used for tracking is, for example, a template image when tracking is performed using template matching, and learning data when tracking is performed using deep learning.

[0063] FIG. 6(A) is a diagram showing an example of a real object 601 detected as a feature by a detection unit 112 in an image of, for example, the N-th frame 600. The detection unit 112 detects the region of the real object 601 to be detected as a feature and sends it to a region determination unit 113. In addition, as shown in FIG. 6(B), the detection unit 112 may send the four corners 602 of the rectangular region surrounding the real object 601 as features to the region determination unit 113.

[0064] In the case of Modification 2, the region determination unit 113 predicts in which region of the image of the next frame the features of the real object to be detected and the features used in the above-described SLAM process will move. For example, when the features of the real object to be detected are detected from the image of the real object imaging unit, the region determination unit 113 predicts in which region of the image of the next frame captured by the real object imaging unit the features will move. Further, when the features used in the above-described SLAM process are detected from the image captured by the position and orientation imaging unit, the region determination unit 113 predicts in which region of the image of the next frame captured by the position and orientation imaging unit the features will move. Further, when the imaging unit 101 does not include the real object imaging unit, the region determination unit 113 predicts in which region of the next frame of the position and orientation imaging unit the features of the real object to be detected detected from the image of the position and orientation imaging unit and the features used in the SLAM process will move.

[0065] FIG. 7(A) is a diagram showing, on the same image, the feature 701 of the real object detected from the image of the N-th frame 700 by the imaging unit for real objects and the feature 702 used in the SLAM process detected from the image of the N-th frame 700 by the imaging unit for position and orientation. When the imaging unit 101 does not include the imaging unit for real objects, FIG. 7(A) is a diagram showing the feature 701 of the real object detected from the image of the N-th frame 700 by the imaging unit for position and orientation and the feature 702 used in the SLAM process. In the case of the example of FIG. 7(A), as shown in FIG. 7(B), the region determination unit 113 predicts, for the next (N + 1)-th frame 710, in which divided region both the feature 702 used in the SLAM process and the feature 703 of the real object to be detected are located. FIG. 7(C) is a diagram showing, among the divided regions of the (N + 1)-th frame 710, the divided region 705 including the position of the feature predicted by the region determination unit 113 as a white region and the divided region 706 not including the position of the feature as a hatched region. That is, in the case of Modification 2, as shown in FIG. 7(C), for the (N + 1)-th frame 710, the region determination unit 113 requests the transmission unit 103 of the HMD 100 to transmit only the image regions corresponding to the divided regions 705 each including the positions of both the features 703 and 704.

[0066] Also, in the case of Modification 2, a posture sensor may be attached to the real object itself to be detected. When the real object to be detected is the hand of an observer, the posture sensor is a sensor that can be attached to the hand, and the posture sensor is, for example, a sensor capable of acquiring sensor information at a higher frequency than the frame rate of the imaging unit 101. The sensor information detected by this posture sensor is sent to the estimation unit 114 of the information processing device 110. Based on the sensor information, the estimation unit 114 calculates the movement and rotation direction of the real object to be detected (such as the hand of the observer) at a high frequency. Also, in the case of this example, the region determination unit 113 may use the sensor information of the posture sensor in addition to the features detected by the detection unit 112. By using the sensor information of the posture sensor, the region determination unit 113 can calculate the movement and rotation direction of the real object to be detected (such as the hand of the observer), so that the position of the real object to be detected in the next frame can be predicted with higher accuracy. In addition, for example, a marker for acquiring position and posture may be attached to the real object to be detected, and the position, rotation direction, etc. of the real object to be detected may be calculated and predicted using the sensor information obtained by detecting the marker using an optical sensor system or the like.

[0067] As described above, according to Modification 2, the real object to be detected is detected using at least one of the images of the imaging unit for real objects and the imaging unit for position and posture. Then, the region determination unit 113 determines only the image region of the region where the real object to be detected is predicted to be present in the next frame as the image region to be transmitted by the transmission unit 103. Thereby, also in Modification 2, the transmission amount can be reduced and the latency can be reduced.

[0068] <Modification 3 of the First Embodiment> In the case of the first embodiment described above, for example, when the position of the predicted feature for the next frame is near the edge of the image region, for example, there is a possibility that the feature may not be detected when the detection unit 112 of the image of the next frame detects the feature. Therefore, as a modification example 3 of the first embodiment, when the predicted feature is near the edge of the image region, the region determination unit 113 determines a range larger than the image region determined based on the predicted feature as the image region to be transmitted by the transmission unit 103 of the HMD 100. In modification example 3, the region determination unit 113 determines a range larger than the image region determined based on the predicted feature according to the setting of feature detection by the detection unit 112, for example, information such as a filter. Hereinafter, the description of the configurations and processes common to the first embodiment will be omitted as appropriate, and the differences will be mainly described. In the case of the information processing system according to modification example 3, the operation of the region determination unit 113 of the information processing apparatus 110 is different from that of the first embodiment.

[0069] In modification example 3, when the predicted feature is near the edge of each divided region as shown in FIG. 4(B), for example, the region determination unit 113 requests the transmission unit 103 to transmit an image region in a range larger than that divided region. The range larger than the divided region can include, for example, a range corresponding to the setting of feature detection by the detection unit 112, such as a range enlarged by the filter size of the Sobel operator or the Prewitt operator when detecting feature points used in the SLAM process. Also, for example, the range larger than the divided region may be a range enlarged by the size of the template image when template matching is performed. Also, for example, the range larger than the divided region may be a range enlarged by the size required when obtaining, for example, SIFT features. In addition, the region determination unit 113 may compare, for example, the position of the feature predicted in the Nth frame with the position of the feature actually detected from the image of the Nth frame, and use a range enlarged by the amount of error between those positions as the range for the next N+1th frame.

[0070] As described above, according to Modification 3, even when the predicted feature is near the edge of the image region, the feature can be detected in the next frame.

[0071] <Modification 4 of the First Embodiment> In the case of the above-described First Embodiment and Modifications 1 to 3, the feature predicted by the region determination unit 113 may be, for example, in the entire image, and as a result, the image region transmitted from the transmission unit 103 of the HMD 100 may become the entire image. In this case, the latency cannot be reduced, and the frame rate may decrease due to an increase in the transmission time.

[0072] Therefore, in Modification 4 of the First Embodiment, when the image region transmitted from the transmission unit 103 becomes the entire image as a result of predicting the feature, the feature points are counted for each divided region, and the divided region in which the number of feature points is equal to or more than a predetermined number threshold is determined as the image region to be transmitted by the transmission unit 103. Hereinafter, the description of the configuration and processing common to the First Embodiment will be omitted as appropriate, and the differences will be mainly described. In the case of the information processing system according to Modification 4, the operation of the region determination unit 113 of the information processing apparatus 110 is different from that of the First Embodiment.

[0073] When the predicted feature points, as described above for the features used in the SLAM process, cover the entire image captured by the position and orientation imaging unit, the region determination unit 113 of Modification Example 4 counts the feature points for each divided region. Then, the region determination unit 113 determines each divided region with the number of feature points equal to or more than a predetermined number threshold as the image region to be transmitted by the transmission unit 103. For example, the region determination unit 113 counts the feature points for each divided region as described in FIG. 4(B) above. Note that the divided region does not necessarily have to be the divided region of 4 vertical blocks and 4 horizontal blocks as described above, and may be a divided region according to the division method set in the information processing device 110 in advance. Then, the region determination unit 113 determines each divided region with the number of feature points equal to or more than a predetermined number threshold as the image region to be transmitted by the transmission unit 103, and requests the transmission unit 103 to transmit the image region. As a result, only the image regions corresponding to the divided regions with the number of feature points equal to or more than a predetermined number threshold are transmitted from the transmission unit 103 of the HMD 100, and it becomes possible to reduce the latency by reducing the transmission amount.

[0074] Note that, as the predetermined number threshold for the number of feature points, for example, the average value of the number of feature points in all divided regions may be used, or a threshold determined using, for example, Otsu's binarization method for the number of feature points in each divided region may be used. In addition, a number threshold arbitrarily set in advance for the information processing device 110 may also be used.

[0075] In the above example, the case where the predicted feature covers the entire image is taken as an example, but it is not limited to the entire image. For example, when the data amount of the image region to be transmitted becomes equal to or more than a predetermined data amount threshold, the region determination unit 113 may determine each divided region with the number of feature points equal to or more than a predetermined number threshold as the image region to be transmitted by the transmission unit 103 as described above. Also in this example case, the transmission amount can be reduced and the latency can be reduced. Note that the data amount threshold for the data amount to be transmitted may be determined by the observer, or may be determined based on the allowable latency and transmission speed set by the observer.

[0076] As described above, according to Modification 4, even when the area to be transmitted is the entire image or the data volume becomes large, the transmission volume can be suppressed and the latency can be reduced.

[0077] <Modification 5 of the First Embodiment> Next, in Modification 5 of the first embodiment, for example, as a result of predicting the position of the feature of the real object of a specific detection target, the image area transmitted from the transmission unit 103 becomes the entire image, and the case where the latency cannot be reduced will be described. In Modification 5, the case of stereometric measurement of a real object is particularly taken as an example, and the area determination unit 113 predicts an area where stereometric measurement cannot be performed, and for the area where stereometric measurement cannot be performed, it is determined that the transmission unit 103 does not transmit. In other words, the area determination unit 113 predicts an area where stereometric measurement can be performed, and determines only the image area corresponding to the area where stereometric measurement can be performed as the image area to be transmitted by the transmission unit 103. Hereinafter, the description of the parts common to the first embodiment will be omitted as appropriate, and the differences will be mainly described. In the case of the information processing system according to Modification 5, the operation of the area determination unit 113 of the information processing apparatus 110 is different from that of the first embodiment.

[0078] In Modification 5, the hand of the observer exemplified in Modification 2 described above is used as a real object of a specific detection target, the imaging unit 101 includes an imaging unit for real objects having a stereo camera configuration, and the detection unit 112 also detects the depth direction of the real object of the detection target. A real object that moves in the real space, such as the hand of an observer, may approach the imaging unit 101. In this case, the area of the real object of the detection target predicted by the area determination unit 113 for the next frame may occupy the entire image or most of the image. Thus, when the area of the real object of the detection target predicted by the area determination unit 113 occupies the entire image or most of the image, the image area transmitted from the transmission unit 103 is also the entire image or the data volume becomes equal to or greater than a predetermined data volume threshold. Note that the data volume threshold in the case of Modification 5 may be determined by the observer as in the case of Modification 4 described above, or may be determined in consideration of the transmission speed from the allowable latency set by the observer.

[0079] When, as in Modification 5, the real object imaging unit of the imaging unit 101 has a stereo camera configuration having two imaging units, the region determination unit 113 acquires the depth value of the real object by stereo measurement using the images captured by the two imaging units of the stereo camera configuration. Further, the region determination unit 113 predicts the depth value of the real object in the next frame based on the depth value of the real object. Then, the region determination unit 113 calculates the overlapping region of the two images captured by the two imaging units of the stereo camera configuration in the next frame, that is, predicts the overlapping region in the next frame, based on the depth value predicted for the next frame. Here, a region that does not overlap between the two images captured by the two imaging units of the stereo camera configuration of the real object imaging unit is a region in which the depth value cannot be obtained by stereo measurement even if features are detected by the detection unit 112. Therefore, the region determination unit 113 determines, for the next frame, a region that does not overlap between the two images obtained by the imaging units of the stereo camera configuration, that is, a region where stereo measurement cannot be performed, as an image region not to be transmitted from the transmission unit 103. In other words, the region determination unit 113 determines, for the next frame, only the image region corresponding to the overlapping region between the two images obtained by the stereo camera configuration as the image region to be transmitted from the transmission unit 103.

[0080] FIG. 8 is a diagram used to explain the overlapping region of two images captured by two imaging units 811 and 812 of a stereo camera configuration. In FIG. 8, the range 801 that can be imaged by one imaging unit 811 of the stereo camera configuration and the range 802 that can be imaged by the other imaging unit 812 are frustums of a cone with the respective imaging units as vertices. A real object existing within the overlapping region 803 of the ranges 801 and 802 that can be imaged by the two imaging units 811 and 812 can have its depth value calculated by stereo measurement.

[0081] The depth value determination unit 113 predicts (calculates) an overlapping region for which a depth value can be calculated in the next frame based on the overlapping region 803 for which a depth value can be calculated in the current frame. Here, in a region outside the overlapping region for which a depth value can be calculated for the next frame, it is a region where a depth value cannot be obtained. For this reason, the depth value determination unit 113 determines, in the next frame, the image region corresponding to the overlapping region for which a depth value can be calculated as the image region to be transmitted by the transmission unit 103. In other words, the depth value determination unit 113 determines, in the next frame, that the region outside the overlapping region is the image region not to be transmitted by the transmission unit 103.

[0082] Note that the prediction of the depth value and the prediction of the overlapping region in the depth value determination unit 113 may be linearly predicted from past data in the same manner as when predicting features. In addition, the depth value determination unit 113 may approximate the transition of the imaging time and the depth value with a polynomial and predict the overlapping region by predicting the depth value from the polynomial, or may predict the overlapping region using a known technique such as a Kalman filter. Also, the frustums corresponding to the two imaging units of the stereo camera configuration are assumed to be estimated in advance by calibration or the like, but may be changed as appropriate. In that case, it may be estimated again by calibration, or appropriately selected from the data of a plurality of frustums already estimated and held by the information processing device 110.

[0083] As described above, according to Modification 5, even when, for example, a real object of a specific detection target approaches the imaging unit and the region transmitted by the transmission unit 103 becomes the entire image or the transmission amount increases, the transmission amount can be reduced and the latency can be reduced.

[0084] <Modification 6 of the First Embodiment> Next, as a sixth modification of the first embodiment, as a result of predicting features as in the aforementioned fourth and fifth modifications, in the case where the image area transmitted from the transmission unit 103 becomes the entire image, an example using a solution method different from the aforementioned fourth and fifth modifications will be described. In the cases of the aforementioned fourth and fifth modifications, the area determination unit 113 predicts and determines the image area transmitted by the transmission unit 103. In contrast, an example will be described in which the area determination unit 113 of the sixth modification converts the image into frequency components by the transmission unit 103 and further determines which frequency region should be transmitted according to the predicted features. Hereinafter, the description of the parts common to the first embodiment will be omitted as appropriate, and the differences will be mainly described. In the case of the information processing system according to the sixth modification, the operations of the transmission unit 103 of the HMD 100, the detection unit 112, and the area determination unit 113 of the information processing apparatus 110 are different from those of the first embodiment.

[0085] In the sixth modification, as a result of the area determination unit 113 performing the same prediction process as described above, it is possible that the feature points become the entire image, or the data amount of the image area to be transmitted becomes equal to or greater than a predetermined data amount threshold. In addition, there may be a case where the real object or marker to be detected approaches the imaging unit and the data amount of the entire image or the image area to be transmitted becomes equal to or greater than the data amount threshold. Note that also in the example of the sixth modification, the data amount threshold may be determined by the observer, or may be determined in consideration of the transmission speed from the allowable latency set by the observer.

[0086] In the sixth modification, as a result of the prediction process, when the entire image or transmission equal to or greater than the data amount threshold occurs, the area determination unit 113 determines to convert the image into frequency components, and further determines which frequency region will be the image area to be transmitted. Then, the area determination unit 113 transmits a transmission request indicating the conversion into frequency components and which frequency region will be the image area to be transmitted to the HMD 100 via the transmission unit 116.

[0087] In Modification Example 6, as an example of conversion into frequency components, discrete wavelet transform will be described. FIG. 9 is a diagram used to explain the discrete wavelet transform and frequency components performed by the transmission unit 103 of the HMD 100 in response to a transmission request from the region determination unit 113. FIG. 9(A) is a diagram showing the original image 900 before being converted into frequency components. FIG. 9(B) is a diagram showing an example of each frequency region obtained by performing discrete wavelet transform on the original image 900. That is, the original image 900 is converted into a low-frequency region 901, an X-direction high-frequency region 902, a Y-direction high-frequency region 903, and an XY-direction high-frequency region 904 as shown in FIG. 9(B) by discrete wavelet transform.

[0088] Here, when the detection unit 112 detects, for example, the characteristics of a real object of a specific detection target or the characteristics of a marker, the region determination unit 113 determines the low-frequency region 901 as the image region to be transmitted by the transmission unit 103. Note that when it is determined to transmit the low-frequency region 901, the detection accuracy of the characteristics of the real object of the detection target or the characteristics of the marker in the detection unit 112 will decrease slightly, but it is still possible to detect those characteristics even in the image of the low-frequency region. Also, for example, when the detection unit 112 detects feature points used in SLAM, the region determination unit 113 determines the image region to be transmitted by the transmission unit 103 to be at least one or more of the X-direction high-frequency region 902, the Y-direction high-frequency region 903, and the XY-direction high-frequency region 904. Note that in the case of SLAM processing, since feature points are detected using edges obtained as a result of applying an operator such as a Sobel operator, feature points can be detected in the image of the high-frequency region. Also in this case, the feature detection accuracy will decrease, but it is possible to reduce the latency, and it is possible to suppress the occurrence of observer's video sickness due to an increase in latency.

[0089] Also, the region determination unit 113 may further perform a discrete wavelet transform on the low-frequency region 901 in FIG. 9(B), and determine the conversion into a low-frequency region 905, an X-direction high-frequency region 906, a Y-direction high-frequency region 907, and an XY-direction high-frequency region 908 as shown in FIG. 9(C). And in the case of this example, when the detection unit 112 detects, for example, the features of a real object or a marker to be detected, the region determination unit 113 determines the image region transmitted by the transmission unit 103 as the low-frequency region 905. Also, for example, when the detection unit 112 detects feature points used in SLAM, the region determination unit 113 determines at least one or more of the X-direction high-frequency regions 906 and 902, the Y-direction high-frequency regions 907 and 903, and the XY-direction high-frequency regions 908 and 904. That is, the region determination unit 113 causes the transmission unit 103 to perform the discrete wavelet transform a plurality of times, and determines which frequency region in the frequency regions of the conversion results of the plurality of times as the image region transmitted by the transmission unit 103.

[0090] In the above description, the discrete wavelet transform is taken as an example, but the region determination unit 113 may determine to perform a Fourier transform. In that case, the transmission unit 103 performs a Fourier transform and determines which region passed through a band-pass filter to transmit.

[0091] As described above, the region determination unit 113 sends a transmission request indicating the determined discrete wavelet transform and the number of times of the transform, and which frequency region is to be transmitted as the image region, to the transmission unit 103. The transmission unit 103 performs a discrete wavelet transform on the image in response to the request from the region determination unit 113, and transmits the frequency region corresponding to the request. That is, in Modification 6, when the detection unit 112 detects the features of a real object or a marker to be detected, the low-frequency region is transmitted as the image region from the transmission unit 103. Also, for example, when the detection unit 112 detects feature points used in SLAM, the high-frequency region is transmitted as the image region from the transmission unit 103. Also, for example, when a transmission request indicating a discrete Fourier transform is sent from the region determination unit 113, the transmission unit 103 performs a discrete Fourier transform on the image, and transmits only the region obtained by applying a band-pass filter to the frequency components according to the request as the image region.

[0092] In the case of Modification 6, the detection unit 112 detects the real object or marker to be detected as a feature from the low-frequency region obtained by the discrete wavelet transform. Further, the detection unit 112 detects feature points used in SLAM from the high-frequency region obtained by the discrete wavelet transform. Further, for example, when an image region subjected to Fourier transform is transmitted, the detection unit 112 performs feature detection from the image after performing inverse Fourier transform.

[0093] As described above, in the case of Modification 6, when the image region to be transmitted is the entire image or the data amount increases as a result of prediction, the region determination unit 113 requests the transmission unit 103 to convert the image into frequency components and determine which frequency region to transmit. The transmission unit 103 converts the image into frequency components according to the transmission request and transmits the requested frequency region. Thereby, the transmission amount and the latency can be reduced between the HMD 100 and the information processing apparatus 110.

[0094] <Modification 7 of the First Embodiment> Next, as Modification 7 of the first embodiment, an example will be described in which an image used to detect a feature with a high detection priority among a plurality of images captured by a plurality of imaging units included in the imaging unit 101 is determined as a partial image region transmitted from the transmission unit 103. In other words, in Modification 7, for an image used to detect a feature with a low detection priority among a plurality of images captured by the plurality of imaging units of the imaging unit 101, the transmission amount is reduced by not transmitting it from the transmission unit 103, and the latency can be reduced. Hereinafter, the description of the parts common to the first embodiment will be omitted as appropriate, and the differences will be mainly described. In the case of the information processing system of Modification 7, the operations of the transmission unit 103, the detection unit 112, the estimation unit 114, and the region determination unit 113 of the HMD 100 are different from those of the first embodiment.

[0095] In the case of Modification Example 7, the region determination unit 113 determines, as the image region to be transmitted from the transmission unit 103, only the image selected according to the priority of the features detected by the detection unit 112 among the plurality of images captured by the plurality of imaging units of the imaging unit 101. When the data amount of the image region to be transmitted from the transmission unit 103 exceeds a predetermined data amount threshold value, the region determination unit 113 of Modification Example 7 sets a priority for the features detected by the detection unit 112 or acquires the priority set by the observer. Then, the region determination unit 113 determines that only the image for detecting the feature with a high priority is the image region to be transmitted from the transmission unit 103. In other words, it is determined not to transmit the image for detecting the feature with a low priority. In Modification Example 7, the data amount threshold value may be the data amount of the entire image captured by the position and orientation imaging unit and the entire image captured by the real object imaging unit, or may be determined in consideration of the time required for transmission from the allowable value of the latency set by the observer.

[0096] Hereinafter, several examples of setting the priority of the detected features will be described. For example, assume that the imaging unit 101 of the HMD 100 is provided with an attitude sensor, and in addition to the right imaging unit, the left imaging unit, the position and orientation imaging unit, a real object imaging unit is provided in the imaging unit 101. When the imaging unit 101 is provided with an attitude sensor in this way, for example, it is not always necessary to estimate the position and orientation of the imaging unit 101 based on the features used in the SLAM process or the features of the marker. In this case, the region determination unit 113 sets a low priority for the features used in the SLAM process and the features of the marker, does not transmit the image captured by the position and orientation imaging unit, and determines that only the image captured by the real object imaging unit is the image region to be transmitted by the transmission unit 106. In this way, the priority may be set based on the configuration of the information processing system.

[0097] For example, the imaging unit 101 does not have an attitude sensor, and the imaging unit 101 is provided with a right imaging unit, a left imaging unit, a position and attitude imaging unit, and a real object imaging unit. It is assumed that the detection unit 112 can detect the feature points and marker features used in the SLAM process from the image of the real object imaging unit. In this example, when the amount of transmission data from the transmission unit 103 exceeds the data amount threshold, the region determination unit 113 sets the priority of the features detected from the image of the position and attitude imaging unit to be low. That is, in this case, the region determination unit 113 does not transmit the image of the position and attitude imaging unit, and determines only the image of the real object imaging unit as the image region to be transmitted by the transmission unit 106. However, since the image of the real object imaging unit has the viewing angle and imaging parameters set for detecting the real object, if the feature points and marker features used in the SLAM process are detected from the image, the stability of the position and attitude may decrease slightly.

[0098] For example, the region determination unit 113 may also acquire the priority set by the observer. Also in this example, it is assumed that the imaging unit 101 does not have an attitude sensor, and the imaging unit 101 is provided with a right imaging unit, a left imaging unit, a position and attitude imaging unit, and a real object imaging unit. Here, for example, when the observer desires the estimation accuracy of the position and attitude of the imaging unit 101 rather than detecting the real object to be detected, it is considered that the priority for the features of the real object to be detected may be lowered. Thus, when the observer makes a setting to lower the priority for the features of the real object to be detected, the region determination unit 113 determines the image region to be transmitted by the transmission unit 103 based on the lowered priority of the features of the real object to be detected. That is, in this case, the region determination unit 113 determines, for example, the image captured by the position and attitude imaging unit as the image region to be transmitted by the transmission unit 103, and determines not to transmit the image captured by the real object imaging unit. When the image for real object detection is not transmitted from the transmission unit 103 as in this example, the detection unit 112 may detect the features of the real object to be detected from, for example, the image for position and attitude calculation.

[0099] In addition, in Modification 7, when the detection unit 112 can detect the feature points and marker features used in the SLAM process, the estimation unit 114 estimates the position and orientation of the imaging unit 101 from those features. On the other hand, when the detection unit 112 cannot detect the features, the estimation unit 114 may estimate the position and orientation from, for example, an attitude sensor provided in the imaging unit 101.

[0100] As described above, in Modification 7, when the amount of transmission data of the transmission unit 103 increases, only the images selected according to the priority of feature detection are determined as the image areas to be transmitted from among the multiple images captured by the multiple imaging units provided in the imaging unit 101. Also, for example, when there is an alternative function for the purpose of detecting features (in this case, feature detection for estimating the position and orientation) as in the case where the imaging unit 101 is equipped with an attitude sensor, the priority of the features corresponding to the alternative function may be set to determine the image area to be transmitted. Further, the priority of feature detection may be set by the observer, and the image area to be transmitted may be determined according to the priority set by the observer.

[0101] As described above, according to Modification 7, when the amount of transmission data increases, the priority is set for the features to be detected, and the image areas used for detecting the features with low priority are not transmitted, thereby reducing the transmission amount and enabling the reduction of latency.

[0102] <Modification 8 of the First Embodiment> In Modifications 4 to 7 of the above-described First Embodiment, examples were given for cases where, as a result of predicting the positions of features used in the SLAM process or the positions of features of real objects of specific detection targets, the transmission area becomes the entire image or the data volume increases. Here, for example, when no features are detected from the image of the current frame, the area determination unit 113 cannot predict the positions of features in the next frame, and as a result, the image area to be transmitted by the transmission unit 103 cannot be determined. In this case, it is possible to transmit all of the plurality of images captured by the plurality of imaging units provided in the imaging unit 101. However, if this is done, the amount of transmitted data increases, the latency cannot be reduced, and the frame rate decreases due to an increase in the data transmission time. Also, for example, even when features are detected from the image of the current frame, the same problem occurs when the prediction of the positions of features for the next frame fails due to some factor.

[0103] In Modification 8, for example, a case will be described in which the detection unit 112 cannot detect features and thus cannot predict the positions of features for the next frame. Hereinafter, descriptions of parts common to the First Embodiment will be omitted as appropriate, and differences will be mainly described. In the case of the information processing system according to Modification 8, the operations of the detection unit 112 and the area determination unit 113 of the information processing apparatus 110 are different from those of the First Embodiment.

[0104] The area determination unit 113 in Modification 8 determines, as the image area to be transmitted, only the images that are essential for transmission in the information processing system or only the images that are minimally required for feature detection among the plurality of images captured by the plurality of imaging units of the imaging unit 101. For example, when the detection unit 112 cannot detect the features used in the SLAM process, the area determination unit 113 determines, as the image area to be transmitted by the transmission unit 103, only the images of the right imaging unit and the left imaging unit among the plurality of images of the imaging unit 101. Also, for example, when the detection unit 112 cannot detect the features of the real object of the detection target, the area determination unit 113 similarly determines, as the image area to be transmitted by the transmission unit 103, only the images of the right imaging unit and the left imaging unit.

[0105] For example, when providing a virtual space to an observer in an information processing system, the images of the right imaging unit and the left imaging unit are not transmitted. On the other hand, even if the entire image of the imaging unit for position and orientation, which is essential for feature detection, is transmitted, the amount of transmitted data is not considered to increase significantly. Therefore, when providing a virtual space to an observer in an information processing system, the region determination unit 113 determines that only the image of the imaging unit for position and orientation is the image region to be transmitted from the transmission unit 103. For example, when providing a virtual space to an observer in an information processing system, the region determination unit 113 may also determine that only the image of the imaging unit for real objects is the image region to be transmitted from the transmission unit 103.

[0106] As described above, when the detection unit 112 fails to detect a feature, the region determination unit 113 determines to transmit only the image that is essential for transmission in the information processing system or the image that is minimally required for feature detection. Here, as described above, the detection unit 112 detects features, markers, or features of a specific real object to be detected used in the SLAM process. Therefore, the region determination unit 113 of the modified example 8 determines which of the plurality of images captured by the imaging unit 101 to transmit according to the detection result of the features acquired from the detection unit 112.

[0107] For example, a case where the detection unit 112 can detect the features used in the SLAM process but cannot detect the features of the real object to be detected will be described. In this example, when the information processing system of the present embodiment provides a composite reality space to an observer, the region determination unit 113 determines that only the image for which the feature can be predicted among the images of the right imaging unit, the left imaging unit, and the imaging unit for position and orientation is the image region to be transmitted by the transmission unit 103. On the other hand, when the information processing system provides a virtual space to an observer, the region determination unit 113 determines that the entire image of the imaging unit for position and orientation is the image region to be transmitted by the transmission unit 103.

[0108] For example, in the detection unit 112, a case will be described where the features used in the SLAM process cannot be detected, but the features of the real object to be detected can be detected. In this example, when the information processing system provides the composite reality space to the observer, the area determination unit 113 determines, among the images of the right imaging unit, the left imaging unit, and the real object imaging unit, only the image for which the features could be predicted as the image area to be transmitted by the transmission unit 103. On the other hand, when the information processing system provides the virtual space to the observer, the area determination unit 113 determines the entire image of the real object imaging unit as the image area to be transmitted by the transmission unit 103.

[0109] For example, in the detection unit 112, a case will be described where neither the features used in the SLAM process nor the features of the real object to be detected can be detected. In this example, when the information processing system provides the composite reality space to the observer, the area determination unit 113 determines the images of the right imaging unit and the left imaging unit as the image area to be transmitted by the transmission unit 103. On the other hand, when the information processing system provides the virtual space to the observer, the area determination unit 113 determines either the entire image of the position and orientation imaging unit or the entire image of the real object imaging unit as the image area to be transmitted by the transmission unit 103.

[0110] Here, in Modification 8 as well, an image suitable for detecting the features with a high priority as described in the aforementioned Modification 7 may be selected, or an image pre-registered by the observer may be selected. And when features can be detected from the image of the next frame, the area determination unit 113 may predict, as described above, the area where the features are present in the image suitable for detecting each feature in the next frame. For example, when neither the features used in the SLAM process by the detection unit 112 nor the features of the real object to be detected can be detected, the area determination unit 113 determines the images of the right imaging unit and the left imaging unit as the image area to be transmitted by the transmission unit 103. As a result, when features are detected in the next frame, the area determination unit 113 predicts whether there are features used in the SLAM process in the image of the position and orientation imaging unit and predicts whether there are features of the real object in the image of the real object imaging unit, and determines the image area to be transmitted.

[0111] Then, the detection unit 112 of Modification 8 detects features used in the SLAM process and features of the real object to be detected from the image transmitted from the transmission unit 103. However, in this case, since it is necessary to detect features from an image in which appropriate viewing angles and imaging parameters for detecting each feature are not set, the stability of feature detection may decrease.

[0112] As described above, in Modification 8, when the detection unit 112 fails to detect features, the region determination unit 113 determines, as the image region to be transmitted, only the images that are essential for transmission in the information processing system or only the images that are minimally required for feature detection. Thus, also in the case of Modification 8, features can be detected without increasing the transmission amount.

[0113] <Second Embodiment> In the first embodiment described above, an example of reducing latency by reducing the data transmission amount from the HMD 100 to the information processing apparatus 110 was explained. In contrast, in the second embodiment, in the information processing apparatus, only a partial image region of the image received from the HMD is stored in the storage unit for calculation, and the image stored in the storage unit for calculation is transmitted to the detection unit. That is, in the second embodiment, an example of reducing latency by reducing the data transmission amount and the processing amount within the information processing apparatus will be explained. Hereinafter, the description of the parts common to the first embodiment will be omitted as appropriate, and the differences will be mainly described.

[0114] FIG. 10 is a diagram showing a configuration example of an information processing system according to the second embodiment. The information processing system according to the second embodiment includes an HMD 1010 and an information processing apparatus 1020. The HMD 1010 and the information processing apparatus 1020 are connected so as to be able to communicate with each other by wire, wirelessly, or both wire and wireless. In the case of the second embodiment, the transmission unit 1003 of the HMD 1010 transmits the captured image of the imaging unit 101 to the information processing apparatus 1020, and the reception unit 1004 of the HMD 1010 sends the image of the composite reality space or virtual space transmitted from the information processing apparatus 1020 to the display unit 102.

[0115] In the case of the second embodiment, the receiving unit 1021 of the information processing apparatus 1020 temporarily stores the image transmitted from the HMD 1010 in an internal buffer memory or the like. Here, the receiving unit 1021 of the present embodiment copies (writes) a part of the image area determined by the same prediction process as in the first embodiment by the area determination unit 113 among the images sent from the HMD 1010 to the calculation storage unit 1022. That is, in the case of the second embodiment, the area determination unit 113 requests the receiving unit 1021 to copy the image area determined by the prediction process to the calculation storage unit 1022. Then, the image copied to the calculation storage unit 1022 is read out by the access from the detection unit 112 and transmitted to the detection unit 112. Thus, also in the case of the second embodiment, among the images captured by the imaging unit 101 of the HMD 1010, a part of the image area determined based on the result of the same prediction process by the area determination unit 113 is transmitted to the detection unit 112.

[0116] Hereinafter, the operations of the respective functional units of the information processing apparatus 110 according to the second embodiment will be described according to the flowchart of FIG. 11. Note that in the HMD 1010 of the second embodiment, since the captured image of the imaging unit 101 is only transmitted from the transmitting unit 1003 and the image sent from the information processing apparatus 1020 is received and displayed by the receiving unit 1004, the flowchart of the HMD 1010 is omitted.

[0117] In S1101, the receiving unit 1021 receives the image transmitted from the HMD 1010, and stores only the image area determined by the prediction process by the area determination unit 113 among the received images in the calculation storage unit 1022. Next, in S312, the detection unit 112 accesses the calculation storage unit 1022 to acquire an image, and detects features from the image. In the second embodiment, since only the image area determined based on the result of the prediction process by the area determination unit 113 is written to the calculation storage unit 1022, the writing time (transferred time) can be shortened compared to the case where all images are written.

[0118] In the processes from S312 to S317 in FIG. 11, since the processes are substantially the same as those from S312 to S317 in FIG. 3(B) described above except that the image stored in the calculation storage unit 1022 is used, the descriptions thereof are omitted. In the flowchart of FIG. 11, when it is determined that no instruction to end the process is input at S317, the process of the information processing apparatus 1020 returns to S1101, and the processes after S1101 are performed.

[0119] Thus, in the case of the information processing system of the second embodiment, the processes related to the transmission of the image area that were performed by the transmission unit 103 of the HMD100 in the first embodiment are performed in the reception unit 1021 and the calculation storage unit 1022 of the information processing apparatus 1020 in this embodiment.

[0120] For example, in the information processing system of the second embodiment, when providing a virtual space to an observer, the transmission unit 1003 of the HMD1010 does not transmit the images of the right imaging unit and the left imaging unit of the imaging unit 101, but only transmits the image captured by the position and orientation imaging unit. At this time, the reception unit 1021 of the information processing apparatus 1020 copies only the image area determined by the prediction process in the area determination unit 113 for the image captured by the position and orientation imaging unit to the calculation storage unit 1022. As a result, only that image area is sent to the detection unit 112, the transmission amount in the information processing apparatus 1020 is reduced, and the processing amount in the detection unit 112 is also reduced, enabling a reduction in latency.

[0121] Also, for example, in the second embodiment, when providing a mixed reality space to an observer, the images of the right imaging unit and the left imaging unit and the image of the position and orientation imaging unit are transmitted from the transmission unit 1003 of the HMD1010. At this time, the reception unit 1021 of the information processing apparatus 1020 copies only a part of the image area determined by the prediction process of the area determination unit 113 for the image of the position and orientation imaging unit to the calculation storage unit 1022. As a result, the transmission amount in the information processing apparatus 1020 is reduced, and the processing amount in the detection unit 112 is also reduced, enabling a reduction in latency.

[0122] Also in the case of the second embodiment, similar to the example of the first embodiment, the region determination unit 113 predicts, based on the information of one or more features detected by the detection unit 112, where those features will be located in the image of the next frame sent from the imaging unit 101. Also in the case of the second embodiment, when the HMD 1010 is equipped with various sensors such as an attitude sensor, the region determination unit 113 may predict the position of the feature in the next frame based on the sensor information. Then, the region determination unit 113 requests the reception unit 1021 to copy only the image region determined based on the predicted position of the feature to the calculation storage unit 1022. Also in the case of the second embodiment, the region determination unit 113 may appropriately change the size of the image region based on the result of comparison between the difference between the position of the feature detected from the image of the N-th frame and the position of the feature predicted in the (N + 1)-th frame and a predetermined difference threshold value.

[0123] As described above, also in the case of the information processing apparatus 1020 of the second embodiment, by transmitting only the image region including the position of the feature predicted in the next frame to the detection unit 112, the amount of data sent to the detection unit 112 can be reduced, and the latency can be reduced.

[0124] Note that also in the case of the second embodiment, the same processing regarding the transmission of the image region as described in Modification Examples 1 to 8 of the first embodiment can be performed. For example, similar to Modification Example 1 described above, the detection unit 112 may detect a marker with a known size as a feature, and the region determination unit 113 may determine the image region to be copied to the calculation storage unit 1022 based on the predicted position of the marker feature.

[0125] Also for example, similar to Modification Example 2 described above, the detection unit 112 may detect a real object of a specific detection target as a feature, and the region determination unit 113 may determine the image region to be copied to the calculation storage unit 1022 based on the predicted position of the real object of the detection target.

[0126] Also, for example, similar to the above-described Modification 3, when the predicted position of the feature is near the edge of the image area, the area determination unit 113 may determine a range larger than the area determined based on the predicted feature as the image area that the receiving unit 1021 copies to the calculation storage unit 1022.

[0127] Also, for example, similar to the above-described Modification 4, the area determination unit 113 may determine, as the image area that the receiving unit 1021 copies to the calculation storage unit 1022, a divided area in which the number of feature points counted for each divided area is equal to or greater than a numerical threshold value.

[0128] Also, for example, similar to the above-described Modification 5, for areas where stereo measurement cannot be performed, the area determination unit 113 may not determine them as the image area that the receiving unit 1021 copies to the calculation storage unit 1022.

[0129] Also, for example, similar to the above-described Modification 6, the area determination unit 113 may determine the frequency component of the image and the frequency area to be transmitted. In the case of this example, for example, it is sufficient if the receiving unit 1021 has a function of converting the image into frequency components.

[0130] Also, for example, similar to the above-described Modification 7, among the plurality of images captured by the imaging unit 101, the area determination unit 113 may determine, as the image area that the receiving unit 1021 copies to the calculation storage unit 1022, the image used for detecting a feature with a high detection priority.

[0131] Also, for example, similar to the above-described Modification 8, the area determination unit 113 may determine, as the image area that the receiving unit 1021 copies to the calculation storage unit 1022, only the images essential for the information processing system or only the images minimally required for feature detection.

[0132] <Third Embodiment> Next, as a third embodiment, an example will be described in which, assuming an information processing system having two information processing apparatuses, the amount of transmission data when transmitting an image between the two information processing apparatuses is reduced to reduce latency. Hereinafter, the differences from the second embodiment will be mainly described. In the following description, unless otherwise specified, it is assumed to be the same as the second embodiment.

[0133] FIG. 12 is a diagram showing a configuration example of an information processing system according to the third embodiment. As shown in FIG. 12, the information processing system according to the third embodiment includes an HMD 1200, a first information processing apparatus 1210 that directly controls the HMD 1200, and a second information processing apparatus 1220 connected to the first information processing apparatus 1210. Also, the HMD 1200 and the first information processing apparatus 1210 are connected so as to be able to communicate with each other by wire, wirelessly, or both, and the first information processing apparatus 1210 and the second information processing apparatus 1220 are similarly connected. Since the HMD 1200 is the same as the HMD 1010 of the second embodiment, its description is omitted.

[0134] Hereinafter, the operations of the functional units of the first information processing apparatus 1210 and the second information processing apparatus 1220 according to the third embodiment will be described according to the flowchart of FIG. 13. Note that since the HMD 1200 of the third embodiment is generally the same as the HMD 1010 of the second embodiment, its flowchart is omitted. FIG. 13(A) is a flowchart of the first information processing apparatus 1210, and FIG. 13(B) is a flowchart of the second information processing apparatus 1220.

[0135] First, the flowchart of FIG. 13(A) in the first information processing apparatus 1210 will be described. In S1301, the reception unit 1211 of the first information processing apparatus 1210 receives the image captured by the imaging unit 101 of the HMD 100 and transmitted from the transmission unit 1003, and temporarily stores it in an internal buffer memory.

[0136] Next, in S1302, the image processing unit 1212 performs appropriate image processing such as white balance adjustment and image distortion correction on the image temporarily stored in the internal buffer memory of the reception unit 1211, and then sends it to the transmission unit 1213. Also, in the case of the third embodiment, a transmission request to transmit only the image area determined by the prediction process similar to the above in the area determination unit 113 of the second information processing apparatus 1220 is transmitted to the first information processing apparatus 1210 via the transmission unit 1226. Then, the transmission request is received by the reception unit 1214 of the first information processing apparatus 1210 and sent to the image processing unit 1212. Then, the image processing unit 1212 sends only a part of the image area determined by the prediction process in the area determination unit 113 of the second information processing apparatus 1220 among the images captured by the imaging unit 101 of the HMD 1200 to the transmission unit 1213. Therefore, from the transmission unit 1213, the image area is transmitted to the second information processing apparatus 1220. Thus, in the third embodiment, the transmission amount of the image transmitted from the first information processing apparatus 1210 to the second information processing apparatus 1220 can be reduced, and the latency can be reduced.

[0137] Also, in S1303, the reception unit 1214 receives the image of the composite reality space or the image of the virtual space generated by the second information processing apparatus 1220. Then, the image processing unit 1212 generates an image for display on the display unit 102 of the HMD 1200 from the received image, and sends it from the transmission unit 1215 to the HMD 1200.

[0138] Thereafter, in S1304, the second information processing apparatus 1210 determines whether an instruction to end the process has been input from the user. If an instruction to end the process has been input, the process of the flowchart in Fig. 11(A) ends. On the other hand, if an instruction to end the process has not been input, the process of the second information processing apparatus 1210 returns to S1301, and the processes after S1301 are performed.

[0139] Next, the flowchart in Fig. 13(B) in the second information processing apparatus 1220 will be described. In S1311, the receiving unit 1221 of the second information processing apparatus 1220 receives the image transmitted from the first information processing apparatus 1210, and stores the received image in the calculation storage unit 1222. The processing from S312 to S317 in FIG. 13(B) is substantially the same as the processing performed by the information processing apparatus 1020 of the second embodiment described above. However, in the case of the second information processing apparatus 1220, a transmission request for the image area determined by the area determination unit 113 based on the prediction process is sent from the transmission unit 1226 to the first information processing apparatus 1210. Further, in the flowchart of FIG. 13(B), when it is determined that no instruction to end the process is input at S317, the process of the information processing apparatus 1220 returns to S1311, and the processes after S1311 are performed.

[0140] As described above, in the case of the information processing system of the third embodiment, the processes related to the transmission of the image area performed by the receiving unit 1021 and the calculation storage unit 1022 of the information processing apparatus 1020 of the second embodiment are performed in the image processing unit 1212 of the first information processing apparatus 1210.

[0141] In the information processing system of the third embodiment, for example, when providing a virtual space to an observer, the HMD1200 transmission unit 1003 does not transmit the images of the right imaging unit and the left imaging unit of the imaging unit 101, but only transmits the image captured by the position and orientation imaging unit. At this time, the image processing unit 1212 of the first information processing apparatus 1210 causes only the image area determined by the prediction process in the area determination unit 113 to be transmitted from the transmission unit 1213 to the second information processing apparatus 1220 for the image captured by the position and orientation imaging unit.

[0142] Also, in the third embodiment, for example, when providing a mixed reality space to an observer, the images of the right imaging unit and the left imaging unit and the image of the position and orientation imaging unit are transmitted from the transmission unit 1003 of the HMD1200. At this time, the image processing unit 1212 of the information processing apparatus 1210 causes only a part of the image area determined by the prediction process of the area determination unit 113 to be transmitted from the transmission unit 1213 to the second information processing apparatus 1220 for the image of the position and orientation imaging unit.

[0143] Also, in the case of the third embodiment, when the HMD 1200 is equipped with various sensors such as a posture sensor, the region determination unit 113 may predict the position of the feature in the next frame based on the sensor information. Then, the region determination unit 113 transmits a transmission request only for the image region determined based on the predicted position of the feature to the first information processing device 1210. Also, in the case of the third embodiment, the region determination unit 113 may appropriately change the size of the image region based on the result of comparison between the difference between the position of the feature detected from the image of the Nth frame and the position of the feature predicted in the (N + 1)th frame and a predetermined difference threshold value.

[0144] As described above, according to the third embodiment, even when transmitting an image between two information processing devices, latency can be reduced by reducing the amount of data transmitted.

[0145] Note that, also in the case of the third embodiment, the same processing regarding the transmission of the image region as described in Modification Examples 1 to 8 of the first embodiment can be performed. For example, similar to Modification Example 1, a marker with a known size may be detected as a feature, and the region determination unit 113 may determine the image region that the first information processing device 1210 transmits to the second information processing device 1220 based on the predicted position of the feature of the marker.

[0146] Also, for example, similar to Modification Example 2, a real object of a specific detection target may be detected as a feature, and the region determination unit 113 may determine the image region that the first information processing device 1210 transmits to the second information processing device 1220 based on the predicted position of the feature of the detected real object of the detection target.

[0147] Also, for example, similar to Modification Example 3, when the predicted position of the feature is near the edge of the image region, the region determination unit 113 may determine a range larger than the region determined based on the predicted feature as the image region that the first information processing device 1210 transmits to the second information processing device 1220.

[0148] Also, for example, similar to Modification 4, the region determination unit 113 may determine, as the image region to be transmitted from the first information processing apparatus 1210 to the second information processing apparatus 1220, a divided region in which the number of feature points counted for each divided region is equal to or greater than a numerical threshold value.

[0149] Also, for example, similar to Modification 5, the region determination unit 113 may determine, as the image region not to be transmitted from the first information processing apparatus 1210 to the second information processing apparatus 1220, a region where stereo measurement cannot be performed.

[0150] Also, for example, similar to Modification 6, the region determination unit 113 may determine the frequency components of the image and the frequency region to be transmitted. In the case of this example, for example, the image processing unit 1212 of the first information processing apparatus 1210 converts the image into frequency components.

[0151] Also, for example, similar to Modification 7, the region determination unit 113 may determine, as the image region to be transmitted from the first information processing apparatus 1210 to the second information processing apparatus 1220, an image used for detecting features with a high detection priority among a plurality of images captured by the imaging unit 101.

[0152] Also, for example, similar to Modification 8, the region determination unit 113 may determine, as the image region to be transmitted from the first information processing apparatus 1210 to the second information processing apparatus 1220, only the images essential for the information processing system or only the images minimally required for feature detection.

[0153] In the above-described third embodiment, the first information processing apparatus 1210 and the second information processing apparatus 1220 are separate apparatuses, and the latency is reduced by reducing the amount of image transmission between these apparatuses. However, the present invention is not limited to this example. For example, the first information processing apparatus 1210 may be included in the housing of the HMD 1200, or the first information processing apparatus 1210 and the second information processing apparatus 1220 may be included in the same housing. Also, in the above-described first and second embodiments, examples in which the HMD and the information processing apparatus have separate configurations have been given. However, the information processing apparatus may be included in the housing of the HMD.

[0154] The present invention can also be realized by supplying a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be realized by a circuit (for example, ASIC) that implements one or more functions. The above-described embodiments are merely examples of implementation when carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from its technical idea or its main features.

[0155] The disclosure of the present embodiment includes the following configurations, methods, and programs. (Configuration 1) Detection means for detecting features from an image obtained by the imaging device capturing the real space, Region determination means for predicting a region having the feature in an image to be next captured by the imaging device based on the detected feature, and determining, as a region to be transmitted to the detection means, only the region having the predicted feature among the image next captured by the imaging device. An information processing apparatus characterized by including the above. (Configuration 2) The information processing apparatus according to Configuration 1, further including generation means for generating an image of a virtual space based on a detection result of the feature by the detection means. (Configuration 3) The information processing apparatus according to Configuration 2, further including acquisition means for acquiring a position and orientation of the imaging device based on the feature detected by the detection means, wherein the generation means generates an image of the virtual space according to the position and orientation of the imaging device. (Configuration 4) The information processing apparatus according to Configuration 2 or 3, wherein the generation means generates a composite image by combining the image of the virtual space and the image captured by the imaging device. (Configuration 5) transmission means for requesting transmission of the region determined by the region determination means among the images to be captured next by the imaging device; receiving means for receiving, among the images of the real space captured next by the imaging device, the image of the region transmitted from the imaging device in response to the transmission request; and has The information processing apparatus according to any one of Configurations 1 to 4, wherein the receiving means sends the image of the region transmitted from the imaging device in response to the transmission request to the detection means. (Configuration 6) having receiving means for receiving the image of the real space captured by the imaging device; The information processing apparatus according to any one of Configurations 1 to 4, wherein the receiving means sends only the image of the region determined by the region determination means among the images of the real space captured next by the imaging device to the detection means. (Configuration 7) receiving means for receiving, from another information processing apparatus that performs image processing on the image of the real space captured by the imaging device, the image after the image processing; transmission means for requesting transmission of the image of the region determined by the region determination means among the images of the real space captured next by the imaging device; and has The information processing apparatus according to any one of Configurations 1 to 4, wherein the receiving means sends the image of the region transmitted from the other information processing apparatus in response to the transmission request to the detection means. (Configuration 8) The information processing apparatus according to any one of Configurations 1 to 7, wherein the region determination means predicts the region having the feature with respect to the image to be captured next by the imaging device using the information detected by the attitude sensor provided in the imaging device. (Configuration 9) The detection means detects a specific real object existing in the real space as the feature; The region determination means predicts the region where the real object is located with respect to the image to be captured next by the imaging device, and determines, as the region to be transmitted to the detection means, only the region where the predicted real object is located among the images captured next by the imaging device. The information processing apparatus according to any one of Configurations 1 to 8. (Configuration 10) The information processing apparatus according to Configuration 9, wherein the specific real object is a hand. (Configuration 11) The region determination means also uses the information detected by a posture sensor provided in a specific real object existing in the real space to predict the region where the real object is located with respect to the image to be captured next by the imaging device, and determines, as the region to be transmitted to the detection means, only the region where the predicted real object is located among the images captured next by the imaging device. The information processing apparatus according to any one of Configurations 1 to 10. (Configuration 12) The detection means detects a predetermined marker provided in the real space as the feature, The region determination means predicts the region where the marker is located with respect to the image to be captured next by the imaging device, and determines, as the region to be transmitted to the detection means, only the region where the predicted marker is located among the images captured next by the imaging device. The information processing apparatus according to any one of Configurations 1 to 11. (Configuration 13) The region determination means determines, as the region to be transmitted to the detection means, only the divided region having the predicted feature among a plurality of divided regions obtained by dividing the image to be captured next by the imaging device. The information processing apparatus according to any one of Configurations 1 to 7. (Configuration 14) The region determination means determines, as the region to be transmitted to the detection means, only the divided region having the predicted feature and having the number of features equal to or more than a predetermined threshold among the divided regions having the predicted feature. The information processing apparatus according to Configuration 13. (Configuration 15) The area determination means determines the area to be transmitted to the detection means to be large according to the difference between the position of the predicted feature and the position of the feature detected by the detection means from the image captured next by the imaging device, and is the information processing apparatus according to any one of Configurations 1 to 14. (Configuration 16) When the feature is at the edge of the area, the area determination means determines the area to be transmitted to the detection means to be large according to the setting of feature detection by the detection means, and is the information processing apparatus according to any one of Configurations 1 to 15. (Configuration 17) The area determination means determines the area to be transmitted to the detection means to be large according to the setting of the filter used by the detection means for detecting the feature, and is the information processing apparatus according to Configuration 16. (Configuration 18) When the imaging device includes two imaging units for stereo measurement, the area determination means predicts the overlapping area of the two images to be captured next by the two imaging units, and determines the predicted overlapping area as the area to be transmitted to the detection means, and is the information processing apparatus according to any one of Configurations 1 to 17. (Configuration 19) Among the plurality of frequency regions obtained by converting the image into frequency components, the area determination means determines only the frequency region corresponding to the feature of the predicted region as the area to be transmitted to the detection means, and is the information processing apparatus according to any one of Configurations 1 to 7. (Configuration 20) When the detection means detects a specific real object existing in the real space or a predetermined marker provided in the real space as the feature, the area determination means determines only the low-frequency region among the plurality of frequency regions as the area to be transmitted to the detection means, and is the information processing apparatus according to Configuration 19. (Configuration 21) When the area determination means detects features used by the detection means in the SLAM (Simultaneous Localization and Mapping) process, the information processing apparatus according to Configuration 19 or 20 is characterized in that only the high-frequency area among the plurality of frequency areas is determined as the area to be transmitted to the detection means. (Configuration 22) Among the plurality of images captured by the plurality of imaging units provided in the imaging device, the information processing apparatus according to any one of Configurations 1 to 7 is characterized in that an image selected according to the priority set for the features detected by the detection means is determined as the area to be transmitted to the detection means. (Configuration 23) Among the plurality of images captured by the plurality of imaging units provided in the imaging device, the information processing apparatus according to any one of Configurations 1 to 7 is characterized in that an image selected according to the priority set according to the configuration provided in the imaging device is determined as the area to be transmitted to the detection means. (Configuration 24) Among the plurality of images captured by the plurality of imaging units provided in the imaging device, the information processing apparatus according to any one of Configurations 1 to 7 is characterized in that an image selected according to the priority set by the user is determined as the area to be transmitted to the detection means. (Configuration 25) When the imaging device has two or more imaging units, the information processing apparatus according to any one of Configurations 1 to 7 is characterized in that when the detection means cannot detect the feature from the image captured by any one of the imaging units, an image captured by another imaging unit is determined as the area to be transmitted to the detection means. (Configuration 26) The detection means detects two or more different features, When any one of the two or more features is detected, the area determination means predicts, based on the detected feature, an area having the feature in the image to be next captured by the imaging device. The information processing apparatus according to any one of Configurations 1 to 7. (Method 1) A detection step of detecting a feature from an image captured by an imaging device of a real space; Based on the detected feature, an area having the feature in the image to be next captured by the imaging device is predicted, and only the area having the predicted feature in the image next captured by the imaging device is determined as an area to be transmitted to the detection step. An area determination step; An information processing method characterized by comprising: (Program 1) A program for causing a computer to function as the information processing apparatus according to any one of Configurations 1 to 26.

Explanation of Signs

[0156] 100, 1010, 1200: HMD, 110, 1020: Information processing apparatus, 1210: First information processing apparatus, 1220: Second information processing apparatus, 103: Transmission unit, 112: Detection unit, 113: Area determination unit, 114: Estimation unit, 115: Generation unit

Claims

1. detection means for detecting features from an image obtained by the imaging device capturing the real space; region determination means for predicting, based on the detected features, a region having the features in an image to be next captured by the imaging device, and determining, as a region to be transmitted to the detection means, only the region having the predicted features among the images next captured by the imaging device; An information processing apparatus, characterized by comprising the above.

2. The information processing apparatus according to claim 1, further comprising generation means for generating an image of a virtual space based on a detection result of the features by the detection means.

3. acquisition means for acquiring the position and orientation of the imaging device based on the features detected by the detection means; The information processing apparatus according to claim 2, wherein the generation means generates an image of the virtual space according to the position and orientation of the imaging device.

4. The information processing apparatus according to claim 3, wherein the generation means generates a composite image by combining the image of the virtual space and the image captured by the imaging device.

5. transmission means for requesting transmission of the region determined by the region determination means among the images to be next captured by the imaging device; reception means for receiving, among the images of the real space next captured by the imaging device, the image of the region transmitted from the imaging device in response to the transmission request; comprising; The information processing apparatus according to claim 1, wherein the reception means sends out the image of the region transmitted from the imaging device in response to the transmission request to the detection means.

6. reception means for receiving the image of the real space captured by the imaging device; The information processing apparatus according to claim 1, wherein the reception means sends out only the image of the region determined by the region determination means among the images of the real space next captured by the imaging device to the detection means.

7. reception means for receiving, from another information processing apparatus that performs image processing on the image of the real space captured by the imaging device, the image after the image processing; transmission means for requesting transmission of the image of the region determined by the region determination means among the images of the real space next captured by the imaging device; comprising; The information processing apparatus according to claim 1, wherein the receiving means sends out an image of the area transmitted from the other information processing apparatus in response to the request for the transmission to the detecting means.

8. The information processing apparatus according to any one of claims 1 to 7, wherein the area determining means predicts an area having the feature with respect to an image to be next captured by the imaging apparatus, using information detected by an attitude sensor provided in the imaging apparatus.

9. The detecting means detects a specific real object existing in the real space as the feature, The information processing apparatus according to any one of claims 1 to 7, wherein the area determining means predicts an area where the real object is located with respect to an image to be next captured by the imaging apparatus, and determines, as an area to be sent to the detecting means, only the area where the predicted real object is located among the images captured by the imaging apparatus next.

10. The information processing apparatus according to claim 9, wherein the specific real object is a hand.

11. The information processing apparatus according to any one of claims 1 to 7, wherein the area determining means predicts an area where the real object is located with respect to an image to be next captured by the imaging apparatus, using information detected by an attitude sensor provided in a specific real object existing in the real space, and determines, as an area to be sent to the detecting means, only the area where the predicted real object is located among the images captured by the imaging apparatus next.

12. The detecting means detects a predetermined marker provided in the real space as the feature, The information processing apparatus according to any one of claims 1 to 7, wherein the area determining means predicts an area where the marker is located with respect to an image to be next captured by the imaging apparatus, and determines, as an area to be sent to the detecting means, only the area where the predicted marker is located among the images captured by the imaging apparatus next.

13. The information processing apparatus according to any one of claims 1 to 7, wherein the area determining means determines, as an area to be sent to the detecting means, only a divided area having the predicted feature among a plurality of divided areas obtained by dividing an image to be next captured by the imaging apparatus.

14. The information processing apparatus according to claim 13, wherein the area determination means determines, as an area to be transmitted to the detection means, only a divided area in which the number of the features is equal to or greater than a predetermined threshold among the divided areas having the predicted features.

15. The information processing apparatus according to any one of claims 1 to 7, wherein the area determination means determines the area to be transmitted to the detection means to be larger according to a difference between a position of the predicted feature and a position of the feature detected by the detection means from an image that the imaging device images next.

16. The information processing apparatus according to any one of claims 1 to 7, wherein when the feature is at an edge of the area, the area determination means determines the area to be transmitted to the detection means to be larger according to a setting of feature detection by the detection means.

17. The information processing apparatus according to claim 16, wherein the area determination means determines the area to be transmitted to the detection means to be larger according to a setting of a filter used by the detection means for detecting the feature.

18. The information processing apparatus according to any one of claims 1 to 7, wherein when the imaging device includes two imaging units for stereo measurement, the area determination means predicts an overlapping area of two images that the two imaging units image next, and determines the predicted overlapping area as an area to be transmitted to the detection means.

19. The information processing apparatus according to any one of claims 1 to 7, wherein the area determination means determines, as an area to be transmitted to the detection means, only a frequency area corresponding to the features of the predicted area among a plurality of frequency areas obtained by converting the image into frequency components.

20. The information processing apparatus according to claim 19, wherein when the detection means detects a specific real object existing in the real space or a predetermined marker provided in the real space as the feature, the area determination means determines, as an area to be transmitted to the detection means, only a low-frequency area among the plurality of frequency areas.

21. The information processing apparatus according to claim 19, wherein when the detection means detects features used in the process of SLAM (Simultaneous Localization and Mapping), the area determination means determines only the high-frequency area among the plurality of frequency areas as the area to be transmitted to the detection means.

22. The information processing apparatus according to any one of claims 1 to 7, wherein the area determination means determines, as the area to be transmitted to the detection means, an image selected according to the priority set for the features detected by the detection means from among a plurality of images captured by a plurality of imaging units provided in the imaging apparatus.

23. The information processing apparatus according to any one of claims 1 to 7, wherein the area determination means determines, as the area to be transmitted to the detection means, an image selected according to the priority set according to the configuration of the imaging apparatus from among a plurality of images captured by a plurality of imaging units provided in the imaging apparatus.

24. The information processing apparatus according to any one of claims 1 to 7, wherein the area determination means determines, as the area to be transmitted to the detection means, an image selected according to the priority set by the user from among a plurality of images captured by a plurality of imaging units provided in the imaging apparatus.

25. The information processing apparatus according to any one of claims 1 to 7, wherein when the imaging apparatus has two or more imaging units, if the detection means cannot detect the feature from an image captured by any one of the imaging units, the area determination means determines an image captured by another imaging unit as the area to be transmitted to the detection means.

26. The detection means detects two or more different features, The information processing apparatus according to any one of claims 1 to 7, wherein when any one of the two or more features is detected, the area determination means predicts, based on the detected feature, the area where the feature exists in the image to be captured next by the imaging apparatus.

27. A detection step of detecting features from an image captured by the imaging apparatus in the real space, Based on the detected features, predict an area with the features in the image to be next captured by the imaging device, and determine, as an area to be transmitted to the detection step, only the area with the predicted features among the images next captured by the imaging device, in an area determination step. An information processing method characterized by comprising the above. **Claim 28** A computer A detection means for detecting features from an image obtained by an imaging device capturing a real space, Based on the detected features, predict an area with the features in the image to be next captured by the imaging device, and determine, as an area to be transmitted to the detection means, only the area with the predicted features among the images next captured by the imaging device, in an area determination means. A program for causing a computer to function as an information processing device having the above.

Images