Information processing device, control method for information processing device, and program

Abstract

To enable the observation of augmented reality images with a field of view that is appropriate to the observer's situation. [Solution] The information processing device 200 for generating a composite reality image to present to an observer includes: an acquisition unit 201 that acquires a real space image obtained by capturing a real space; a determination unit 204 that determines whether or not to widen the field of view of the composite reality image by the observer according to the observer's settings or the observer's composite reality environment; an extraction unit 205 that extracts a real space extract image from the real space image based on the determination result of the determination unit 204; and a generation unit 207 that generates a composite reality image to be displayed on the display unit 103 by combining the real space extract image and a virtual space image corresponding to the real space extract image.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for the information processing apparatus, and a program.

Background Art

[0002] In recent years, in the fields of design and manufacturing, there has been a demand for shortening the evaluation period and reducing costs using prototypes. For example, a Mixed Reality (MR) system for evaluating ease of assembly and maintainability using design (shape / design) data created by a CAD (Computer Aided Design) system has been introduced.

[0003] For example, head-mounted image display devices (Head Mounted Display: HMD) equipped with an MR system include a video see-through method and an optical see-through method. Here, the video see-through type HMD generates a composite reality image obtained by superimposing and drawing a virtual space image generated according to the position and orientation of the camera on a real space image captured by the camera as a display image. At this time, examples of the virtual space image include images such as virtual objects and character information drawn by computer graphics (CG). Thereafter, in the video see-through type HMD, the generated display image (composite reality image) is displayed on the display unit, and the observer observes the composite reality space by visually recognizing the display image displayed on the display unit with the observer's eyes. In this video see-through type HMD, in order to facilitate the optical axis coincidence of the captured image, which is a real space image, and the display image, which is a composite reality image, the imaging angle (imaging angle of view) for imaging may be wider than the display angle (display angle of view) displayed on the display unit. Further, since the captured image has a wider angle of view than the display image, it is possible to accurately detect feature points for a subject existing at the edge of the display image, and to accurately calculate the position and orientation of the camera.

[0004] Patent Document 1 describes a technique for cropping an image when generating a display image (mixed reality image) to be displayed on the display unit of a video see-through type HMD, such that the viewer perceives a wide field of view when observing with both eyes. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2018-23026 [Overview of the project] [Problems that the invention aims to solve]

[0006] However, in the technology described in Patent Document 1, when an observer views the augmented reality image displayed on the display unit with both eyes, the augmented reality image is displayed in such a way that the observer always perceives a wide field of view, which may cause the observer to feel uncomfortable. In other words, the technology described in Patent Document 1 has the problem that it is difficult to observe the augmented reality image with a field of view that is appropriate to the observer's situation.

[0007] This invention has been made in view of these problems, and aims to enable the observation of augmented reality images with a field of view that is appropriate to the observer's situation. [Means for solving the problem]

[0008] The present invention relates to an information processing device that generates a composite reality image to be presented to an observer, comprising: acquisition means for acquiring a real-space image obtained by imaging a real space; determination means for determining whether or not to widen the field of view of the composite reality image by the observer according to the observer's settings or the observer's composite reality environment; extraction means for extracting a real-space extract image from the real-space image based on the result of the determination by the determination means; and generation means for generating a composite reality image to be displayed on a display unit by combining the real-space extract image and a virtual space image corresponding to the real-space extract image. [Effects of the Invention]

[0009] According to the present invention, augmented reality images can be observed with a field of view that is appropriate to the observer's situation. [Brief explanation of the drawing]

[0010] [Figure 1A] This figure shows a first example of the schematic configuration of an information processing system according to the first embodiment. [Figure 1B] This figure shows an example of the left eye optical system and the right eye optical system included in the HMD according to the first embodiment. [Figure 2] This figure shows a second example of the schematic configuration of the information processing system according to the first embodiment. [Figure 3] This figure shows an example of the hardware configuration of the information processing device according to the first embodiment. [Figure 4] This flowchart shows an example of a processing procedure in the control method for the information processing device according to the first embodiment. [Figure 5] This figure shows a first embodiment and illustrates an example of the cropping range when the cropping unit extracts a real-space cropped image from a real-space image when the result of the determination unit's determination does not widen the observer's field of view. [Figure 6A] This figure shows a first embodiment and illustrates an example of the cropping range when the cropping unit extracts a real-space cropped image from a real-space image when the result of the determination unit's determination widens the observer's field of view. [Figure 6B] This figure shows a first embodiment, illustrating an example of a mixed reality image for the left eye, a mixed reality image for the right eye, and a mixed reality image that the observer sees when observing with both eyes, in a case where the determination result of the determination unit widens the observer's field of view. [Figure 7] This figure shows a modified example of the first embodiment, specifically a third modification, illustrating an example of the cropping range when the cropping unit extracts a real-space cropped image from a real-space image when the determination result of the determination unit widens the observer's field of view. [Figure 8] This figure shows an example of a schematic configuration of an information processing system according to the second embodiment. [Figure 9] It is a flowchart showing an example of a processing procedure in a control method of an information processing apparatus according to a second embodiment. [Figure 10] It shows a second embodiment, and is a diagram showing an example of a composite reality image for the left eye, a composite reality image for the right eye, and a composite reality image that can be seen when an observer observes with both eyes when the result of determination by a determination unit widens the field of view of the observer.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments (embodiments) for carrying out the present invention will be described with reference to the drawings.

[0012] (First Embodiment) First, the first embodiment will be described.

[0013] FIG. 1A is a diagram showing a first example of a schematic configuration of an information processing system 10 according to the first embodiment. Hereinafter, the information processing system 10 according to the first example of the first embodiment shown in FIG. 1A will be described as the "information processing system 10-1".

[0014] As shown in FIG. 1A, the information processing system 10-1 includes an HMD (Head Mounted Display) 100 and an information processing apparatus 200. A communication connection is established between the HMD 100 and the information processing apparatus 200 shown in FIG. 1A so that data communication can be performed between them by wire and / or wirelessly.

[0015] As shown in FIG. 1A, the HMD 100 includes an imaging unit 101, a measurement unit 102, and a display unit 103. An observer can observe a composite reality space through the display unit 103 of the HMD 100 by wearing the HMD 100 on his / her head so as to cover both of his / her eyes. In this embodiment, the HMD 100 is cited as an example of a head-mounted display device, but other types of head-mounted display devices may be applied. Further, in the present invention, the present invention is not limited to a head-mounted display device, and any display device that is browsed by an observer in order to allow the observer to observe a composite reality space, for example, other types of display devices such as a hand-held display device may be applied.

[0016] The imaging unit 101 generates a real-space image by imaging the real space.

[0017] The measurement unit 102 is fixedly attached to the HMD 100, and measures the position and orientation (hereinafter simply referred to as "position and orientation") of the measurement unit 102 itself (or may be interpreted as the HMD 100).

[0018] The display unit 103 displays a composite reality image, which is an image of the composite reality space transmitted from the information processing device 200.

[0019] In this embodiment, the imaging unit 101 may include a left-eye imaging unit provided corresponding to the left eye of the observer and a right-eye imaging unit provided corresponding to the right eye of the observer. Further, the display unit 103 may include a left-eye display unit that displays a composite reality image for the left eye of the observer and a right-eye display unit that displays a composite reality image for the right eye of the observer.

[0020] FIG. 1B is a diagram showing an example of a left-eye optical system 110L and a right-eye optical system 110R included in the HMD 100 according to the first embodiment.

[0021] As shown in Figure 1B, the left-eye optical system 110L is an optical system provided in accordance with the observer's left eye EL, and includes a left-eye imaging unit 101L and a left-eye display unit 103L. The left-eye imaging unit 101L is provided in accordance with the observer's left eye EL and captures a real-space image (a real-space image for the left eye) to be provided to the observer's left eye EL. The left-eye display unit 103L is provided in accordance with the observer's left eye EL and displays a composite reality image for the left eye.

[0022] Furthermore, as shown in Figure 1B, the right-eye optical system 110R is an optical system provided in accordance with the observer's right eye ER, and includes a right-eye imaging unit 101R and a right-eye display unit 103R. The right-eye imaging unit 101R is provided in accordance with the observer's right eye ER and captures a real-space image (a real-space image for the right eye) to be provided to the observer's right eye ER. The right-eye display unit 103R is provided in accordance with the observer's right eye ER and displays a composite reality image for the right eye.

[0023] In this embodiment, as shown in Figure 1B, the horizontal display field of view of the left eye display unit 103L and the right eye display unit 103R is set to 60 degrees. The left eye imaging unit 101L and the right eye imaging unit 101R each capture moving images of the real space as real-space images. The left eye imaging unit 101L and the right eye imaging unit 101R are mounted parallel to each other in the HMD 100 and sequentially transmit images of each frame constituting the moving image of the real space to the information processing device 200.

[0024] In this embodiment, it is necessary to maintain the relative positional relationship between the left eye imaging unit 101L and the right eye imaging unit 101R. Furthermore, it is also necessary to maintain the internal parameters of each left eye imaging unit 101L and right eye imaging unit 101R (the focal length, principal point, and field of view of each imaging unit). In this embodiment, as shown in Figure 1B, the horizontal imaging field of view of the left eye imaging unit 101L and the right eye imaging unit 101R is assumed to be 90 degrees each. In this case, since the imaging field of view of 90 degrees is wider than the display field of view of 60 degrees, it is necessary to crop a portion of the real-space image, which is the captured image, to match the display field of view.

[0025] Now, let's return to the explanation of Figure 1A. The information processing device 200 is a device that generates augmented reality images to be presented to an observer. As shown in Figure 1A, the information processing device 200 has an acquisition unit 201, an estimation unit 202, a setting unit 203, a determination unit 204, an extraction unit 205, a holding unit 206, and a generation unit 207.

[0026] The acquisition unit 201 is an acquisition means that acquires a real-space image obtained by imaging the real space from the imaging unit 101. In this embodiment, the acquisition unit 201 acquires, for example, a real-space image for the left eye captured by the left eye imaging unit 101L and a real-space image for the right eye captured by the right eye imaging unit 101R.

[0027] The estimation unit 202 is an estimation means for estimating the position and orientation of the imaging unit 101 in the HMD 100. In this embodiment, the estimation unit 202 estimates, for example, the position and orientation of the left eye imaging unit 101L and the right eye imaging unit 101R.

[0028] The setting unit 203 is a setting means that allows the observer to set whether or not they want to widen their field of view.

[0029] The determination unit 204 is a determination means that determines whether or not to widen the field of view of the mixed reality image by the observer, according to the observer settings or the observer's mixed reality environment set in the setting unit 203.

[0030] The extraction unit 205 is an extraction means that extracts a real-space extraction image from the real-space image acquired by the acquisition unit 201 based on the determination result of the determination unit 204. In this embodiment, the extraction unit 205 extracts, for example, a real-space extraction image for the left eye from the real-space image for the left eye acquired by the acquisition unit 201, and a real-space extraction image for the right eye from the real-space image for the right eye, based on the determination result of the determination unit 204.

[0031] The holding unit 206 is a holding means for holding CG (Computer Graphics) information for constructing a virtual space image.

[0032] The generation unit 207 is a generation means that generates a composite reality image to be displayed on the display unit 103 by combining a real-space cropped image obtained by the cropping unit 205 and a virtual space image generated using CG information from the holding unit 206 that corresponds to the real-space cropped image. In this embodiment, for example, the generation unit 207 generates a composite reality image for the left eye to be displayed on the left eye display unit 103L by combining a real-space cropped image for the left eye obtained by the cropping unit 205 and a virtual space image for the left eye that corresponds to the real-space cropped image for the left eye. Also in this embodiment, for example, the generation unit 207 generates a composite reality image for the right eye to be displayed on the right eye display unit 103R by combining a real-space cropped image for the right eye obtained by the cropping unit 205 and a virtual space image for the right eye that corresponds to the real-space cropped image for the right eye.

[0033] Figure 2 shows a second example of the schematic configuration of the information processing system 10 according to the first embodiment. Hereinafter, the information processing system 10 according to the second example of the first embodiment shown in Figure 2 will be referred to as "information processing system 10-2". In Figure 2, the same reference numerals are used for components that are the same as those shown in Figures 1A and 1B, and their detailed descriptions are omitted.

[0034] Figure 2 illustrates observer K wearing the HMD100 on their head so as to cover both of their eyes. In the example shown in Figure 2, the measurement unit 102 of the HMD100 functions as a receiver for the magnetic field sensor system and measures its own position and orientation (which may be interpreted as the HMD100 itself).

[0035] As shown in Figure 2, the information processing system 10-2 includes, in addition to the HMD 100 and information processing device 200 shown in Figure 1A, a magnetic field generator 310 and a controller 320.

[0036] The magnetic field generator 310 functions as a transmitter in the magnetic field sensor system, is fixedly positioned in a predetermined location in real space, and generates a magnetic field around itself.

[0037] The controller 320 controls the operation of the magnetic field generator 310. The operation of the controller 320 is controlled by the information processing device 200.

[0038] Furthermore, Figure 2 illustrates the world coordinate system 301 and the sensor coordinate system 302. The world coordinate system 301 is a coordinate system for aligning the real space and the virtual space, and as shown in Figure 2, it is a coordinate system in which three mutually orthogonal axes are defined as the X axis, Y axis, and Z axis, respectively. The sensor coordinate system 302 has the position of the magnetic field generator 310 as its origin, and as shown in Figure 2, it is a coordinate system in which three mutually orthogonal axes are defined as the x axis, y axis, and z axis, respectively, with the origin as the reference point.

[0039] The measurement unit 102 is fixedly attached to the HMD 100 and measures the change in the magnetic field according to its own position and orientation within the magnetic field generated by the magnetic field generator 310, and transmits the measurement results to the controller 320. Based on the measurement results from the measurement unit 102, the controller 320 generates information (signal values, etc.) indicating the position and orientation of the measurement unit 102 in the sensor coordinate system 302 and transmits it to the information processing device 200.

[0040] In this embodiment, instead of the magnetic field sensor system shown in Figure 2, a system using, for example, an ultrasonic sensor system, an optical sensor system, an acceleration sensor, or a gyro sensor may be applied, or a combination of these systems may be applied. Furthermore, although the information processing system 10-1 shown in Figure 1A and the information processing system 10-2 shown in Figure 2 are configured as separate units for the HMD 100 and the information processing device 200, the present invention is not limited to this configuration. For example, a configuration in which the information processing device 200 is incorporated into the HMD 100 to form a standalone HMD is also applicable to the present invention.

[0041] Figure 3 shows an example of the hardware configuration of the information processing device 200 according to the first embodiment. The information processing device 200 is composed of, for example, a computer device such as a PC (personal computer), a portable terminal device such as a smartphone or tablet terminal, etc.

[0042] As shown in Figure 3, the information processing device 200 has a hardware configuration comprising a processor 210, memory 220, storage medium 230, input I / F 240, and output I / F 250.

[0043] In Figure 3, the processor 210 is, for example, a CPU, which controls the overall operation of the information processing device 200 and performs various processes. The memory 220 is, for example, RAM, which temporarily stores programs and various information. The storage medium 230 is a computer-readable storage medium, such as a hard disk or CD-ROM, which stores programs and various information for a long period of time. In this embodiment, the program stored in the storage medium 230 is read into the memory 220. The processor 210 then executes the program read into the memory 220, thereby realizing the various functional configurations (201 to 207) of the information processing device 200 shown in Figure 1A.

[0044] Furthermore, in Figure 3, the input I / F 240 receives various types of information (including various images) from external devices such as the HMD 100 and controller 320 as input information (including various images) in a format that can be processed by the information processing device 200. The output I / F 250 outputs various types of information (including various images) processed by the processor 210 as output information (including various images) in a format that can be processed by the external devices such as the HMD 100 and controller 320.

[0045] Figure 4 is a flowchart showing an example of a processing procedure in the control method of the information processing device 200 according to the first embodiment. The processing shown in the flowchart in Figure 4 enables observer K to observe a mixed reality image with a field of view that corresponds to the observer K's situation, such as the observer K's settings.

[0046] First, in step S101 of Figure 4, the acquisition unit 201 acquires a real-space image obtained by imaging the real space from the imaging unit 101. In this embodiment, the acquisition unit 201 acquires, for example, a real-space image for the left eye captured by the left eye imaging unit 101L and a real-space image for the right eye captured by the right eye imaging unit 101R. Furthermore, the acquisition unit 201 acquires information indicating the position and orientation of the measurement unit 102 in the sensor coordinate system 302 from the measurement unit 102 via the controller 320.

[0047] Next, in step S102 of Figure 4, the estimation unit 202 sets the right eye imaging unit 101R as the right viewpoint and the left eye imaging unit 101L as the left viewpoint, and then estimates the position and orientation of the HMD 100 in the world coordinate system 301 for both the right and left viewpoints.

[0048] The following is a detailed explanation of step S102 in Figure 4. First, as a prerequisite, conversion information for converting the position and orientation in the sensor coordinate system 302 to the position and orientation in the world coordinate system 301 is assumed to be determined in advance and registered in the information processing device 200. Furthermore, the relative positional relationship between the measurement unit 102 and the right eye imaging unit 101R (right eye bias) and the relative positional relationship between the measurement unit 102 and the left eye imaging unit 101L (left eye bias) are assumed to be determined in advance and registered in the information processing device 200. When the estimation unit 202 receives information (signal values, etc.) indicating the position and orientation of the measurement unit 102 in the sensor coordinate system 302 from the measurement unit 102 via the controller 320, it converts the position and orientation indicated by this information to the position and orientation in the world coordinate system 301 using the conversion information described above. Then, the estimation unit 202 estimates the position and orientation of the right viewpoint in the world coordinate system 301 by adding the right eye bias described above to the position and orientation converted to the world coordinate system 301. Similarly, the estimation unit 202 estimates the position and orientation of the left viewpoint in the world coordinate system 301 by converting to the world coordinate system 301 and adding the left-eye bias described above to the position and orientation. In the following description, explanations common to both the right and left viewpoints may be referred to collectively as "viewpoint" as needed. In addition to the method described above, various other methods can be applied to determine the position and orientation of the viewpoint in the world coordinate system 301. For example, a method can be applied in which markers assigned to the world coordinate system 301 are extracted from the real-space image captured by the imaging unit 101, and the position and orientation of the viewpoint in the world coordinate system 301 is estimated based on the position and orientation of the extracted markers. Alternatively, the position and orientation of the viewpoint may be estimated by performing SLAM (Simultaneous Localization And Mapping) processing based on feature points captured in the real-space image captured by the imaging unit 101.

[0049] Next, in step S103 of Figure 4, the determination unit 204 obtains setting information from the setting unit 203 regarding whether or not observer K wants to widen the field of view, and determines whether or not to widen the field of view of the augmented reality image by observer K based on the obtained setting information. Observer K can set whether or not to widen the field of view, for example, via a GUI displayed by the information processing device 200, and this setting can be stored as setting information in the setting unit 203.

[0050] Next, in step S104 of Figure 4, the cropping unit 205 obtains judgment result information from the determination unit 204 regarding whether or not to widen the field of view of the augmented reality image as determined by observer K, and determines the cropping range of the real space image acquired in step S101 based on the judgment result information.

[0051] Figure 5 shows a first embodiment and illustrates an example of the cropping range when the cropping unit 205 crops a real-space cropped image from a real-space image when the determination result of the determination unit 204 does not widen the field of view of observer K. In Figure 5, the same reference numerals are used for the same components as those shown in Figure 1B, and their detailed explanations are omitted.

[0052] Figure 5 shows the alignment direction plane H in the alignment direction of the left eye imaging unit 101L and the right eye imaging unit 101R, and a predetermined plane S that is parallel to the alignment direction plane H and is located at a predetermined distance (1000 mm in the example of Figure 5) from the alignment direction plane H. In this embodiment, the center of the cropping range when the cropping unit 205 crops the real space cropped image is set to point P on the predetermined plane S shown in Figure 5. Specifically, point P is a point 1000 mm away in the imaging direction from the center position between the tip of the left eye imaging unit 101L and the tip of the right eye imaging unit 101R. The 1000 mm described here may be changed depending on the specifications of the information processing system. For example, it may be the focal length of the imaging unit 101 in the left eye imaging unit 101L and the right eye imaging unit 101R.

[0053] When the cropping unit 205 receives judgment result information from the determination unit 204 indicating that the field of view should not be widened, it determines the cropping range so that the left and right sides of point P are each 30 degrees, as shown by the thick line in Figure 5. That is, when the cropping unit 205 crops a real-space cropped image for the left eye from a real-space image for the left eye captured by the left eye imaging unit 101L, it determines the cropping range Lh1 at a cropping angle of 60 degrees, the same as the display field of view shown in Figure 1B. Similarly, when the cropping unit 205 crops a real-space cropped image for the right eye from a real-space image for the right eye captured by the right eye imaging unit 101R, it determines the cropping range Rh1 at a cropping angle of 60 degrees, the same as the display field of view shown in Figure 1B. In this case, since the cropping range Lh1 and the cropping range Rh1 are the same range that completely coincide on a predetermined plane S, the real-space cropped image for the left eye and the real-space cropped image for the right eye cropped by the cropping unit 205 are identical. Note that in Figure 5, the cropped field of view is 60 degrees, which is smaller than the 90-degree imaging field of view.

[0054] Figure 6A shows a first embodiment and illustrates an example of the cropping range when the cropping unit 205 crops a real-space cropped image from a real-space image when the determination result of the determination unit 204 widens the field of view of observer K. In Figure 6A, the same reference numerals are used for components that are the same as those shown in Figures 1B and 5, and their detailed explanations are omitted.

[0055] In Figure 6A, as in Figure 5, the alignment direction plane H in the alignment direction of the left eye imaging unit 101L and the right eye imaging unit 101R, and a predetermined plane S that is parallel to the alignment direction plane H and is located at a predetermined distance (1000 mm in the example of Figure 6A) from the alignment direction plane H are shown. In Figure 6A, point P on the predetermined plane S is, as in Figure 5, a point 1000 mm away in the imaging direction from the center position between the tip of the left eye imaging unit 101L and the tip of the right eye imaging unit 101R.

[0056] When the cropping unit 205 receives determination result information from the determination unit 204 indicating that the field of view should be widened, it determines the cropping range so that the cropping angle is 60 degrees, the same as the display angle, and the cropping range on the predetermined plane S is wider compared to the case in Figure 5, as shown by the thick line in Figure 6A. Specifically, when the cropping unit 205 crops a real-space cropped image for the left eye from a real-space image for the left eye captured by the left eye imaging unit 101L, it determines the cropping range Lh2 with a cropping angle of 40 degrees to the left (outside) and 20 degrees to the right (inside) relative to point P. Similarly, when the cropping unit 205 crops a real-space cropped image for the right eye from a real-space image for the right eye captured by the right eye imaging unit 101R, it determines the cropping range Rh2 with a cropping angle of 40 degrees to the right (outside) and 20 degrees to the left (inside) relative to point P. In this case, on a predetermined surface S, the cropping range Lh2 and cropping range Rh2 overlap in some inner areas and differ in some outer areas. Therefore, the real-space cropped image for the left eye and the real-space cropped image for the right eye, cropped by the cropping section 205, will have overlapping inner areas and differing outer areas.

[0057] Specifically, the cutting unit 205 cuts out a real-space cutout image from the real-space image so that, if the determination result of the determination unit 204 is to widen the field of view (Figure 6A), the rendering range of the composite reality image generated by the generation unit 207 is wider than if the field of view is not widened (Figure 5). Also, if the determination result of the determination unit 204 is not to widen the field of view (Figure 5), the cutting unit 205 cuts out a real-space cutout image for the left eye and a real-space cutout image for the right eye so that they render the same range on a predetermined plane S. On the other hand, if the determination result of the determination unit 204 is to widen the field of view (Figure 6A), the cutting unit 205 cuts out a real-space cutout image for the left eye and a real-space cutout image for the right eye so that at least a part of the range is different on a predetermined plane S.

[0058] When the judgment result information indicating a widening of the field of view, as shown in Figure 6A, is received, the total width of the cropped ranges Lh2 and Rh2 is wider than the total width of the cropped ranges Lh1 and Rh1 when the judgment result information indicating not to widen the field of view, as shown in Figure 5, is received. For example, when observer K looks with both eyes, if the judgment result information indicating not to widen the field of view, as shown in Figure 5, is received, a range of 60 degrees is visible, but if the judgment result information indicating widening the field of view, as shown in Figure 6A, a range of 80 degrees is visible. Therefore, when switching from receiving the judgment result information indicating not to widen the field of view, as shown in Figure 5, to receiving the judgment result information indicating widening the field of view, as shown in Figure 6A, observer K will feel that their field of view has widened. However, in the case shown in Figure 6A, there is a range that can only be seen by observer K's left eye (left edge) and a range that can only be seen by the right eye (right edge), and since the visibility at the edges may be a concern for some observers K, it is important that this can be set in the setting unit 203.

[0059] Furthermore, as shown in Figure 1B, by setting the ratio of the field of view to be equal in each left-eye imaging unit 101L and left-eye display unit 103L, when observer K observes the augmented reality image, the sense of size may be perceived differently from the size in real space. Here, the ratio of the field of view on the left and right sides of point P may be switched depending on how wide the field of view is to be. This ratio may be stored in advance by the information processing device 200, or it may be set by observer K. Subsequently, if the resolution of the extracted real-space image differs from the resolution of the display unit 103, the real-space image may be enlarged or reduced, and a magnification may be determined so that the resolutions of both match.

[0060] As shown in Figure 6A, when widening the observer K's field of view, the display field of view of the display unit 103 is allocated equally to the left and right, as shown in Figure 1B. However, as shown in Figure 6A, the imaging field of view of the imaging unit 101 in the cropped image is not allocated equally to the left and right. Therefore, observer K may perceive objects as having different sizes. To address this, as described above, the cropped real-space image may be enlarged or reduced to match the resolution of the display unit 103, and the cropped real-space image may also be enlarged or reduced to match the perceived size. The magnification when enlarging or reducing the real-space image to match the perceived size may be set by observer K while confirming, or the magnification for the change in the imaging field of view may be stored in advance in the information processing device 200.

[0061] Furthermore, in order to generate a virtual space image that matches the real-space cropped image described later, it is necessary to calculate the principal point, field of view, and focal length, which are internal parameters of the imaging unit 101. In this case, the principal point is the position of point P on the image. Specifically, the principal point is the center of the image when the field of view is not widened. On the other hand, when the field of view is widened, the principal point can be calculated from the ratio of the left field of view and the right field of view as described above. In this embodiment, the cropped field of view is 60 degrees, the same as the display field of view, and the focal length can be calculated by multiplying the originally determined focal length by the magnification factor when the image is enlarged or reduced as described above.

[0062] Now, let's return to the explanation of Figure 4. Once the process in step S104 in Figure 4 is completed, the process proceeds to step S105. When the process proceeds to step S105 in Figure 4, the cropping unit 205 performs the process of cropping a real-space cropped image from the real-space image acquired in step S101 within the cropping range determined in step S104. Specifically, in this embodiment, the cropping unit 205 crops a real-space cropped image for the left eye from the real-space image for the left eye captured by the left eye imaging unit 101L, and also crops a real-space cropped image for the right eye from the real-space image for the right eye captured by the right eye imaging unit 101R.

[0063] Next, in step S106 of Figure 4, the generation unit 207 first constructs a virtual space where virtual objects are placed using CG information acquired from the holding unit 206. Then, the generation unit 207 generates virtual space images that can be seen from the right and left viewpoint positions and orientations of the HMD 100 estimated in step S102, using the internal parameters of the imaging unit 101 calculated by the cropping unit 205 in step S104. Specifically, in this embodiment, the generation unit 207 generates a virtual space image for the left eye corresponding to the real-space cropped image for the left eye obtained in step S105, and also generates a virtual space image for the right eye corresponding to the real-space cropped image for the right eye obtained in step S105. Here, since known techniques can be used for generating virtual space images viewed from a viewpoint, their explanation is omitted. The generation unit 207 can generate virtual space images that match the real-space cropped image cropped by the cropping unit 205 by utilizing the internal parameters of the imaging unit 101 calculated by the cropping unit 205. Therefore, if the setting is to widen the field of view, a virtual space image will be generated to reflect that wider field of view.

[0064] Next, in step S107 of Figure 4, the generation unit 207 combines the real-space cutout image obtained in step S105 and the virtual-space image obtained in step S106 to generate a composite reality image to be displayed on the display unit 103. Specifically, in this embodiment, the generation unit 207 combines the real-space cutout image for the left eye obtained in step S105 and the virtual-space image for the left eye obtained in step S106 to generate a composite reality image for the left eye to be displayed on the left-eye display unit 103L. Similarly, in this embodiment, the generation unit 207 combines the real-space cutout image for the right eye obtained in step S105 and the virtual-space image for the right eye obtained in step S106 to generate a composite reality image for the right eye to be displayed on the right-eye display unit 103R. Here, the synthesis process between the real-space image and the virtual-space image is performed by overlaying the virtual-space image on top of the real-space image. That is, pixels in the virtual-space image that are not in the region of a virtual object, etc., become a composite image in which the pixels of the real-space image are displayed. Subsequently, the left-eye display unit 103L of the HMD 100 displays the left-eye composite reality image generated in step S107, and the right-eye display unit 103R of the HMD 100 displays the right-eye composite reality image generated in step S107. As a result, observer K can observe the composite reality space.

[0065] Figure 6B shows an example of the first embodiment, illustrating the mixed reality image 610 for the left eye, the mixed reality image 620 for the right eye, and the mixed reality image 630 that observer K sees when he uses both eyes, when the determination result of the determination unit 204 widens the observer K's field of view. In Figure 6B, the same reference numerals are used for the same components as in Figure 6A, and their detailed explanations are omitted.

[0066] The mixed reality image 610 for the left eye, shown in Figure 6B, has a width equal to the cropping range Lh2 shown in Figure 6A. Similarly, the mixed reality image 620 for the right eye, also shown in Figure 6B, has a width equal to the cropping range Rh2 shown in Figure 6A. Furthermore, the mixed reality image 630 seen by observer K with both eyes has an overlapping inner portion between the mixed reality image 610 for the left eye and the mixed reality image 620 for the right eye, while differing in some outer portions.

[0067] Now, let's return to the explanation of Figure 4. Once the process in step S107 in Figure 4 is completed, the process proceeds to step S108. When the process proceeds to step S108 in Figure 4, the processor 210 determines, for example, via the input I / F 240, whether or not observer K has given an instruction to end the observation of the mixed reality image. If, in step S108 of Figure 4, the processor 210 determines that observer K has not given an instruction to end the observation of the mixed reality image (S108 / No), the process returns to step S101 and repeats the processing from step S101 onwards (processing of the next frame).

[0068] Furthermore, in step S108 of Figure 4, if the processor 210 determines that observer K has given an instruction to end the observation of the mixed reality image (S108 / Yes), the process of the flowchart shown in Figure 4 is terminated.

[0069] The information processing device 200 according to the first embodiment described above is an information processing device that generates a composite reality image to be presented to observer K, and has the following configuration. Specifically, the information processing device 200 has an acquisition unit 201 that acquires a real space image obtained by imaging the real space, and a determination unit 204 that determines whether or not to widen the field of view of the composite reality image for observer K according to observer K's settings. The information processing device 200 also has an extraction unit 205 that extracts a real space extract image from the real space image based on the determination result of the determination unit 204. Furthermore, the information processing device 200 has a generation unit 207 that synthesizes the real space extract image obtained by the extraction unit 205 and the virtual space image corresponding to the real space extract image to generate a composite reality image to be displayed on the display unit 103. With this configuration, if observer K sets the system to widen the field of view to observe the augmented reality image, the display unit 103 can display the augmented reality image with a widened field of view. This allows observer K to observe the augmented reality image with a field of view appropriate to their situation. Furthermore, by cropping the real-space image to widen the field of view only when observer K needs to, it is possible to reduce the sense of discomfort experienced by observer K.

[0070] <Modification 1 of the first embodiment> In the first embodiment described above, the cropping range for extracting the real-space image from the real-space image was changed so that the field of view was widened according to the setting that observer K wanted to widen the field of view to observe the mixed reality image. In this modified example 1, the cropping range for extracting the real-space image from the real-space image is changed according to the size of the virtual object to be displayed (i.e., according to the mixed reality environment of observer K). In the description of the modified example 1 of the first embodiment described below, the explanation of matters common to the first embodiment described above will be omitted, and matters that differ from the first embodiment described above will be explained.

[0071] In the information processing device 200 according to Modification 1 of the First Embodiment, the processing content of the determination unit 204 shown in Figure 1A differs from that of the First Embodiment described above.

[0072] The determination unit 204 according to Modification 1 of the First Embodiment acquires CG information from the holding unit 206. Then, if the size of the virtual object used when generating a virtual space image based on the acquired CG information is greater than or equal to a threshold, the determination unit 204 according to Modification 1 of the First Embodiment transmits determination result information to the cropping unit 205 to widen the field of view. On the other hand, if the size of the virtual object used when generating a virtual space image based on the acquired CG information is less than a threshold, the determination unit 204 according to Modification 1 of the First Embodiment transmits determination result information to the cropping unit 205 to not widen the field of view. Here, the size of the virtual object may be the size of the longest side of the bounding box of the CG, or the size may be the distance between the two furthest points among the vertex information of the CG.

[0073] As described above, in Modification 1 of the first embodiment, the determination unit 204 determines whether or not to widen the field of view based on the size of the virtual objects used when generating the virtual space image. Specifically, the determination unit 204 in Modification 1 determines whether to widen the field of view if the size of the virtual objects used when generating the virtual space image is greater than or equal to a threshold, thereby widening the observer K's field of view so that all the virtual objects are visible. On the other hand, the determination unit 204 in Modification 1 determines whether or not to widen the field of view if the size of the virtual objects used when generating the virtual space image is less than a threshold, thereby allowing the observer K to observe the mixed reality image without being bothered by the appearance of the edges.

[0074] <Modification 2 of the first embodiment> In Modification 1 of the First Embodiment described above, the determination of whether or not to widen observer K's field of view was made based on whether or not the size of the virtual object used to generate the virtual space image in observer K's mixed reality environment was greater than or equal to a threshold. In Modification 2 of this embodiment, the determination of whether or not to widen observer K's field of view is made based on whether or not the virtual object used to generate the virtual space image in observer K's mixed reality environment is within observer K's field of view. In the description of Modification 2 of the First Embodiment described below, the explanation of matters common to the First Embodiment and its Modification 1 described above will be omitted, and the explanation of matters that differ from the First Embodiment and its Modification 1 described above will be given.

[0075] In the information processing device 200 according to Modification 2 of the First Embodiment, the processing contents of the determination unit 204 and the cutting unit 205 shown in Figure 1A differ from those of the First Embodiment and Modification 1 described above.

[0076] The determination unit 204 according to Modification 2 of the First Embodiment uses the position and orientation of the imaging unit 101 of the HMD 100 estimated by the estimation unit 202, the internal parameters of the imaging unit 101 previously calculated by the cropping unit 205, and the CG information held in the holding unit 206. The determination unit 204 according to Modification 2 of the First Embodiment uses these three pieces of information to determine whether or not the virtual object used when generating the virtual space image is within the observer K's field of view. Specifically, for example, the determination unit 204 according to Modification 2 uses the position and orientation of the imaging unit 101 of the HMD 100 and the internal parameters of the imaging unit 101 to perform a perspective projection transformation on the virtual object used when generating the virtual space image, thereby converting it to a position in the image's coordinate system. The determination unit 204 according to Modification 2 then determines whether or not the virtual object is within the observer K's field of view based on whether or not this position in the image's coordinate system is within the real space cropped image cropped by the cropping unit 205. Since publicly known techniques can be used to determine whether or not an object is within the field of view of observer K, a detailed explanation of these techniques will be omitted.

[0077] The determination unit 204 in this modified example 2 determines that observer K's field of view should be widened if the virtual object used to generate the virtual space image is not within observer K's field of view. On the other hand, the determination unit 204 in this modified example 2 determines that observer K's field of view should not be widened if the virtual object used to generate the virtual space image is within observer K's field of view.

[0078] Furthermore, depending on how observer K observes the mixed reality image, the determination result of the determination unit 204 may change frequently, potentially making it difficult for observer K to observe (experience) the mixed reality image. For this reason, the determination of whether or not to widen observer K's field of view may also include whether or not a virtual object has been within observer K's field of view for a period exceeding a threshold. For example, the determination unit 204 in this modified example 2 determines not to widen observer K's field of view if a virtual object used to generate the virtual space image has been within observer K's field of view for a period exceeding a threshold (continuously), and transmits this determination result information to the extraction unit 205. On the other hand, the determination unit 204 in this modified example 2 determines to widen observer K's field of view if a virtual object used to generate the virtual space image has not been within observer K's field of view for a period exceeding a threshold (continuously), and transmits this determination result information to the extraction unit 205. The threshold in this case may be set by observer K and held by the information processing device 200.

[0079] Furthermore, if the virtual object used to generate the virtual space image is close to the position of the HMD 100, the probability of the virtual object always being within the observer K's field of view decreases. Therefore, the determination unit 204 in this modified example 2 may determine whether or not to widen the observer K's field of view depending on the distance between the virtual object and the HMD 100 (e.g., the imaging unit 101). For example, the determination unit 204 in this modified example 2 may determine to widen the observer K's field of view if the distance between the virtual object and the HMD 100 (e.g., the imaging unit 101) is less than a threshold. On the other hand, the determination unit 204 in this modified example 2 may determine not to widen the observer K's field of view if the distance between the virtual object and the HMD 100 (e.g., the imaging unit 101) is greater than or equal to a threshold. Here, the distance between the virtual object and the HMD 100 may be calculated by the distance between the center of the bounding box of the CG and the left eye imaging unit 101L, or by the distance between the point closest to the left eye imaging unit 101L among the vertices of the CG and the left eye imaging unit 101L. Furthermore, regarding the distance between the virtual object and the HMD 100, if the condition is met for a predetermined period (continuously) above a threshold, a determination may be made as to whether or not to widen the observer K's field of view. Moreover, the determination based on the distance between the virtual object and the HMD 100 described here and the determination based on whether or not the virtual object is within the observer K's field of view may be combined and performed simultaneously. For example, in this case, the determination unit 204 according to this modified example 2 may transmit the determination result information to widen the field of view to the cropping unit 205 if at least one of the two determinations is determined to widen the field of view.

[0080] If the determination unit 204 determines that observer K's field of view should be widened, the cropping unit 205 switches the cropping range of the real-space cropped image from the state where observer K's field of view is not widened. In the first embodiment described above, when observer K's field of view was not widened, the cropping unit 205 adjusted the cropping angle to a display angle of 60 degrees. On the other hand, when observer K's field of view was widened, the cropping unit 205 set the cropping range so that the field of view for both of observer K's eyes became 80 degrees. It can be dangerous if the angle of view suddenly changes while observer K is observing (experiencing) the augmented reality image. Therefore, in this modified example 2, the cropping range may be changed gradually. As one method of gradually changing the cropping range, a threshold may be set for the cropping range that can be changed in a single change process. For example, in a single modification process, for the real-space cropped image for the left eye, the cropping range Lh2 is set and cropped such that the left (outer) side of point P in Figure 6A is 31 degrees and the right (inner) side is 29 degrees. Similarly, for example, in a single modification process, for the real-space cropped image for the right eye, the cropping range Rh2 is set and cropped such that the right (outer) side of point P in Figure 6A is 31 degrees and the left (inner) side is 29 degrees. In this way, the predetermined amount by which the cropping range is shifted may be limited to a maximum of 1 degree. Furthermore, the predetermined amount by which the cropping range is shifted may be determined by the observer K. That is, in the cropping unit 205 according to this modified example 2, when the determination result of the determination unit 204 is to widen the field of view, the cropping unit changes the cropping range by a predetermined amount when cropping a series of real-space cropped images. In addition, a threshold that allows for a total shift of the cropping range may be defined separately from the predetermined amount by which the cropping range is shifted in a single modification process. In the example shown in Figure 6A of the first embodiment described above, the cropping ranges Lh2 and Rh2 were shifted so that the field of view for both of observer K's eyes reached 80 degrees. In this way, a total threshold may be set so that the field of view for both of observer K's eyes reaches a maximum of 80 degrees. Furthermore, this total threshold may be set by observer K.

[0081] According to Modification 2 of the First Embodiment, it is possible to determine whether or not to widen the observer K's field of view depending on whether or not the virtual object is within the observer K's field of view, whether or not the distance between the virtual object and the HMD 100 is below a threshold, etc. If it is determined that the observer K's field of view should be widened, according to Modification 2 of the First Embodiment, the observer K's field of view can be gradually widened to observe the augmented reality image.

[0082] <Modification 3 of the first embodiment> In the first embodiment described above, the cropping range of the real-space cropped image was shifted as shown in Figure 6A in order to widen the field of view of observer K. In this modified example 3, the field of view of observer K is widened by widening the cropping range in both the vertical and horizontal directions with respect to a predetermined plane S. In the description of the modified example 3 of the first embodiment described below, matters common to the first embodiment described above will be omitted, and matters that differ from the first embodiment described above will be explained.

[0083] In the information processing device 200 according to modification 3 of the first embodiment, the processing content of the cutting section 205 shown in Figure 1A differs from that of the first embodiment described above.

[0084] In this modified example 3, if the determination result of the determination unit 204 does not widen the observer K's field of view, the cropping unit 205 determines the cropping range Lh1 and Rh1 by adjusting the cropping angle to 60 degrees, the display angle of view, similar to Figure 5 in the first embodiment described above. In this modified example 3, if the determination result of the determination unit 204 widens the observer K's field of view, the cropping unit 205 determines the cropping range by setting the horizontal cropping angle with respect to a predetermined surface S to an angle greater than 60 degrees, the display angle of view.

[0085] Figure 7 shows a third modification of the first embodiment, illustrating an example of the cropping range when the cropping unit 205 crops a real-space cropped image from a real-space image when the determination result of the determination unit 204 widens the field of view of observer K. In Figure 7, the same reference numerals are used for the same components as those shown in Figures 1B, 5, and 6A, and their detailed explanations are omitted.

[0086] In Figure 7, as in Figure 5, the alignment direction plane H in the alignment direction of the left eye imaging unit 101L and the right eye imaging unit 101R, and a predetermined plane S that is parallel to the alignment direction plane H and is located at a predetermined distance (1000 mm in the example of Figure 7) from the alignment direction plane H are shown. In Figure 7, point P on the predetermined plane S is, as in Figure 5, a point 1000 mm away in the imaging direction from the center position between the tip of the left eye imaging unit 101L and the tip of the right eye imaging unit 101R.

[0087] In the modified example 3 of the first embodiment, when the cutting unit 205 receives determination result information from the determination unit 204 indicating that the field of view should be widened, it determines the cutting range so that the left and right sides of point P have a field of view of 40 degrees, as shown by the thick line in Figure 7. That is, when the cutting unit 205 cuts out a real-space cutout image for the left eye from a real-space image for the left eye captured by the left-eye imaging unit 101L, it determines the cutting range Lh3 at a cutting field of view of 80 degrees, which is greater than the cutting field of view of 60 degrees shown in Figure 5, as shown in Figure 7.

[0088] In other words, in this modified example 3, if the determination result of the determination unit 204 does not widen the field of view, the cutting unit 205 determines the cutting ranges Lh1 and Rh1 so as to depict the first identical range on a predetermined surface S, as shown in Figure 5, and cuts out a real-space cutout image. On the other hand, if the determination result of the determination unit 204 widens the field of view, the cutting unit 205 determines the cutting ranges Lh3 and Rh3 so as to depict a second identical range that is wider than the first identical range described above on a predetermined surface S, as shown in Figure 7, and cuts out a real-space cutout image.

[0089] Furthermore, when widening the field of view of observer K as shown in Figure 7, the cropped field of view becomes 80 degrees compared to the display field of view of 60 degrees. In this case, the cropped image of the real space may be reduced in size so that it can be displayed at the resolution of the display unit 103. In addition, the field of view in the direction perpendicular to a predetermined plane S may be increased at the same ratio as the increase in the horizontal field of view.

[0090] According to Modification 3 of the First Embodiment, when observer K observes the mixed reality image, the sense of size appears to change, but by widening the cropping range in both the vertical and horizontal directions, the sense of unnaturalness in the appearance of the edges can be reduced while widening observer K's field of view.

[0091] (Second embodiment) Next, a second embodiment will be described. In the description of the second embodiment below, matters common to the first embodiment described above will be omitted, and matters that differ from the first embodiment described above will be explained.

[0092] In the first embodiment described above, as well as its modifications 1 and 2, the decision to widen observer K's field of view was made based on the observer's settings and the size of the virtual object and whether or not the virtual object was within observer K's field of view as part of the observer's mixed reality environment. The second embodiment is a configuration that allows observer K to avoid danger by changing observer K's field of view depending on whether or not there is danger when observer K experiences the mixed reality space.

[0093] Figure 8 shows an example of the schematic configuration of the information processing system 20 according to the second embodiment. In Figure 8, the same reference numerals are used for components that are the same as those shown in Figures 1A and 2, and their detailed descriptions are omitted.

[0094] As shown in Figure 8, the information processing system 20 includes an HMD 100 and an information processing device 200. The HMD 100 and the information processing device 200 shown in Figure 8 are connected via wired and / or wireless communication to enable data communication between them. In the information processing system 20 shown in Figure 8, the HMD 100 and the information processing device 200 are configured as separate units, but the present invention is not limited to this configuration. For example, a configuration in which the information processing device 200 is incorporated into the HMD 100 to create a standalone HMD is also applicable to the present invention.

[0095] The HMD100 shown in Figure 8 has an imaging unit 101, a measurement unit 102, and a display unit 103, similar to Figure 1A.

[0096] The information processing device 200 shown in Figure 8 includes an acquisition unit 201, an estimation unit 202, a transmission / reception unit 221, a determination unit 222, an extraction unit 205, a holding unit 206, and a generation unit 223.

[0097] The transmitting / receiving unit 221 transmits and receives information about objects other than observer K shown in Figure 2. For example, the transmitting / receiving unit 221 transmits and receives information about a second observer, different from observer K shown in Figure 2, as information about other objects in observer K's mixed reality environment. Specifically, for example, the transmitting / receiving unit 221 transmits and receives information about the position and orientation of the second observer's imaging unit as information about other objects in observer K's mixed reality environment.

[0098] The determination unit 222 is a determination means that determines whether or not to widen the field of view of the augmented reality image by the observer, based on information about other objects received by the transmitting / receiving unit 221 and the position and orientation of the imaging unit 101 estimated by the estimation unit 202.

[0099] The generation unit 223 is a generation means that generates a composite reality image to be displayed on the display unit 103 by combining the real-space cropped image obtained by the cropping unit 205 and a virtual space image generated using the CG information of the holding unit 206 that corresponds to the real-space cropped image.

[0100] Figure 9 is a flowchart showing an example of a processing procedure in the control method of the information processing device 200 according to the second embodiment. The processing shown in the flowchart in Figure 9 widens the field of view of observer K when there is danger to observer K, thereby enabling observer K to avoid danger. In addition, in the processing shown in the flowchart in Figure 9, the same step numbers are used for processing steps with the same processing content as in the flowchart in Figure 4, and a detailed explanation is omitted.

[0101] First, in steps S101 and S102 of Figure 9, the same processes as in steps S101 and S102 of Figure 4 are performed, respectively.

[0102] Next, in step S201 of Figure 9, the transmitting / receiving unit 221 first transmits and receives information about the position and orientation of the second observer's imaging unit as information about other objects in the mixed reality environment for observer K. Then, the determination unit 222 calculates the distance between the HMD 100's imaging unit 101 and the second observer's imaging unit based on the position and orientation of the HMD 100's imaging unit 101 estimated in step S102 and the position and orientation of the second observer's imaging unit received by the transmitting / receiving unit 221. Then, the determination unit 222 determines whether or not to widen the field of view of the mixed reality image for observer K based on the calculated distance between the HMD 100's imaging unit 101 and the second observer's imaging unit. Specifically, if the distance between the HMD 100's imaging unit 101 and the second observer's imaging unit is less than a threshold, the determination unit 222 determines that there is a possibility of contact between the observers, and therefore widens the field of view of the mixed reality image for observer K. This improves the likelihood that observer K can detect a second observer. On the other hand, if the distance between the imaging unit 101 of the HMD 100 and the imaging unit of the second observer is greater than or equal to a threshold, the determination unit 222 determines that observer K should not widen the field of view of the augmented reality image.

[0103] Furthermore, the information processing device 200 may store the position and region in the world coordinate system 301 of other objects that observer K may collide with. The position and region of these other objects in the world coordinate system 301 may take the form specified by observer K. Similarly, the determination unit 222 may also make a determination to widen the field of view of the augmented reality image by observer K if the distance between the position and region of these other objects in the world coordinate system 301 and the imaging unit 101 of the HMD 100 falls below a threshold.

[0104] In addition, the determination unit 222 may use the real-space image acquired by the acquisition unit 201 and before the extraction unit 205 extracts it as a real-space extracted image to calculate the three-dimensional position in real space of other objects that observer K may collide with. The left-eye imaging unit 101L and the right-eye imaging unit 101R included in the imaging unit 101 are in a stereo camera configuration, so it is possible to measure the depth of pixels from the real-space image for the left eye and the real-space image for the right eye captured by these imaging units. Based on this depth information and the position and orientation of the imaging unit 101, the determination unit 222 may generate a three-dimensional point cloud in the world coordinate system 301 of other objects that observer K may collide with. If the point in the generated three-dimensional point cloud that is closest to the position and orientation of the imaging unit 101 is below a threshold, the determination unit 222 may determine to widen the field of view of the composite reality image for observer K.

[0105] Next, in steps S104 and S105 of Figure 9, the same processes as in steps S104 and S105 of Figure 4 are performed, respectively.

[0106] Next, in step S202 of Figure 9, the generation unit 223 first generates a virtual space image using the same process as in step S106 of Figure 4. Furthermore, if the determination unit 222 determines that the observer K's field of view should be widened, the generation unit 223 performs a process to include a warning mark in the generated virtual space image in order to display a warning mark to observer K.

[0107] Next, in step S107 of Figure 9, the generation unit 207 combines the real-space cutout image obtained in step S105 and the virtual-space image obtained in step S202 to generate a composite reality image to be displayed on the display unit 103.

[0108] Figure 10 shows a second embodiment, illustrating an example of a composite reality image 1010 for the left eye, a composite reality image 1020 for the right eye, and a composite reality image 1030 that observer K sees when he observes with both eyes, when the determination result of the determination unit 204 widens the observer K's field of view. In Figure 10, the same reference numerals are used for the same components as those shown in Figures 6A and 6B, and their detailed explanations are omitted.

[0109] The mixed reality image 1010 for the left eye and the mixed reality image 1020 for the right eye shown in Figure 10, as well as the mixed reality image 1030 that observer K sees with both eyes, all display the warning mark 1031 generated in step S202 of Figure 9. Thus, the warning mark 1031 shown in Figure 10 is displayed in a position that is visible to both eyes even when observer K widens their field of view. As a result, observer K can notice the warning without the warning mark 1031 becoming difficult to see.

[0110] Here, the timing of when the cropping unit 205 changes the cropping range to widen the field of view and when the warning mark 1031 is displayed may be staggered. For example, the generation unit 223 may display the warning mark 1031 on the display unit 103, and then change the cropping range in a process several frames later. In that case, the determination unit 222 may first send determination result information to the generation unit 223 indicating that observer K's field of view should be widened, and then send determination result information to the cropping unit 205 indicating that observer K's field of view should be widened in a process several frames later. This allows observer K to recognize that their field of view has been widened by the warning, and to widen their field of view without being surprised.

[0111] Now, let's return to the explanation of Figure 9. Once the process in step S107 in Figure 9 is completed, the process proceeds to step S108. When the process proceeds to step S108 in Figure 9, the same process as in step S108 in Figure 4 is performed, and the process shown in the flowchart in Figure 9 is completed.

[0112] According to the second embodiment, if observer K approaches other objects (hazardous materials) such as other observers or objects in the real world while experiencing the mixed reality space, the warning mark 1031 is displayed and the field of view is widened so that observer K can recognize the other objects.

[0113] <Modification 1 of the second embodiment> In the second embodiment described above, when observer K approaches another object (hazardous material) in the mixed reality environment of observer K, the observer K's field of view is widened to make it easier to recognize the other object (hazardous material). In this modification 1, after observer K approaches another object (hazardous material) and widens their field of view, if observer K recognizes the other object (hazardous material) in the mixed reality environment of observer K, the field of view is returned to its original state before widening. In the description of modification 1 of the second embodiment described below, the explanation of matters common to the second embodiment described above will be omitted, and matters that differ from the second embodiment described above will be explained.

[0114] In the information processing device 200 according to Modification 1 of the second embodiment, the processing content of the determination unit 222 shown in Figure 8 differs from that of the second embodiment described above.

[0115] The determination unit 222 according to the second embodiment described above determined that observer K's field of view should be widened when another object (hazardous material) approached observer K. Here, the determination unit 222 according to Modification 1 of the second embodiment uses the position and orientation of the imaging unit 101, the internal parameters previously calculated by the cropping unit 205, and the position and orientation of the other object (hazardous material) to determine whether the other object is within observer K's field of view or in a position visible to both eyes. Then, if the other object (hazardous material) is within observer K's field of view or in a position visible to both eyes, the determination unit 222 according to Modification 1 of the second embodiment determines that observer K has recognized the other object and does not widen observer K's field of view. This is because, as described above, when observer K's field of view is widened, the sense of size may be perceived differently, so if observer K has recognized the other object (hazardous material), it is considered better for observer K to observe (experience) the mixed reality image with the correct sense of size. Furthermore, the determination unit 222 according to the modified example 1 of the second embodiment determines that if other objects (hazardous materials) are not within the observer K's field of view or in a position visible to both eyes, the observer K is unable to recognize the other objects, and therefore the observer K's field of view is widened.

[0116] Furthermore, the determination unit 222 according to Modification 1 of the second embodiment may determine whether or not observer K was able to recognize another object based on the position and orientation history of the imaging unit 101. The determination unit 222 can determine that if it determines that another object (hazardous material) is within the field of view, using the position and orientation of the imaging unit 101 predicted from the position and orientation history of the imaging unit 101, then it can determine that the imaging unit 101 is moving in the direction of the other object (hazardous material). In this case, the determination unit 222 according to Modification 1 of the second embodiment may determine that observer K's field of view should not be widened. Here, the method for predicting the position and orientation of the imaging unit 101 may be obtained by extrapolating the position and orientation a certain number of seconds later from the position and orientation history of the imaging unit 101, or it may be predicted by deep learning or the like. In that case, the training data may be stored in the information processing device 200. Also, the observer K may set how many seconds later to predict. In addition, methods for predicting the position and orientation of the imaging unit 101 can be applied. For example, the position and orientation of the imaging unit 101 may be predicted using information from an acceleration sensor or gyro sensor and an extended Kalman filter.

[0117] In Modification 1 of the Second Embodiment, the determination unit 222 determines whether or not to widen the field of view of the mixed reality image by observer K, according to the mixed reality environment of observer K. Specifically, in Modification 1 of the Second Embodiment, the mixed reality environment of observer K described above is whether or not observer K perceives other objects different from the observer. Furthermore, whether or not other objects are perceived includes whether or not those other objects are within observer K's field of view. Furthermore, whether or not other objects are perceived includes whether or not there are other objects within observer K's field of view, which is calculated based on the position and orientation of the imaging unit 101 that captures the real-space image.

[0118] According to Modification 1 of the Second Embodiment, if observer K recognizes another object (hazardous material) while experiencing the mixed reality space, the field of view can be returned to a state where it is not widened. This allows observer K to observe the other object (hazardous material) with an accurate sense of its size, thus enabling the observer to experience the mixed reality space more safely.

[0119] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. This program and a computer-readable storage medium storing said program are included in the present invention.

[0120] The embodiments of the present invention described above are merely examples of how the invention can be implemented, and the technical scope of the invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0121] This embodiment includes the following configurations, methods, and programs. [Configuration 1] An information processing device that generates a mixed reality image to present to an observer, An acquisition means for acquiring a real-space image obtained by imaging the real space, A determination means for determining whether or not to widen the field of view of the augmented reality image by the observer, depending on the observer's settings or the observer's augmented reality environment, Based on the determination result of the determination means, an extraction means extracts a real-space extracted image from the real-space image, A generation means for generating a composite reality image to be displayed on a display unit by combining the real-space cutout image and a virtual space image corresponding to the real-space cutout image, An information processing device characterized by having the following features. [Configuration 2] The acquisition means acquires the real-space image for the left eye, captured by a left-eye imaging unit provided corresponding to the observer's left eye, and the real-space image for the right eye, captured by a right-eye imaging unit provided corresponding to the observer's right eye. The extraction means extracts the left eye's real-space extracted image from the left eye's real-space image and the right eye's real-space extracted image from the right eye's real-space image, based on the result of the determination by the determination means. The generating means is The real-world image for the left eye and the virtual-world image for the left eye corresponding to the real-world image for the left eye are combined to generate the composite reality image for the left eye to be displayed on the left-eye display unit. The real-world image for the right eye and the virtual-world image for the right eye corresponding to the real-world image for the right eye are combined to generate the composite reality image for the right eye that is displayed on the right-eye display unit. The information processing device according to configuration 1, characterized by the above. [Configuration 3] The cutting means is, If the determination result of the determination means widens the field of view, the real space cropped image is cropped from the real space image so that the rendering range of the augmented reality image generated by the generation means is wider than if the determination result of the determination means does not widen the field of view. An information processing device according to configuration 1 or 2, characterized by the above. [Structure 4] The acquisition means acquires the real-space image for the left eye, captured by a left-eye imaging unit provided corresponding to the observer's left eye, and the real-space image for the right eye, captured by a right-eye imaging unit provided corresponding to the observer's right eye. The extraction means extracts the left eye's real-space extracted image from the left eye's real-space image and the right eye's real-space extracted image from the right eye's real-space image, based on the result of the determination by the determination means. The cutting means is, If the determination result of the determination means does not widen the field of view, The left eye real-space cropped image and the right eye real-space cropped image are cropped so that the same range is depicted on a predetermined plane that is parallel to the plane in the direction of arrangement of the left eye imaging unit and the right eye imaging unit, and is at a predetermined distance from the plane in the direction of arrangement. If the determination result of the determination means widens the field of view, the real-space cropped image for the left eye and the real-space cropped image for the right eye are cropped so that at least a portion of the predetermined surface depicts a different area. An information processing device according to any one of configurations 1 to 3. [Composition 5] The acquisition means acquires the real-space image for the left eye, captured by a left-eye imaging unit provided corresponding to the observer's left eye, and the real-space image for the right eye, captured by a right-eye imaging unit provided corresponding to the observer's right eye. The extraction means extracts the left eye's real-space extracted image from the left eye's real-space image and the right eye's real-space extracted image from the right eye's real-space image, based on the result of the determination by the determination means. The cutting means is, If the determination result of the determination means does not widen the field of view, In a predetermined plane that is parallel to the plane in the direction of alignment of the left eye imaging unit and the right eye imaging unit, and is at a predetermined distance from the plane in the direction of alignment The real-space cropped image for the left eye and the real-space cropped image for the right eye are cropped to depict the first identical range, If the determination result of the determination means widens the field of view, the real-space cropped image for the left eye and the real-space cropped image for the right eye are cropped so as to depict a second identical area that is wider than the first identical area on the predetermined surface. An information processing device according to any one of configurations 1 to 3. [Composition 6] The aforementioned observer setting is a setting indicating whether or not the observer wants to widen the field of view of the augmented reality image. An information processing apparatus according to any one of configurations 1 to 5, characterized by the above. [Composition 7] The observer's mixed reality environment is determined by whether or not the size of the virtual object used to generate the virtual space image is above a certain threshold. An information processing device according to any one of configurations 1 to 6. [Structure 8] The observer's mixed reality environment is whether or not the virtual objects used to generate the virtual space image are within the observer's field of view. An information processing device according to any one of configurations 1 to 6. [Composition 9] The observer's mixed reality environment is determined by whether or not the virtual objects used to generate the virtual space image remain within the observer's field of view for a period exceeding a certain threshold. An information processing device according to any one of configurations 1 to 6. [Configuration 10] When the determination result of the determination means widens the field of view, the cropping means changes the cropping range by a predetermined amount when cropping a series of real-space cropped images. An information processing apparatus according to any one of configurations 1 to 9, characterized by the above. [Composition 11] The augmented reality environment of the observer is determined by whether the distance between the observer and other objects different from the observer is below a threshold. An information processing device according to any one of configurations 1 to 6. [Composition 12] The aforementioned other object is a second observer, different from the aforementioned observer. The information processing apparatus according to configuration 11, characterized by the features described above. [Composition 13] The aforementioned other object is an object located at a specified position or in a designated area in the real space. The information processing apparatus according to configuration 11, characterized by the features described above. [Composition 14] The other object is a point cloud in the real space. The information processing apparatus according to configuration 11, characterized by the features described above. [Composition 15] The acquisition means acquires the real-space image for the left eye, captured by a left-eye imaging unit provided corresponding to the observer's left eye, and the real-space image for the right eye, captured by a right-eye imaging unit provided corresponding to the observer's right eye. The point cloud in the aforementioned real space is calculated using the real space image for the left eye and the real space image for the right eye. The information processing apparatus according to configuration 14, characterized by the features described herein. [Composition 16] The observer's mixed reality environment is whether or not the observer perceives other objects different from the observer. An information processing device according to any one of configurations 1 to 6. [Composition 17] Whether or not the aforementioned other object is recognized depends on whether or not the aforementioned other object is within the observer's field of view. The information processing device according to configuration 16, characterized in that... [Composition 18] Whether or not the aforementioned other object is recognized is determined by whether or not the aforementioned other object is within the observer's field of view, which is calculated based on the position and orientation of the imaging unit that captures the real-space image. The information processing device according to configuration 16, characterized in that... [Composition 19] The generation means generates the virtual space image using the parameters used by the extraction means when obtaining the real space extraction image. An information processing apparatus according to any one of configurations 1 to 18, characterized by the above. [Configuration 20] The generation means enlarges or reduces the real-space cropped image to a size that can be displayed on the display unit. An information processing apparatus according to any one of configurations 1 to 19, characterized by the above. [Composition 21] When the generation means obtains the real-space cropped image by cropping the real-space image based on the result of the determination that the cropping means widens the field of view, The real-space excerpt image is enlarged or reduced so that the sense of size of the mixed-reality space observed by the observer matches the sense of size of the real space. An information processing apparatus according to any one of configurations 1 to 20, characterized by the above. [Composition 22] The generation means generates a warning mark in the virtual space image at a position where the observer can see it with both eyes, if the distance between the observer and another object other than the observer is below a threshold. An information processing device according to any one of configurations 1 to 21, characterized by the above. [Composition 23] The cropping means switches the cropping range for cropping the real-space image after the warning mark is displayed on the display unit. The information processing apparatus according to configuration 22, characterized by the features described above. [Composition 24] The observer's mixed reality environment is determined by whether or not the distance between the virtual object used to generate the virtual space image and the imaging unit that captures the real space image is below a threshold. An information processing device according to any one of configurations 1 to 6. [Composition 25] The observer's mixed reality environment is determined by whether or not the distance between the virtual object used to generate the virtual space image and the imaging unit that captures the real space image fell below a threshold over a predetermined period of time. An information processing device according to any one of configurations 1 to 6. [Method 1] A method for controlling an information processing device that generates a mixed reality image to be presented to an observer, An acquisition step to obtain a real-space image obtained by imaging the real space, A determination step of determining whether or not to widen the field of view of the mixed reality image by the observer, depending on the observer's settings or the observer's mixed reality environment, Based on the result of the determination step, an extraction step is performed to extract a real-space extracted image from the real-space image, A generation step of generating a composite reality image to be displayed on a display unit by combining the real-space cutout image and a virtual space image corresponding to the real-space cutout image, A control method for an information processing device, characterized by having the following features. [Program 1] A program for causing a computer to function as one of the means of an information processing device described in any one of configurations 1 to 25. [Explanation of Symbols]

[0122] 10: Information processing system, 100: HMD, 101: Imaging unit, 102: Measurement unit, 103: Display unit, 200: Information processing device, 201: Acquisition unit, 202: Estimation unit, 203: Setting unit, 204, 221: Judgment unit, 205: Cutting unit, 206: Holding unit, 207, 223: Generation unit, 221: Transmit / receive unit

Claims

1. An information processing device that generates a mixed reality image to present to an observer, An acquisition means for acquiring a real-space image obtained by imaging the real space, A determination means for determining whether or not to widen the field of view of the augmented reality image by the observer, depending on the observer's settings or the observer's augmented reality environment, Based on the determination result of the determination means, an extraction means extracts a real-space extracted image from the real-space image, A generation means for generating a composite reality image to be displayed on a display unit by combining the real-space cutout image and a virtual space image corresponding to the real-space cutout image, An information processing device characterized by having the following features.

2. The acquisition means acquires the real-space image for the left eye, which is captured by a left-eye imaging unit provided corresponding to the observer's left eye, and the real-space image for the right eye, which is captured by a right-eye imaging unit provided corresponding to the observer's right eye. The extraction means extracts the left eye's real-space extracted image from the left eye's real-space image and the right eye's real-space extracted image from the right eye's real-space image, based on the result of the determination by the determination means. The generating means is The real-world image for the left eye and the virtual-world image for the left eye corresponding to the real-world image for the left eye are combined to generate the composite reality image for the left eye to be displayed on the left-eye display unit. The real-world image for the right eye and the virtual-world image for the right eye corresponding to the real-world image for the right eye are combined to generate the composite reality image for the right eye that is displayed on the right-eye display unit. The information processing apparatus according to feature 1.

3. The extraction means extracts the real-space extraction image from the real-space image if the determination result of the determination means widens the field of view, so that the rendering range of the augmented reality image generated by the generation means is wider than if the determination result of the determination means does not widen the field of view. The information processing apparatus according to feature 1.

4. The acquisition means acquires the real-space image for the left eye, which is captured by a left-eye imaging unit provided corresponding to the observer's left eye, and the real-space image for the right eye, which is captured by a right-eye imaging unit provided corresponding to the observer's right eye. The extraction means extracts the left eye's real-space extracted image from the left eye's real-space image and the right eye's real-space extracted image from the right eye's real-space image, based on the result of the determination by the determination means. The cutting means is, If the determination result of the determination means does not widen the field of view, the real space cropped image for the left eye and the real space cropped image for the right eye are cropped so that the same range is depicted on a predetermined plane that is parallel to the alignment direction plane in the alignment direction of the left eye imaging unit and the right eye imaging unit, and is at a predetermined distance from the alignment direction plane. If the determination result of the determination means widens the field of view, the real-space cropped image for the left eye and the real-space cropped image for the right eye are cropped so that at least a portion of the predetermined surface depicts a different area. The information processing apparatus according to feature 1.

5. The acquisition means acquires the real-space image for the left eye, which is captured by a left-eye imaging unit provided corresponding to the observer's left eye, and the real-space image for the right eye, which is captured by a right-eye imaging unit provided corresponding to the observer's right eye. The extraction means extracts the left eye's real-space extracted image from the left eye's real-space image and the right eye's real-space extracted image from the right eye's real-space image, based on the result of the determination by the determination means. The cutting means is, If the determination result of the determination means does not widen the field of view, the left eye real space cropped image and the right eye real space cropped image are cropped so that the first identical range is depicted on a predetermined plane that is parallel to the alignment direction plane in the alignment direction of the left eye imaging unit and the right eye imaging unit, and is at a predetermined distance from the alignment direction plane. If the determination result of the determination means widens the field of view, the real-space cropped image for the left eye and the real-space cropped image for the right eye are cropped so that a second identical area wider than the first identical area is depicted on the predetermined surface. The information processing apparatus according to feature 1.

6. The aforementioned observer setting is a setting indicating whether or not the observer wants to widen the field of view of the augmented reality image. The information processing apparatus according to feature 1.

7. The observer's mixed reality environment is determined by whether or not the size of the virtual object used to generate the virtual space image is above a certain threshold. The information processing apparatus according to feature 1.

8. The observer's mixed reality environment is whether or not the virtual objects used to generate the virtual space image are within the observer's field of view. The information processing apparatus according to feature 1.

9. The observer's mixed reality environment is determined by whether or not the virtual objects used to generate the virtual space image remain within the observer's field of view for a period exceeding a certain threshold. The information processing apparatus according to feature 1.

10. When the determination result of the determination means widens the field of view, the extraction means changes the extraction range by a predetermined amount when extracting a series of real-space extraction images. The information processing apparatus according to feature 1.

11. The augmented reality environment of the observer is determined by whether the distance between the observer and other objects different from the observer is below a threshold. The information processing apparatus according to feature 1.

12. The aforementioned other object is a second observer, different from the aforementioned observer. The information processing apparatus according to feature 11.

13. The aforementioned other object is an object located at a specified position or in a designated area in the real space. The information processing apparatus according to feature 11.

14. The other object is a point cloud in the real space. The information processing apparatus according to feature 11.

15. The acquisition means acquires the real-space image for the left eye, which is captured by a left-eye imaging unit provided corresponding to the observer's left eye, and the real-space image for the right eye, which is captured by a right-eye imaging unit provided corresponding to the observer's right eye. The point cloud in the real space is calculated using the real space image for the left eye and the real space image for the right eye. The information processing apparatus according to feature 14.

16. The observer's mixed reality environment is whether or not the observer perceives other objects different from the observer. The information processing apparatus according to feature 1.

17. Whether or not the aforementioned other object is recognized depends on whether or not the aforementioned other object is within the observer's field of view. The information processing apparatus according to feature 16.

18. Whether or not the aforementioned other object is recognized is determined by whether or not the aforementioned other object is within the observer's field of view, which is calculated based on the position and orientation of the imaging unit that captures the real-space image. The information processing apparatus according to feature 16.

19. The generation means generates the virtual space image using the parameters used by the extraction means when obtaining the real space extraction image. The information processing apparatus according to feature 1.

20. The generation means enlarges or reduces the real-space cropped image to a size that can be displayed on the display unit. The information processing apparatus according to feature 1.

21. When the generation means obtains a real-space cropped image by cropping the real-space image based on the determination that the cropping means widens the field of view, it enlarges or reduces the real-space cropped image so that the sense of size of the mixed-reality space observed by the observer matches the sense of size of the real space. The information processing apparatus according to feature 1.

22. The generation means generates a warning mark in the virtual space image at a position where the observer can see it with both eyes, if the distance between the observer and another object other than the observer is below a threshold. The information processing apparatus according to feature 1.

23. The cropping means switches the cropping range for cropping the real-space image after the warning mark is displayed on the display unit. The information processing apparatus according to feature 22.

24. The observer's mixed reality environment is determined by whether or not the distance between the virtual object used to generate the virtual space image and the imaging unit that captures the real space image is below a threshold. The information processing apparatus according to feature 1.

25. The observer's mixed reality environment is determined by whether or not the distance between the virtual object used to generate the virtual space image and the imaging unit that captures the real space image fell below a threshold over a predetermined period of time. The information processing apparatus according to feature 1.

26. A method for controlling an information processing device that generates a mixed reality image to be presented to an observer, An acquisition step to obtain a real-space image obtained by imaging the real space, A determination step of determining whether or not to widen the field of view of the mixed reality image by the observer, depending on the observer's settings or the observer's mixed reality environment, Based on the result of the determination step, an extraction step is performed to extract a real-space extracted image from the real-space image, A generation step of generating a composite reality image to be displayed on a display unit by combining the real-space cutout image and a virtual space image corresponding to the real-space cutout image, A control method for an information processing device, characterized by having the following features.

27. A program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 25.

Images