System, method, program and display device

Abstract

To provide a system, a method, a program and a display device for properly superimposing and displaying a picked-up image and an image of a three-dimensional model.SOLUTION: A system includes a superimposition image generation part 423 for generating superimposition display data obtained by superimposing an image of a three-dimensional model and a picked-up image, a superimposition image positioning part 424 for performing positioning processing of an object included in the three-dimensional model and a subject included in the image, a superimposition image capture part 425 for capturing a superimposition image obtained by positioning the object and the position of the subject, and an image display part 401 for controlling a display that projects the image captured by the superimposition image capture part 425 onto a virtual sphere.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a system, a method, a program, and a display device for displaying an image. [Background technology]

[0002] With the development of information processing technology, the methods of displaying three-dimensional images have become more diverse. For example, a method of superimposing a three-dimensional computer graphic image on a photographed image of an actual object in order to compare the actual object with a three-dimensional model is known.

[0003] Regarding this point, Japanese Patent No. 5390377 (Patent Document 1) discloses a technology for superimposing an image of a three-dimensional model on an actual image in accordance with a relative position change in the field of view. According to Patent Document 1, it is possible to display an object in the field of view of an imaging device and a virtual three-dimensional computer graphics image as if they were integrated.

[0004] However, in conventional technologies such as Patent Document 1, when the image of the object to be superimposed on the three-dimensional computer graphics image is a celestial sphere image, if the position of the viewpoint for displaying the image moves, a deviation occurs between the position of the object in the three-dimensional model image and the position of the subject in the celestial sphere image. Therefore, visibility is reduced when comparing the real thing and the three-dimensional model, making it difficult to make an appropriate comparison.

[0005] For this reason, there was a demand for technology that could display the spherical image and the three-dimensional model image without any misalignment, even when the viewpoint for displaying the image moved. Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been made in consideration of the problems in the conventional technology described above, and has an object to display a captured image and an image of a three-dimensional model in an appropriate superimposed manner. [Means for solving the problem]

[0007] That is, according to the present invention, A generating means for generating superimposed display data in which the three-dimensional model and the captured image are superimposed; a registration processing means for performing registration processing between an object included in the three-dimensional model and a subject included in the image; a capture means for capturing a superimposed image in which an object included in the three-dimensional model and a subject included in the image are aligned by the alignment processing means; a display control means for controlling the image captured by the capture means to be projected and displayed on a virtual sphere; A system is provided that includes: Effect of the Invention

[0008] According to the present invention, a captured image and an image of a three-dimensional model can be appropriately superimposed and displayed. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of the hardware of an entire display system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a cross-sectional view of an image capturing device that constitutes the display system of the present embodiment. [Diagram 3] FIG. 2 is a diagram showing the hardware configuration of various devices constituting the display system of the embodiment. [Figure 4] FIG. 2 is a block diagram of software included in the display system of the present embodiment. [Diagram 5] 11A and 11B are diagrams showing an example of superimposing a wide-field-of-view image on an image of a three-dimensional model in the present embodiment. [Figure 6] 3A to 3C are diagrams showing examples of images displayed on the HMD in the embodiment. [Figure 7] 5A and 5B are diagrams showing examples of superimposed images viewed from inside a virtual sphere in the embodiment. [Figure 8]5A to 5C are diagrams showing examples of alignment in the present embodiment. [Figure 9] FIG. 13 is a diagram showing an example of a case where the viewpoint is moved after alignment in the prior art. [Figure 10] 4 is a flowchart showing a process executed by the display system of the embodiment. [Figure 11] 6A to 6C are diagrams showing examples of displaying a capture image in the embodiment. [Figure 12] 6A to 6C are diagrams showing an example of changing the transmittance of an image projected onto a virtual sphere in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the present invention will be described with reference to an embodiment, but the present invention is not limited to the embodiment described below. In addition, in each drawing referred to below, the same reference numerals are used for common elements, and the description thereof will be omitted as appropriate. In addition, in the following embodiment, a wide-field image (hereinafter referred to as a "wide-field image") having a wide viewing angle such as a 360-degree image (also called a 360-degree omnidirectional image, omnidirectional image, or omnidirectional image) in which the entire 360-degree surroundings are captured is illustrated and described, but the embodiment is not limited thereto, and may be, for example, a super-wide-angle panoramic image or an image captured 360 degrees around a horizontal plane.

[0011] 1 is a diagram showing a schematic configuration of the overall hardware of a display system 1 in this embodiment. The display system 1 includes a head mounted display (HMD) 10 that functions as a display unit worn on the user's head, a controller 11 that functions as an operation unit that the user holds in his / her hand or wears on his / her hand and operates, and a personal computer (PC) 12 that functions as an information processing device. The display system 1 also includes a position detection sensor 13 that functions as a detection unit that detects the position of the HMD 10 and the inclination of the HMD 10 relative to a reference direction, and the position of the controller 11 and the inclination of the controller 11 relative to a reference direction. The display system 1 also includes an image capture device 14 that captures a wide-field image.

[0012] The HMD 10, the controller 11, the PC 12, and the image capturing device 14 are connected to each other via a network 15 or a cable, etc., so that they can communicate with each other. The connection between each piece of hardware may be a wired connection, or a wireless connection such as Bluetooth (registered trademark) or WiFi (registered trademark). The HMD 10 and the position detection sensor 13 may also be wirelessly connected via WiFi (registered trademark) or the like. The exchange of image data between the PC 12 and the image capturing device 14 is not limited to the above-mentioned wired or wireless communication, and may be performed, for example, by various recording media.

[0013] The HMD 10 is a display device having a display for displaying images to the user, and displaying images on the display corresponding to the position of the HMD 10 and the inclination relative to a reference direction. The images are two images corresponding to the left and right eyes, respectively, to make the images appear three-dimensional using the parallax between the left and right eyes of the user. For this reason, the HMD 10 is provided with two displays for displaying images corresponding to the left and right eyes, respectively. The reference direction is, for example, any direction parallel to the floor. The HMD 10 has a light source such as an infrared LED (Light Emitting Diode) and emits infrared rays.

[0014] The controller 11 is an operating means that is held by the user in the hand or worn at the hand position, and has a button, a wheel, a touch sensor, etc., receives information input from the user, and transmits the received information to the HMD 10. The controller 11 also has an infrared LED light source and emits infrared rays. The controller 11 shown in Fig. 1 is shown in a form that is held by the user in the hand, but this is not particularly limited to the embodiment, and may be in the form of a glove equipped with a sensor, for example.

[0015] The position detection sensor 13 is disposed at an arbitrary position in front of the user, detects the position and inclination of the HMD 10 and the controller 11 with respect to a reference direction from infrared rays emitted from the HMD 10 and the controller 11, and outputs the position information and inclination information. The position detection sensor 13 is, for example, an infrared camera, and can detect the position and inclination of the HMD 10 and the controller 11 based on an image captured by the image capture device 14. Note that the HMD 10 and the controller 11 are provided with a plurality of light sources in order to detect the position and inclination of the HMD 10 and the controller 11 with high accuracy. The position detection sensor 13 is composed of one or more sensors, and when a plurality of sensors are used, they can be provided on the side, rear, etc.

[0016] Based on the position information and tilt information of the HMD 10 output from the position detection sensor 13, the position information of the controller 11, and, if necessary, the tilt information of the controller 11, the PC 12 generates a user object for supporting input from the user in the three-dimensional virtual space displayed on the display of the HMD 10. Based on the position information and tilt information of the HMD 10 and the three-dimensional virtual space display data displayed on the HMD 10, the PC 12 generates an image (images corresponding to the left and right eyes) of the user's viewing direction in the three-dimensional virtual space (more precisely, the tilt direction of the HMD 10), and executes a process of displaying the image on the display of the HMD 10. The PC 12 can also create three-dimensional models of various structures. The three-dimensional models created by the PC 12 are output as image data and can be displayed on the HMD 10.

[0017] In another preferred embodiment, the display system 1 may include a server device instead of the PC 12. That is, the display system 1 may be configured to execute the processing performed by the PC 12 in the described embodiment on the cloud and provide a service.

[0018] The imaging device 14 is a device that captures a plurality of wide-angle lens images, fisheye lens images, etc., and captures an image having a solid angle of 4π steradian (hereinafter referred to as a "wide-field image") centered on the imaging device 14. The detailed configuration of the imaging device 14 will be described later.

[0019] The display system 1 of this embodiment can display a wide-field image and an image of a three-dimensional model in a superimposed manner. For example, in the field of architecture, when comparing a three-dimensional model of the internal structure of a building to be constructed with the internal structure of an actually constructed building, a wide-field image can be taken inside the building and an image of the three-dimensional model can be superimposed and displayed to confirm whether a structure conforming to the three-dimensional model has been constructed. Note that this is only one form of use, and the present invention is not limited to this form of use. Therefore, for example, the present invention can be applied to the display of the internal structure of the human body (organs, skeleton, brain, etc.), the medical field such as surgery simulation, and other fields.

[0020] In the example shown in Fig. 1, the HMD 10 and the controller 11 have a light source and the position detection sensor 13 is arranged at an arbitrary position, but the HMD 10 and the controller 11 may have the position detection sensor 13 and a light source or a marker that reflects infrared rays may be arranged at an arbitrary position. When a marker is used, the HMD 10 and the controller 11 are provided with a light source and the position detection sensor 13, infrared rays emitted from the light source are reflected by the marker, and the reflected infrared rays are detected by the position detection sensor 13, so that the position and inclination of the HMD 10 and the controller 11 can be detected. In another preferred embodiment, the HMD 10 and the controller 11 can be configured to include an inertial sensor such as a gyro sensor or an acceleration sensor to detect the movement of the HMD 10 and the controller 11.

[0021] If there is an object between the position detection sensor 13 and the HMD 10 and controller 11, the infrared rays will be blocked, making it impossible to accurately detect the position and inclination. For this reason, when using the HMD 10 and controller 11 to perform operations or displays, it is desirable to do so in an open space.

[0022] In the example shown in Figure 1, a space is provided in which a user can wear an HMD 10, hold a controller 11 in their hand, and stretch or unstretch their arms, and a PC 12 and a position detection sensor 13 are placed outside the space.

[0023] In this embodiment, an example is shown in which the HMD 10 is used as the display device, but this is not intended to limit the embodiment. Therefore, an image may be displayed using a display device other than the HMD 10. For example, an image may be displayed on a flat panel display (FPD) or a display device 47 of the PC 12, which will be described later.

[0024] Fig. 2 is a cross-sectional view of the photographing device 14 constituting the display system 1 of this embodiment. The photographing device 14 shown in Fig. 2 includes an imaging body 141, a housing 142 that holds the imaging body 141 and components such as a controller and a battery, and a photographing button 143 provided on the housing 142. The photographing device 14 also includes various sensors such as an acceleration sensor and a geomagnetic sensor (not shown), and devices such as a communication I / F. The photographing device 14 is capable of acquiring posture data, transferring images to the PC 12, and the like, by the above-mentioned various devices.

[0025] The imaging body 141 shown in FIG. 2 includes two lens optical systems 144A and 144B and two imaging elements 145A and 145B. The imaging elements 145A and 145B are CMOS (Complementary Metal Oxide Semiconductor) sensors or CCD (Charge Coupled Device) sensors. The imaging elements 145A and 145B are arranged so that their imaging surfaces are opposite each other. The lens optical system 144 is configured as a fisheye lens with 6 groups and 7 elements or 10 groups and 14 elements, for example. In the embodiment shown in FIG. 2, the fisheye lens has a total angle of view larger than 180 degrees (=360 degrees / n; the number of optical systems n=2), and preferably has a total angle of view of 190 degrees or more. In the embodiment described, two fisheye lenses with a total angle of view of 180 degrees or more are used, but three or more lens optical systems and imaging elements may be included as long as a predetermined angle of view is obtained as a whole. In addition, in the embodiment described below, a fisheye lens is used, but as long as a specified overall angle of view is obtained, there is no obstacle to using another wide-angle lens or an ultra-wide-angle lens instead of the fisheye lens.

[0026] The optical elements (lenses, prisms, filters, and aperture stops) of the two lens optical systems 144A and 144B are positioned relative to the image sensors 145A and 145B. The optical elements of the lens optical systems 144A and 144B are positioned so that the optical axes of the optical elements are positioned perpendicular to the center of the light receiving area of ​​the corresponding image sensor 145, and the light receiving area is positioned as the image forming surface of the corresponding fisheye lens. In the embodiment described below, in order to reduce parallax, a bending optical system is adopted in which light collected by the two lens optical systems 144A and 144B is distributed to the two image sensors 145A and 145B by two 90-degree prisms, but the present invention is not limited to this, and a three-fold refraction structure may be used to further reduce parallax, or a straight optical system may be used to reduce costs.

[0027] In the embodiment shown in FIG. 2, the lens optical systems 144A and 144B are of the same specification and are combined in the opposite direction so that their optical axes coincide with each other. The image pickup elements 145A and 145B convert the light distribution of the received light into an image signal and sequentially output the images to an image processing block on the controller. As will be described in detail later, the images captured by the image pickup elements 145A and 145B are synthesized to generate a wide-field image. The wide-field image is an image of all directions that can be seen from the shooting point. The wide-field image can be captured as a still image or as a video.

[0028] 3 is a diagram showing a hardware configuration included in various devices constituting the display system 1 of this embodiment. The HMD 10 includes an external I / F 20, a CPU 21, a display 22, a memory 23, a HDD 24, a light source 25, and a microphone 26. The CPU 21 controls the entire HMD 10 and executes processes such as emission of light from the light source 25, communication with the outside, and display on the display 22. The external I / F 20 is an interface for communicating with the PC 12. The display 22 may be a liquid crystal display or an organic EL (Electro Luminescence) display.

[0029] The memory 23 provides a working area for the CPU 21. The HDD 24 stores three-dimensional virtual space display data to be displayed on the HMD 10, etc. The light source 25 is an infrared LED or the like, which emits infrared rays. The infrared rays can be emitted in a flashing manner in a predetermined pattern. The microphone 26 is a voice input device that allows the user to input by voice.

[0030] The controller 11 includes an operation I / F 30, an external I / F 31, and a light source 32. The operation I / F 30 is a button, a wheel, a touch sensor, etc., and is arranged on the outer surface of the controller 11 to enable operation by a user and to accept input of operation information. The external I / F 31 is wirelessly connected to the PC 12, and transmits operation information accepted by the operation I / F 30 to the PC 12. The light source 32 emits infrared rays so as to blink in a predetermined pattern. The light source 32 can be distinguished from the HMD 10 by blinking in a pattern different from the light source of the HMD 10.

[0031] The PC 12 includes a CPU 40 , a ROM 41 , a RAM 42 , a HDD 43 , an external I / F 44 , an input / output I / F 45 , an input device 46 , and a display device 47 .

[0032] The CPU 40 executes the process of controlling the entire PC 12, generating the above-mentioned user object, and generating an image in the user's viewing direction in the three-dimensional virtual space displayed on the HMD 10 to display on the display of the HMD 10. The ROM 41 stores a boot program for starting up the PC 12, firmware for controlling the HDD 43 and the external I / F 44, etc. The RAM 42 provides a working area for the CPU 40.

[0033] The HDD 43 stores an OS (Operating System), programs for executing the above processes, image data, etc. The external I / F 44 is connected to the network shown in Fig. 1 and communicates with other devices via the network. The external I / F 44 is also connected to the HMD 10 and the position detection sensor 13 via a cable or the like, and is wirelessly connected to the controller 11, and communicates with the HMD 10, the controller 11, and the position detection sensor 13.

[0034] The input device 46 is a mouse, a keyboard, etc., and accepts information input and operations from the user. The display device 47 provides a display screen for the user and displays information input by the user, processing results, etc. The input / output I / F 45 is an interface that controls the input of information from the input device 46 and the output of information to the display device 47.

[0035] The photographing device 14 includes an operation I / F 50, an external I / F 51, a photographing I / F 52, a sound collection I / F 53, and a storage I / F .

[0036] The operation I / F 50 is an interface for operating the imaging device 14, and may be, for example, various buttons, switches, etc. A user can operate the imaging device 14 via the operation I / F 50 to perform imaging of a wide-field image, transmission and reception of data, etc.

[0037] The external I / F 51 is an interface for communicating with other devices, and can be, for example, a network communication interface such as a NIC (Network Interface Card) or a port for connecting various connectors such as a USB. By using the external I / F 51, the imaging device 14 can communicate with other devices constituting the display system 1 directly via the network 15 or indirectly without via the network 15.

[0038] The imaging I / F 52 is an interface for capturing a wide-field image via the imaging body 141 .

[0039] The sound collection I / F 53 is an interface for recording sound together with an image when shooting a moving image. The sound collection I / F 53 can be a microphone, and may include multiple microphones depending on the embodiment.

[0040] The storage I / F 54 is an interface for storing the captured wide-field image. The wide-field image may be stored in a storage device such as a ROM included in the imaging device 14, or in an external storage device such as an SD card.

[0041] The above describes the hardware configuration included in each device constituting the display system 1 of this embodiment. Next, the functional means executed by each piece of hardware in this embodiment will be described with reference to FIG.

[0042] Fig. 4 is a software block diagram included in the display system 1 of this embodiment. As shown in Fig. 4, in the display system 1 of this embodiment, the PC 12 includes a three-dimensional model data acquisition unit 421, a wide-field image acquisition unit 422, a superimposed image generation unit 423, a superimposed image position adjustment unit 424, a superimposed image capture unit 425, and a storage unit 426. The HMD 10 includes an image display unit 401, and the image capture device 14 includes a wide-field image capture unit 441. Each functional unit will be described in detail below.

[0043] First, the functional means included in the PC 12 will be described. The three-dimensional model data acquisition unit 421 is a means for acquiring data of a three-dimensional model. The three-dimensional model data acquisition unit 421 of this embodiment can acquire three-dimensional model data from, for example, a storage unit 426. Note that, in the embodiment to be described, the three-dimensional model can be, for example, a BIM (Building Information Model) that represents the internal structure of a building, but this is not a particular limitation of the embodiment.

[0044] The wide-field-of-view image acquiring unit 422 is a means for acquiring a wide-field-of-view image captured by the photographing device 14. The wide-field-of-view image acquiring unit 422 of this embodiment may acquire the wide-field-of-view image by receiving it from the photographing device 14 via the network 15 or the like, or may acquire it from the storage unit 426. In addition, the wide-field-of-view image acquired by the wide-field-of-view image acquiring unit 422 may be a still image or a moving image.

[0045] The superimposed image generating unit 423 is a means for generating an image in which the three-dimensional model acquired by the three-dimensional model data acquiring unit 421 and the wide-field-of-view image acquired by the wide-field-of-view image acquiring unit 422 are superimposed. The superimposed image generating unit 423 of this embodiment may be configured to generate superimposed display data for displaying the superimposed image. Here, the superimposition of images performed by the superimposed image generating unit 423 of this embodiment will be described with reference to FIG. 5.

[0046] FIG. 5 is a diagram showing an example of superimposing a wide-field image on a three-dimensional model in this embodiment. FIG. 5(a) is an example of a wide-field image, and shows an image of an indoor space of a building photographed by a photographing device 14. The wide-field image can be projected onto a virtual sphere as shown in FIG. 5(b). By arranging such a sphere in the space of a three-dimensional model as shown in FIG. 5(c), the wide-field image can be superimposed onto the three-dimensional model. That is, by arranging the viewpoint position within the sphere and displaying the wide-field image projected onto the sphere with a certain transmittance, the wide-field image can be displayed as an image superimposed onto the three-dimensional model.

[0047] The superimposed image generating unit 423 of this embodiment can generate a superimposed image as shown in FIG. 5(c).

[0048] In the example of superimposition shown in Fig. 5, an example in which a sphere onto which a wide-field image is projected is placed within a three-dimensional model has been described, but this does not particularly limit the embodiment. Therefore, for example, a sphere onto which a three-dimensional model is projected may be placed within the omnidirectional image to superimpose the omnidirectional image and the three-dimensional model. In addition, a three-dimensional model whose transmittance can be changed may be placed within the omnidirectional image to superimpose the omnidirectional image and the three-dimensional model.

[0049] Returning to FIG. 4 for the explanation, the superimposed image alignment unit 424 is a means for performing alignment processing between a subject in a wide-field-of-view image and an object in an image of a three-dimensional model in a superimposed image of the wide-field-of-view image and the three-dimensional model. The alignment processing in this embodiment may be performed manually by a user's operation, or may be performed automatically based on the feature amounts of the object and the subject. Note that when the alignment processing is performed automatically, it may be started by a user's operation, for example. Furthermore, when the alignment adjustment is completed, the superimposed image alignment unit 424 may notify the completion of the alignment processing by sound or display.

[0050] The superimposed image capture unit 425 is a means for capturing the superimposed image aligned by the superimposed image alignment unit 424. By capturing the superimposed image, even if the viewpoint moves, it is possible to prevent the wide-field image and the image of the three-dimensional model from being displayed out of alignment, and it is possible to display an image with improved visibility. Moreover, the capture of the superimposed image is a process similar to a so-called screenshot, and can be performed by acquiring an image from a point where a virtual camera is placed in the virtual space, that is, an image that can be seen from the point where the virtual camera is placed, and texturing the image. Here, texturing refers to making the image into a state where it can be pasted as an image on the surface of a virtual sphere for projection.

[0051] The storage unit 426 is a means for controlling the operation of the HDD 43 of the PC 12 and storing various data, etc. The storage unit 426 of this embodiment can store wide-field images, three-dimensional model data, superimposed images, and the like.

[0052] Next, the functional means included in the HMD 10 will be described. The image display unit 401 is an image display control means that controls the operation of the display 22 and controls the display of images. The image display unit 401 of this embodiment can display images of a three-dimensional virtual space. The image display unit 401 is also configured as a means for acquiring various images such as a wide-field image, three-dimensional model data, a superimposed image, and a captured image, and controls the projection display of the acquired various images. The various images displayed by the image display unit 401 can be acquired, for example, from the PC 12 or the imaging device 14. Furthermore, the image display unit 401 of this embodiment can acquire display data aligned by the superimposed image alignment unit 424, and project and display it on the display 22.

[0053] Next, the functional means included in the photographing device 14 will be described. The wide-field image photographing section 441 is a means for photographing a wide-field image by two image pickup elements 145A and 145B. The wide-field image photographed by the wide-field image photographing section 441 is provided to the PC 12 via the network 15, a communication cable, various storage media, etc. The wide-field image may be provided from the photographing device 14 to the PC 12 in real time, or may be temporarily stored in the photographing device 14 and then transmitted.

[0054] The above-mentioned software blocks correspond to functional means realized by the CPU executing the program of the present embodiment and causing each piece of hardware to function. The functional means shown in each embodiment may be realized entirely in software, or part or all of them may be implemented as hardware that provides the same functions.

[0055] Furthermore, all of the above-mentioned functional means do not necessarily have to be included in the configuration shown in Fig. 4. For example, in another preferred embodiment, each functional means may be realized by cooperation of each device. Also, for example, the image display unit 401 included in the HMD 10 in Fig. 4 may be included in the PC 12.

[0056] Fig. 6 is a diagram showing an example of an image displayed on the HMD 10 in this embodiment. Fig. 6(a) shows an example in which the user wears the HMD 10 on his / her head, and Fig. 6(b) shows an image of a three-dimensional virtual space displayed on the HMD 10 and visually recognized by the user of Fig. 6(a).

[0057] The display area of ​​the image of the three-dimensional virtual space displayed on the HMD 10 changes in accordance with the movement of the head of the user wearing the HMD 10. For example, when the user moves his / her neck from side to side as shown by the arrow in Fig. 6(a), the area of ​​the displayed three-dimensional space also moves left and right as shown by the arrow in Fig. 6(b). In the same way, the display area can also move up and down in conjunction with the movement of the user's head.

[0058] The display area can also be manipulated by the controller 11, allowing the user to freely change the size, viewpoint, etc. of the display area.

[0059] Note that the display screen shown in Fig. 6(b) is an example and does not limit the embodiment. Therefore, other than what is shown in Fig. 6(b) may be displayed, and may include various display objects such as icons, buttons, cursors, etc. for performing operations on the screen display.

[0060] FIG. 7 is a diagram showing an example of a superimposed image seen from inside a virtual sphere in this embodiment. Here, when explaining FIG. 7, FIG. 5 is appropriately referred to. Note that the superimposed image in the exemplary embodiment to be explained is an image in which a virtual sphere onto which a wide-field-of-view image is projected is arranged in the three-dimensional virtual space of the three-dimensional model data. However, this is not intended to limit the embodiment, and for example, an image in which a virtual sphere onto which an image of three-dimensional model data is projected is arranged in the three-dimensional virtual space of the wide-field-of-view image may be used as the superimposed image. In addition, an image in which a three-dimensional model capable of changing the transmittance is arranged in the wide-field-of-view image may be used as the superimposed image.

[0061] Consider the configuration as shown in FIG. 5(c) in which a virtual sphere (see FIG. 5(b)) onto which a wide-field image is projected is placed in a three-dimensional virtual space of three-dimensional model data (see FIG. 5(a)). A user wearing an HMD 10 can move the position of a viewpoint in the three-dimensional virtual space by operating a controller 11 or the like. Here, when the user performs an operation to move the position of the viewpoint in the three-dimensional virtual space and the position of the viewpoint enters the inside of the virtual sphere, the user can see both the three-dimensional model and the wide-field image. At this time, the wide-field image projected onto the virtual sphere is displayed transparently, so that the user can simultaneously see the wide-field image and the image of the three-dimensional model on one display screen in the HMD 10.

[0062] Fig. 7(a) shows a superimposed image seen from inside a virtual sphere, and the virtual sphere onto which the wide-field-of-view image is projected is displayed with a certain transmittance. In the example shown in Fig. 7(a), a pillar is displayed as a subject of the wide-field-of-view image and a pillar is displayed as an object of the three-dimensional model. In Fig. 7(a), the position and angle of the virtual sphere are not appropriate, so the subject and object representing the same pillar are displayed out of alignment. Therefore, when a user tries to compare the three-dimensional model and the wide-field-of-view image, visibility is low and it is difficult to make an appropriate comparison.

[0063] The user can adjust the display so that the subject of the wide-field image overlaps with the object of the three-dimensional model corresponding to the subject by operating the HMD 10 and the controller 11. The display can be adjusted by adjusting the position and angle at which the virtual sphere is placed, the zoom in and out of the display, etc.

[0064] When the user performs an operation on the superimposed image and adjusts it so that the wide-field image and the three-dimensional model overlap, the display becomes as shown in Fig. 7(b). That is, when the user performs an appropriate adjustment, subjects such as pillars, windows, floors, ceilings, and walls in the wide-field image and objects such as pillars, windows, floors, ceilings, and walls that constitute the three-dimensional model corresponding to each subject are displayed overlapping each other, as shown in Fig. 7(b). In the embodiment described below, the adjustment so that the subjects and objects are displayed overlapping each other as shown in Fig. 7(b) is referred to as "alignment".

[0065] As shown in Fig. 7(b), a superimposed image in which the subject of the wide-field image and the object of the 3D model are superimposed is displayed, making it easier for the user to compare the two images. For example, if there is a building constructed based on a 3D model and the user wants to check whether the internal structure of the building is constructed according to the 3D model, the user can easily recognize the difference between the 3D model and the actual building by checking the superimposed image aligned as shown in Fig. 7(b).

[0066] The alignment process in this embodiment will be described below. Fig. 8 is a diagram showing an example of alignment in this embodiment. For example, consider a case where a user is trying to see a cube-shaped object 1 and a spherical object 2 in a superimposed image of a three-dimensional virtual space, as shown in Fig. 8(a).

[0067] In the case of Fig. 8(a), if the alignment process is not performed, the subject and the object are displayed out of alignment as shown in Fig. 8(b), making it difficult for the user to compare the wide-field image with the three-dimensional model.

[0068] Therefore, the user performs a process of aligning the subject with the object by performing operations on the superimposed image using the HMD 10, the controller 11, etc. The process of aligning the subject with the object can be performed by changing the coordinates of the virtual sphere (for example, matching the viewpoint position with the center position of the virtual sphere), rotating the virtual sphere (for example, rotating the virtual sphere in the yaw, roll, or pitch direction), enlarging or reducing the wide-field image (for example, changing the diameter of the virtual sphere), etc.

[0069] When the display is appropriately adjusted and the alignment process is performed, a superimposed image as shown in Fig. 8(c) is displayed on the HMD 10. In the superimposed image shown in Fig. 8(c), the object of the object 1 in the three-dimensional model and the object 1 included in the wide-field-of-view image are displayed overlapping each other. This makes it easier for the user to compare the three-dimensional model with the wide-field-of-view image. Therefore, if there is some defect (e.g., incorrect dimensions or a different shape) in the object 1 in the wide-field-of-view image, the user will be able to easily recognize the defect.

[0070] However, even if the positioning is performed as shown in Fig. 8, if the viewpoint position or line of sight is subsequently moved, a misalignment may occur between the object and the subject displayed in the three-dimensional virtual space. Therefore, for example, when checking the internal structure of a building, if the user moves his / her line of sight to check another object after aligning the position based on a pillar, the object and the subject will be misaligned, making it difficult to check the object properly.

[0071] The movement of the viewpoint after the alignment will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of the movement of the viewpoint after the alignment in the conventional technology. Fig. 9 shows an example of the movement of the viewpoint position or the line of sight direction by the user's operation after the alignment as shown in Fig. 8(c).

[0072] Fig. 9(a-1) shows an example of a case where the user's viewpoint position moves after the positioning is performed as shown in Fig. 8(c) in the three-dimensional virtual space of Fig. 8(a). That is, the user in Fig. 9(a-1) looks at a cube-shaped object 1 as in Fig. 8, but looks at the object 1 from a position moved from the viewpoint position of Fig. 8(a). In this case, even if the positioning is performed as shown in Fig. 8(c), the wide-field image projected by the virtual sphere and the image of the three-dimensional model in the three-dimensional virtual space are misaligned due to the subsequent movement of the viewpoint position, so that the subject and the object are displayed misaligned as shown in Fig. 9(a-2).

[0073] Also, Fig. 9(b-1) shows an example of a case where the user's viewpoint position moves after the positioning is performed as shown in Fig. 8(c) in the three-dimensional virtual space of Fig. 8(a). That is, the user in Fig. 9(a-1) moves the viewpoint where the HMD 10 image is displayed from the object 1 side to the object 2 side. In this case, even if the positioning is performed as shown in Fig. 8(c), the wide-field image projected by the virtual sphere and the image of the three-dimensional model in the three-dimensional virtual space are misaligned due to the subsequent movement of the viewpoint position (for example, even if the direction of the line of sight is simply rotated, the viewpoint position may move due to the movement of the head), so that the subject and the object are displayed misaligned as shown in Fig. 9(b-2) in the same way as in Fig. 9(a-2).

[0074] In this way, even if the positioning process is performed appropriately, if the viewpoint position moves even slightly after that, the positioning will be misaligned, and the object and the subject will be displayed misaligned, reducing the visibility of the superimposed image. Therefore, the display system 1 of this embodiment captures the superimposed image as a wide-field image in a state where the positioning process is performed, and projects the captured image onto a virtual sphere and displays it on the HMD 10. As a result, an image that maintains the positioning state is displayed on the HMD 10, and even if the viewpoint position is moved thereafter, there will be no misalignment between the object and the subject, making it easier to perform an appropriate comparison. Note that by performing the capture process continuously, it is possible to project and display a captured image that follows the viewpoint movement.

[0075] Next, the process executed in this embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the process executed by the display system 1 of this embodiment. Fig. 10 illustrates the process executed when comparing a wide-field image with a three-dimensional model corresponding to the space in which the wide-field image was captured. The display system 1 starts the process from step S1000.

[0076] In step S1001, the three-dimensional model data acquisition unit 421 acquires three-dimensional model data. The acquired three-dimensional model data is output to the superimposed image generation unit 423. Next, in step S1002, the wide-field image acquisition unit 422 acquires a wide-field image. The wide-field image acquired by the wide-field image acquisition unit 422 is output to the superimposed image generation unit 423. Note that the processing of steps S1001 and S1002 does not necessarily have to be performed in the order shown in FIG. 10. For example, the processing may be performed in the reverse order to that of FIG. 10, or the two processing may be performed in parallel.

[0077] Next, in step S1003, the superimposed image generating unit 423 generates a superimposed image from the three-dimensional model data and the wide-field-of-view image acquired in steps S1001 and S1002. In this embodiment, the superimposed image generating unit 423 generates a superimposed image by placing a virtual sphere onto which the wide-field-of-view image is projected in a three-dimensional virtual space based on the three-dimensional model data. The generated superimposed image is output to a display device such as the HMD 10 and displayed by various display means, allowing the user to view it.

[0078] Thereafter, in step S1004, the superimposed image alignment unit 424 performs alignment processing between the object constituting the three-dimensional model and the subject included in the wide-field image. The alignment processing may be performed by the user's operation while the user is viewing the superimposed image, or may be performed automatically based on the feature amount of the object. Here, the alignment processing can be performed by adjusting parameters such as the coordinate position, angle, and size of the virtual sphere. For example, when performing alignment manually by the user's operation, the amount of movement may be adjusted while displaying and referring to an arrow (vector) indicating the movement from the original position, or the amount of movement may be adjusted by referring to the coordinate position indicated by a numerical value. In addition, during alignment, the transmittance may be changed during the movement to make it easier to see.

[0079] After performing the positioning process in step S1004, in step S1005, the superimposed image capture unit 425 captures the superimposed image. The captured superimposed image (hereinafter referred to as the "captured image") is also configured in the form of a wide-field image with a solid angle of 4π steradians. The generated captured image is output to the HMD 10, but may be stored in the storage unit 426.

[0080] Next, in step S1006, the image display unit 401 projects the captured image onto the virtual sphere and displays it on the HMD 10. This allows the user to compare the wide-field image with the image of the three-dimensional model by viewing the captured image in step S1007.

[0081] Then, in step S1008, the display system 1 ends the process.

[0082] Fig. 11 is a diagram showing an example of displaying a captured image in this embodiment. Fig. 11 shows an example of displaying an image captured after performing the position alignment process for the superimposed images as shown in Fig. 7(b).

[0083] The images shown in Figures 11(a) to (c) are images captured after the alignment, with the viewpoint moved. As shown in Figures 11(a) to (c), even if the viewpoint moves, the image is projected and displayed while maintaining the alignment between the object and the subject, as in Figure 7(b).

[0084] In addition, in this embodiment, in order to make it easier to compare the image of the three-dimensional model with the wide-field image, the transmittance of the image projected onto the virtual sphere can be changed. Fig. 12 is a diagram showing an example of changing the transmittance of the image projected onto the virtual sphere in this embodiment. Fig. 12(a) is an example of a superimposed image when the transmittance of the virtual sphere is set to 100%, Fig. 12(b) is an example of a superimposed image when the transmittance of the virtual sphere is set to 28%, and Fig. 12(c) is an example of a superimposed image when the transmittance of the virtual sphere is set to 0%.

[0085] As shown in FIG. 12(a), when the transmittance of the virtual sphere is set to 100%, the wide-field image projected onto the virtual sphere is not displayed, and only the image of the three-dimensional model is displayed.

[0086] As shown in Fig. 12(b), when the transmittance of the virtual sphere is set to 28%, the wide-field image projected onto the virtual sphere is displayed transparently and superimposed on the image of the 3D model. In Fig. 12(b), the pillars and ceiling patterns included in the wide-field image are displayed relatively unclearly.

[0087] As shown in Fig. 12(c), when the transmittance of the virtual sphere is set to 0%, only the wide-field image projected onto the virtual sphere is displayed, and the image of the 3D model is not displayed. Therefore, the pillars, ceiling patterns, etc., included in the wide-field image are displayed clearly.

[0088] The transmittance can be changed arbitrarily by a user operation. Therefore, by viewing the superimposed image with the transmittance changed, the user can easily recognize the difference between the three-dimensional model and the wide-field image.

[0089] The change in transmittance as shown in Fig. 12 can also be applied to the virtual sphere constituting the captured image. That is, the captured image of this embodiment is composed of an aligned three-dimensional virtual space and a virtual sphere, and changing the transmittance of the virtual sphere constituting the captured image can eliminate the misalignment of objects and facilitate the comparison between the wide-field image and the image of the three-dimensional model.

[0090] As described above, according to the embodiment of the present invention, a captured image and a three-dimensional model can be appropriately superimposed and displayed.

[0091] Each function of the above-described embodiments of the present invention can be realized by a device-executable program written in C, C++, C#, Java (registered trademark), etc., and the program of this embodiment can be stored on a device-readable recording medium such as a hard disk drive, CD-ROM, MO, DVD, flexible disk, EEPROM (registered trademark), EPROM, etc. and distributed, and can also be transmitted over a network in a format that can be used by other devices.

[0092] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0093] Although the present invention has been described above with reference to an embodiment, the present invention is not limited to the above-described embodiment, and as long as the functions and effects of the present invention are achieved within the scope of embodiments that a person skilled in the art can imagine, they are included in the scope of the present invention. [Explanation of symbols]

[0094] 1… represents a system, 10… HMD, 11… controller, 13… position detection sensor, 14… imaging device, 15… network, 20… external I / F, 21… CPU, 22… display, 23… memory, 24… HDD, 25… light source, 26… microphone, 30… operation I / F, 31… external I / F, 32… light source, 40… CPU, 41… ROM, 42… RAM, 43… HDD, 44… external I / F, 45… input / output I / F, 46… input device, 47… display device, 50… operation I / F, 51… external I / F, 52… imaging I / F, 53… sound collection I / F, 54… memory I / F, 141… imaging element, 142… housing, 143… imaging button, 144… lens optical system, 145… imaging device, 401… image display unit, 421… three-dimensional model data acquisition unit, 422… wide-field image acquisition unit, 423… superimposed image generation unit, 424… superimposed image alignment unit, 425… superimposed image capture unit, 426… storage unit, 441… wide-field image imaging unit

Prior Art Documents

Patent Documents

[0095]

Patent Document 1

Claims

1. A generating means for generating superimposed display data in which the three-dimensional model and the captured image are superimposed; a registration processing means for performing registration processing between an object included in the three-dimensional model and a subject included in the image; a capture means for capturing a superimposed image in which an object included in the three-dimensional model and a subject included in the image are aligned by the alignment processing means; a display control means for controlling the image captured by the capture means to be projected and displayed on a virtual sphere; Including, the system.

2. The generating means includes: generating a superimposed image in which a virtual sphere onto which the image captured by the capture means is projected is placed in a three-dimensional virtual space related to the three-dimensional model; The system of claim 1 .

3. The position alignment processing means includes: The alignment process is performed by a user operation. The system of claim 1 .

4. The position alignment processing means includes: The method is characterized in that a position matching process is performed by detecting an object included in the three-dimensional model and a subject corresponding to the object. The system of claim 1 .

5. The generating means is capable of changing a transmittance of the image projected onto the virtual sphere. The system of claim 1 .

6. generating superimposed display data in which the three-dimensional model and the captured image are superimposed; A step of performing a registration process between an object included in the three-dimensional model and a subject included in the image; capturing an overlay image in which an object included in the three-dimensional model is aligned with a subject included in the image; a step of controlling the image captured in the capturing step to be projected and displayed on a virtual sphere; A method comprising:

7. A program executed by an information processing device that displays an image, generating superimposed display data in which the three-dimensional model and the captured image are superimposed; A step of performing a registration process between an object included in the three-dimensional model and a subject included in the image; capturing an overlay image in which an object included in the three-dimensional model is aligned with a subject included in the image; a step of controlling the image captured in the capturing step to be projected and displayed on a virtual sphere; program.

8. A display device for displaying an image on a display, An acquisition means for acquiring an image capturing a superimposed image in which an object included in a three-dimensional model and a subject included in the image are aligned; a display control means for controlling the projection and display of the image acquired by the acquisition means on the display; A display device comprising:

Images