Information presentation system

Abstract

To provide an information presentation system for presenting information of an omnidirectional image not only in a virtual space but also in a real space in front of a visual field at a time.SOLUTION: Image information or video information and software for display are introduced from a personal computer 14 to a memory 18 in a HMD16 mounted on the head part of an observer. The HMD16 includes not only the memory 18 but also the CPU20, the controller 22, the image display apparatus 24, and the like, and the memory 18, the controller 22, and the image display apparatus 24 are connected to the CPU20.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information presentation system for a display that expands a person's field of view, and is an information presentation system suitable for a head-mounted display or an external display. [Background technology]

[0002] As conventional methods for expanding the field of view, various methods have been reported that present real-world images or virtual space images taken with a 360-degree panoramic camera on a head-mounted display (hereinafter referred to as HMD) or a desktop computer display.

[0003] The technique of presenting an image captured by a hemispherical fisheye lens as a circular fisheye image or as a rectangular image using equirectangular projection, etc., has been well known for some time, and it is possible to capture and present the image using a camera with a built-in fisheye lens. However, with images created using this technique, only information within the hemisphere in front of the human field of vision can be seen at one time.

[0004] In response to this, methods for expanding the field of view beyond the hemisphere include, for example, methods such as those described in the following non-patent documents 1, 2, and 4, which present a horizontally elongated image without distortion, and methods such as those described in the following non-patent document 3, which distorts and compresses the area of ​​the image presented in the peripheral vision before presenting it.

[0005] Specifically, Non-Patent Document 1 discloses a system in which two cameras, each with a fisheye lens, are installed above the head in opposing positions to capture information about a celestial sphere image, and the image is displayed on an HMD as a horizontally long panoramic image.

[0006] Non-patent document 2 shows a system that displays information in a wide-angle image obtained using a lens that can acquire information about the surroundings up to 238 degrees, by compressing the image area that mainly shows the peripheral vision on both sides, and fitting the image into an HMD with a 90-degree field of view.

[0007] Non-patent document 3 devised a field of view expansion method in which the area in which a 360-degree image is presented in the peripheral vision is reduced horizontally in three different ways, converted into an image of a 200-degree range of the visual field, and displayed on three external displays, and showed that an experiment was conducted to evaluate at what position in the visual field a target object moving horizontally could be recognized.

[0008] Non-patent document 4 describes a system that uses a wide-angle HMD with a diagonal viewing angle of 200 degrees, and presents images expanded to 160 degrees, 280 degrees, or 360 degrees on the horizontal screen of the HMD based on image information of the real space acquired by two 360-degree omnidirectional cameras. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] J. Ardouin, A. Lecuyer, M. Marchal, C. Riant, E. Marchand, “FlyVIZ: A Novel Display Device to Provide Humans with 360° Vision by Coupling Catadioptric Camera with HMD”, Proceedings of ACM Symposium on Virtual Reality Software and Technology (VRST'12), Pages 41-44 (2012) [Non-patent document 2] J. Orlosky, Q. Wu, K. Kiyokawa, H. Takemura, C. Nitschke, “Fisheye Vision: Peripheral Spatial Compression for Improved Field of View in Head Mounted Displays”, Proceedings of the 2nd ACM Symposium on Spatial User Interaction (SUI '14), Pages 54-61 (2014) [Non-patent document 3] Takeshi Hayata, Munetoshi Iwakiri, “A Study on Visual Field Expansion by Displaying Extra-Visual Space in Peripheral Vision,” IPSJ Technical Report, Vol. 2015-CG-158, No. 10, Pages 1-5 (2015) [Non-patent document 4] T. Koishi, K. Watanuki, K. Kaede, “Effects of Field-of-View Expansion Using a Wide-field HMD on Active Linear Motion”, Affective and Pleasurable Design, AHFE 2022 International Conference, AHFE Open Access, Vol. 41, Pages 203-210 (2022) Summary of the Invention [Problem to be solved by the invention]

[0010] However, in Non-Patent Document 1, when viewing with a normal HMD with a horizontal viewing angle of about 90-120 degrees, if information about a celestial sphere image is included in the image, the image becomes horizontally long and it is difficult to visually recognize objects. Furthermore, when viewing information about a celestial sphere image included in an HMD with a wider viewing angle, it is necessary to present information to the person's peripheral vision, making it difficult to immediately visually recognize the information about the celestial sphere image.

[0011] The methods in Non-Patent Document 2 and Non-Patent Document 3 did not allow the user to view information about a celestial sphere image, including information behind the user, at once within a field of view close to the central field of view. The method in Non-Patent Document 4 also uses peripheral vision to present the information to an HMD with a wide field of view, but it is thought to be difficult to instantly recognize objects in the peripheral field of view, and therefore, in order to apply this technology, it is necessary to use an HMD with a wide field of view.

[0012] As such, none of the conventional methods or systems can present spherical image information in virtual space at once using images within a circular area, and as a result, it is difficult to quickly and smoothly recognize objects and take subsequent actions (for example, when you find the object, shoot, capture, run away, etc.).

[0013] The present invention has been made in view of the above background, and has a first object to provide an information presentation system that presents information on omnidirectional images in not only virtual space but also real space in front of the user's field of view at once. The second object is to provide an information presentation system that supports fast and smooth performance of actions and operations for each purpose following the discovery of an object using information on the omnidirectional image presented by the information presentation system achieved by the first object, and is effective in determining the position of the discovered object in three-dimensional space. [Means for solving the problem]

[0014] The invention described in claim 1 that solves the above problem provides an image in front of and an image behind an observer in a virtual space or a real space, Using the equidistant projection method of a fisheye lens, which is proportional to the distance from the center of the screen, The control means combines the image in front of the observer and the image behind the observer into a single virtual image that is a fisheye image of a circular area in front of the observer's field of view, This information presentation system projects this virtual image onto an image display device that is visible to the observer.

[0015] According to the invention of claim 1, the control means converts images in front of and behind the observer in a prepared virtual space or real space into a single virtual image that is a fisheye image of a circular area in front of the observer's field of view using an equidistant projection method of a fisheye lens proportional to the distance from the center of the screen. As a result, this virtual image can be projected onto an image display device that can be viewed by the observer.

[0016] Therefore, according to the information presentation system of the present claimed invention, by presenting a single image of a circular area in front of the observer's field of view as a virtual image that fits on the screen of an image display device such as an HMD, it is possible to present the observer with information on a celestial sphere image, which is a 360-degree area around the circular area, at once, and thereby enable the observer to quickly view and find objects that may exist in all directions.

[0017] According to the information presentation system of the invention of claim 2, the image display device is installed in a head-mounted display that can be worn by the observer. Accordingly, when the observer plays a game, the display is made into a head-mounted display that allows the observer to be more immersed in the game, thereby making the game more enjoyable.

[0018] According to the information presentation system of the invention of claim 3, auxiliary lines that are effective for determining the position of an object in the three-dimensional space in the virtual image are arranged in the virtual image. The virtual image has an advantage over the normal view in terms of quickly finding objects in space, due to the use of a fisheye lens' equidistant projection method, which is proportional to the distance from the center of the screen. However, since areas far from the center of the screen appear distorted, it is not necessarily easy to instantly and accurately identify the location of an object in that area in 3D space, although this varies from person to person. Therefore, auxiliary lines are placed within the virtual image to support actions after object discovery, allowing users to quickly and smoothly perform actions and operations specific to their purpose after object discovery.

[0019] According to the information presentation system of the invention of claim 4, the auxiliary line is a ring with a diameter twice the height from the observer's viewpoint to the ground and a lower end that touches the ground, with the observer's viewpoint as the center. In other words, in accordance with the use of the equidistant projection method of the fisheye lens proportional to the distance from the center of the screen, the auxiliary line is arranged to enable quick object discovery, but is a ring with a diameter twice the height from the observer's viewpoint to the ground, which is large enough to enable quicker discovery. Here, the observer's viewpoint means the point in the middle of both eyes.

[0020] According to the information presentation system of the invention of claim 5, the auxiliary lines are four rings arranged at 45-degree intervals from the observer's viewpoint. Accordingly, by arranging four rings, which is the optimum number for the observer to visually recognize, at equal intervals of 45 degrees, not only can the observer find objects more quickly, but also the observer can perform actions and operations for each purpose more quickly and smoothly.

[0021] According to the information presentation system of the invention of claim 6, the system includes a function of displaying auxiliary lines, a function of resetting the display position of the auxiliary lines, and a function of switching to the normal view. Accordingly, by providing a group of basic functions such as the display function, the function of resetting the display position of the auxiliary lines, and the function of switching to the normal view, it is possible to further support the observer's actions after discovering an object. [Effects of the Invention]

[0022] As described above, the first object of the present invention has the excellent effect of providing an information presentation system that presents information on omnidirectional images in not only virtual space but also real space in the front field of view at once. Similarly, according to the second object of the present invention, it is possible to provide an information presentation system that supports fast and smooth performance of actions and operations for each purpose following the discovery of an object using information from a celestial sphere image presented by the information presentation system according to the first object, and that is effective in determining the position of the discovered object in three-dimensional space. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram showing the concept of a field of view related to an information presentation system according to a first embodiment of the present invention. [Figure 2] FIG. 10 is a diagram illustrating an equidistant projection method. [Figure 3] FIG. 1 is a block diagram of an information presentation system according to a first embodiment of the present invention. [Figure 4]1A and 1B are images showing a virtual space by an information presentation system according to a first embodiment of the present invention, in which FIG. 1A shows an image of the virtual space as seen in a normal view, and FIG. 1B shows an image showing information of a spherical image by a spherical view. [Figure 5] 3 is a conceptual diagram showing an example of basic positions of auxiliary lines arranged in all directions of the observer in the information presentation system according to the first embodiment of the present invention. FIG. [Figure 6] FIG. 10 illustrates an example of a depicted image of a ring as seen through a spherical view. [Figure 7] FIG. 3 is a flowchart showing a process for generating and displaying a display image according to the first embodiment of the present invention. [Figure 8] FIG. 10 is a diagram showing a flow chart when addition of an auxiliary line is selected according to the first embodiment of the present invention. [Figure 9] These are diagrams seen through a spherical view, where (A) shows a sphere discovered diagonally to the rear left in the spherical view, (B) shows a laser beam being emitted in a state facing straight to the side in the spherical view, and (C) shows a laser beam being emitted after switching to the normal view after the sphere has been discovered. [Figure 10] 1A and 1B are diagrams showing how an observer irradiates a laser beam, in which (A) shows how the laser beam is irradiated diagonally backward to the left, and (B) shows how the laser beam is irradiated directly behind. [Figure 11] FIG. 10 is a block diagram of an information presentation system according to a second embodiment of the present invention. [Figure 12] FIG. 10 is a diagram showing a flow diagram of image generation and display processing for real space according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0024] A first embodiment of an information presentation system according to the present invention will be described in detail below with reference to the accompanying drawings. Fig. 1 shows a concept of the field of view of information presentation system 10 according to this embodiment. A bird's-eye view, which is a plan view, is shown at the bottom of Fig. 1, and a circle, the cross section of which is taken at the maximum diameter part of a sphere W, is virtually placed around a person who is an observer P in this plan view, and information on a 360-degree spherical image around the observer P, which is information on everything in front of and behind the observer P, is displayed on this sphere W. Here, the spherical image refers to an image obtained by capturing or drawing the entire sphere including 360 degrees horizontally and 180 degrees vertically.

[0025] The hemisphere including the solid line above this sphere W represents the hemispherical region FH in front of the person, and the hemisphere including the dotted line below the sphere W represents the hemispherical region RH behind the person. The image representation area is also shown at the top of Figure 1, and an image is projected from the front hemispherical region FH onto the inner circular region indicated by the solid line L1 at a scale proportional to the distance. Furthermore, an image is projected from the rear hemispherical region RH onto the outer circular region indicated by the dotted line L2, also at a scale proportional to the distance. This makes it possible to prepare, for example, images in front of and behind the observer P in virtual space. The solid line L1 and dotted line L2 represent lines seen from angles of 45 degrees, 90 degrees, 135 degrees to the left and right, and 180 degrees directly behind, when the angle in front of the observer P is 0 degrees.

[0026] The spherical image (i.e., all information in front and behind) displayed on the spherical surface W is rendered as described above in the circular area formed by the entire circular area indicated by the solid line L1 and the annular area indicated by the dotted line L2 shown in the image representation area in Figure 1. Accordingly, by projecting this virtual image onto the HMD 16 shown in Figure 3 that is visible to the observer P, the observer P can view the image or video at once. Then, the image proportional to the distance from the center of the screen is made into an equidistant projection method of the fisheye lens, and is placed in a single virtual image that is a fisheye image within a circular area in front of the field of view of the observer P, and this virtual image is called the spherical view.

[0027] Here, a general explanation of the equidistant projection method using mathematical formulas will be given below. The upper line segment L shown in Figure 2 is part of a straight line when a flat image seen by a person is viewed from directly above, and the lower circle C is what is seen when a sphere of radius r is viewed from directly above with the viewpoint of observer P at the center. The following relationship holds between the interior angle θ of circle C and the distance y of the arc. y = rθ

[0028] According to this relationship, y and θ are proportional to each other, and an area with length y is projected onto the flat image shown at the top of Figure 2 with a scale of cy using another constant c. As a result of these two relationships, the area on the sphere around the viewpoint of observer P is projected onto an area of ​​the flat image proportional to the size of the interior angle.

[0029] In the method for generating a celestial sphere image represented along with the concept of the field of view according to the present embodiment, as shown in FIG. 1 , arc portions at intervals of 45 degrees are shown to be projected onto portions of the celestial sphere image after projection, and according to the above explanation of the relational expressions, all of the lengths d shown in FIG. 1 are equidistant.

[0030] 3, image information or video information of the virtual space and display software are loaded in advance from a personal computer 14, which is an external computer, into a memory 18 in an HMD 16 worn on the head of an observer P. Along with this, the image information on the spherical surface W is also loaded into the memory 18. The personal computer 14 is then removed from the HMD 16.

[0031] 3 includes not only memory 18 but also a CPU 20 serving as control means, a controller 22 serving as an input device, and an image display device 24 such as a liquid crystal display, and the memory 18, controller 22, and image display device 24 are connected to the CPU 20. However, the CPU 20 also includes an image generation unit 20A that generates a normal view and a spherical view and an auxiliary line addition unit 20B that adds auxiliary lines, and the controller 22 can be held by hand away from the HMD 16 main body.

[0032] Furthermore, in the information presentation system 10 of this embodiment, the CPU 20 in the HMD 16 is configured to read out display software from the memory 18 and also read out image information or video information. This read-out image information and the like is combined into an image in front of the observer P and an image behind the observer P, and in this case, the image is expressed as a celestial sphere view in which a fisheye image is displayed within a circular region in front of the observer P's field of view using the above-mentioned equidistant projection method.

[0033] Next, the method of this embodiment will be described below with reference to an image of FIG. 4 showing a virtual space, and examples of views that the observer P can actually see through the HMD 16 will be shown below. Figure 4(A) shows an image of a town with roads and buildings in a virtual 3D city space as seen by a person using a normal view. In contrast, Figure 4(B) shows information about a spherical image taken at the same location and with the same body orientation of observer P, using a spherical view. Figures 4(A) and (B) were created using the free asset "Japanese Dosanko City" from the Unity game development environment.

[0034] Here, the buildings that are also seen in FIG. 4(A) are visible in a portion close to the center of FIG. 4(B) of the spherical view, and roads and buildings directly to the side and behind the observer P are visible in a distorted and elongated form in a position outside of that, far from the center. Note that in FIG. 4(B), the roads, buildings, and sky are displayed as further elongated between the rectangular frame on the outer periphery of the large circle R, but this is only displayed in the implementation by the software in this embodiment. This is redundant and unnecessary information that is a partial inversion of information inside the circle R, and this portion may be filled in with black or the like, and the spherical image according to this embodiment is entirely drawn inside the large circle R.

[0035] That is, the information presentation system 10 according to an embodiment of the present invention displays a 360-degree celestial sphere image within an image representation area, which is a circular area in front of the field of view of the observer P, and presents the image as an image that fits on the screen of a display such as the HMD 16, thereby providing a field of view expansion method that enables the observer P to quickly view and find objects that may exist in all directions.

[0036] Next, a description will be given of how to add auxiliary lines that are effective when determining the position in three-dimensional space of an object found in a virtual image of the information presentation system 10 according to the above embodiment. In the spherical view, the spherical image is presented in a position close to the central visual field of the observer P, and therefore has an advantage over the normal view in terms of quickly finding an object in space. However, because areas far from the center of the visual field appear distorted, it is not necessarily easy to instantly and accurately identify the position in three-dimensional space of an object in that area, although this differs from person to person.

[0037] For this reason, auxiliary lines that are effective in determining the position of objects in three-dimensional space are placed within the virtual image to support the actions that follow object discovery and enable actions and operations for each purpose to be performed quickly and smoothly. Specifically, the CPU 20 in the HMD 16 creates an auxiliary line with a diameter twice the height from the viewpoint of the observer P to the ground and with its lower end touching the ground, with the viewpoint of the observer P at the center, as shown in Fig. 5, and projects it so that one ring R1 and three rings R2, which are auxiliary lines, are drawn every 45 degrees around the observer P, who is considered to be in the correct position in the virtual image. The example shown in Fig. 5 is defined as the basic positions of the rings R1 and R2 that are arranged in all directions around the observer P.

[0038] In this embodiment, in order to make it easier to identify the four rings R1 and R2 directly to the side of the experiencer, i.e., the observer P (90 degrees to the left and right when viewed from the front), only the ring R1 located directly to the side is shown in red, and the other rings R2 are shown in yellow, which is a different color from red.

[0039] Figure 6 shows an example of an image in which four rings R1 and R2 are drawn as auxiliary lines on a spherical view. As shown in Figure 6, four rings R1 and R2 are drawn at equal intervals on a virtual sphere that surrounds the entire body of the user or part of the body including the face. This makes it easy to identify the position of objects in space relative to the user using information about their relative position with the rings.

[0040] As described above, the information presentation system 10 according to the present embodiment can present a spherical image in the form of a spherical view in front of the viewer P upon request, thereby enabling the viewer P to easily find an object. Accordingly, rather than simply forming auxiliary lines with a diameter twice the height from the viewpoint of the viewer P to the ground, which is large enough to enable the viewer P to find an object more quickly, four rings R1 and R2 are arranged at equal intervals of 45 degrees, which is the optimal number for the viewer P to recognize, as shown in FIG. 5. Accordingly, when the viewer views the outside world in three-dimensional space, the position of a target object or building can be easily identified based on the positional relationship between the rings R1 and R2 and the objects or buildings in the space.

[0041] These features not only enable faster object detection, but also enable faster and smoother actions and operations for each application, supporting actions after object detection. It is also possible to change the colors of the four rings R1 and R2 to different colors, and to set the size and position of the spherical view including the four rings R1 and R2 to different finite parameter values ​​or coordinates depending on the application. It is also possible to display these four rings in the normal view.

[0042] Next, a flow diagram of the image generation and display process for the virtual space is shown in FIG. 7, and the following explanation will be given based on this diagram. Image information of the virtual space and software for display are loaded in advance from an external personal computer 14 into the memory 18 of the HMD 16, and after the user, who is the observer P, wears the HMD 16, the CPU 20 of the HMD 16 reads the image information and software from the memory 18 and makes it possible to display the image.

[0043] Then, the operation starts, and by turning on the switch of the controller 22, first, from the start, "obtaining the viewpoint position in the user's space" is performed in step S11, and then the process moves to step S12, where it is determined "Should an auxiliary line be added?" If the determination is YES, the process moves to step S13, where "an auxiliary line addition process" is performed by the CPU 20, and then the process moves to step S14. However, even if the determination is NO in step S12, the process moves directly to step S14.

[0044] In step S14, it is determined "Which image should be displayed, the normal view or the spherical view?", and if the spherical view as shown in Fig. 4(B) is selected, the process proceeds to step S15, where "processing to generate a spherical view image" is performed by CPU 20, and then the process proceeds to step S17. However, if it is determined in step S14 that the normal view as shown in Fig. 4(A) has been selected, the process proceeds directly to step S16, where "processing to generate a normal view image" is performed by CPU 20, and then the process proceeds to step S17.

[0045] In step S17, the CPU 20 performs "image display processing" and the image is displayed on the image display device 24, and then the process proceeds to step S18, where it is determined whether "processing is to end?" If the determination is YES, the process ends. However, if the determination is NO in step S18, the process proceeds to step S19, where the user's position in the virtual space is changed, which is "user movement processing," and the process returns to step S11, where the process is executed again.

[0046] Next, a group of basic functions associated with the function, such as a function for displaying auxiliary lines, a function for resetting the display position of auxiliary lines, and a function for switching between a normal view and a spherical view, will be described below. Regarding how these auxiliary lines move when the experiencer (observer P) moves in three-dimensional space, the following two modes, Mode 1 and Mode 2, are provided, taking into consideration ease of use when using the four rings of auxiliary lines to identify the position of an object or when moving within the three-dimensional virtual space.

[0047] (1) Mode 1: This is a basic position mode in which the positions of the four rings are kept fixed in three-dimensional space. Accordingly, only when the user himself / herself uses the buttons on the controller 22 or the like to perform a reset operation to the basic positions of the rings corresponding to his / her own position and the direction of his / her face (or the direction of the HMD 16 if the HMD 16 is worn), the four rings R1 and R2 are redrawn in the basic positions as shown in Figure 5 for the user.

[0048] (2) Mode 2: This is a parallel movement mode in which the four rings R1 and R2 automatically move and follow the body so that the four rings are always displayed in their basic positions according to the position of the viewer in three-dimensional space and the direction of their face. In other words, as shown in Figure 5, the red ring R1 is always positioned directly to the side of the viewer P.

[0049] Mode 1 is used when, after an object is found in space, its location is identified based on its position relative to the fixed ring. Mode 2 is used when moving within space before finding an object. Mode 1 and Mode 2 can be switched instantly by operating the mode switch on the controller 22.

[0050] Next, a flow diagram when "addition of auxiliary lines" is selected in the flow diagram of the display image generation and display processing shown in FIG. 7 is shown in FIG. 8, and the following description will be given based on this diagram. First, from the start, in step S21, a determination is made as to whether "the display image of the immediately preceding frame includes auxiliary lines?" If the determination is YES, the process proceeds to step S22, where a determination is made as to whether "the direction of the auxiliary lines should follow the direction of the face?" If the determination is YES, the process proceeds to step S24, where the CPU 20 performs "processing to generate auxiliary lines for the basic position at the user's viewpoint position in the current frame" as in mode 2, and the auxiliary lines are displayed in the spherical view on the image display device 24 as shown in FIG. 6.

[0051] On the other hand, if the determination in step S21 is NO, the process proceeds to step S24, where the CPU 20 similarly performs "processing to generate an auxiliary line of the basic position at the user's viewpoint position in the current frame." Also, if the determination in step S22 is NO, the process proceeds to step S23, where the CPU 20 performs "processing to generate an auxiliary line by translating the auxiliary line in the immediately preceding frame to the user's viewpoint position in the current frame," as in mode 1. Then, when the processes in steps S23 and S24 are completed, the process ends.

[0052] In the above process, multiple auxiliary lines are generated around the user's viewpoint, and two types of display methods can be used. As an effect of this, for example, by making it possible to select one for each frame, the auxiliary lines can be always displayed in a basic position according to the direction of the user's face, or the auxiliary lines can be simply translated without changing their angle in space for a certain period of time regardless of the direction of the user's face, making it easier to identify the position of an object in space (including an object behind the user that is not normally visible in the forward field of view) based on the relative positional relationship between the object and the auxiliary lines.

[0053] Furthermore, to confirm the usability and operability of the auxiliary lines that can be added to the spherical view, we developed a VR application that allows users to discover an object that appears in a 3D space and then shoot the object with a beam in a virtual space. An example of how to use this VR application is explained below.

[0054] After observer P finds an object in the three-dimensional space of this VR application, he or she can use auxiliary lines to identify the object's location, aim at the object in the spherical view, or switch to the normal view and aim at the object, and shoot with the controller 22 held in his or her hand. This series of actions can be smoothly performed by actually wearing the HMD 16 and playing the game.

[0055] A specific example of a game played using the information presentation system 10 according to this embodiment will be described below. First, Figure 9(A) shows the state in which a sphere S has been discovered diagonally behind and to the left of the user in the spherical view. Figure 9(B) shows the state in which the user is facing directly to the side while keeping rings R1 and R2 fixed and irradiating the sphere S with a laser beam. In Figure 9(B), the red ring R1 (i.e., the auxiliary line directly to the user's side) appears to be directly in front of the user because the rings R1 and R2 are fixed in "Mode 1" above and the user's body and face wearing the HMD 16 are facing directly to the side. Figure 9(C) also shows the state in which the user has discovered sphere S in the spherical view, switched to normal view, and is irradiating the laser beam. In Figures 9(B) and 9(C), the laser beam directed at sphere S is depicted by arrow K.

[0056] 9(A), when a sphere S appears diagonally behind and to the left, Fig. 10(A) shows a state in which the user twists his / her body and irradiates a laser beam using the controller 22. On the other hand, after the user discovers in the spherical view that a sphere S has appeared directly behind him / her, he / she leans back directly upward without twisting his / her body left or right, points the controller 22 backward, and irradiates a laser beam, as shown in Fig. 10(B).

[0057] The linking of the visibility of objects such as the sphere S with the action and control of shooting cannot be done in the normal view, and the ability to see, move, and control such new objects is also an effect of the spherical view and auxiliary lines.

[0058] Next, a second embodiment of the information presentation system 10 according to the present invention will be described in detail with reference to the accompanying drawings. In the information presentation system 10 according to this embodiment, the HMD 16 shown in FIG. 11 is worn on the head of the observer P, and a front fisheye camera 32 is installed in front of the HMD 16, and a rear fisheye camera 34 is installed behind the HMD 16.

[0059] The front fisheye camera 32 captures an image in front of the observer P, and the rear fisheye camera 34 captures an image behind the observer P. The image in front of the observer P and the real image of the real space behind the observer P, which are actually captured and prepared by these fisheye cameras 32 and 34, are then integrated by the CPU 20 in the HMD 16. Furthermore, using an equidistant projection method of the fisheye lens proportional to the distance from the center of the screen, the image in front of the observer P and the image behind the observer P are combined into a single virtual image that is a fisheye image of a circular area in front of the observer P's field of view.

[0060] On the other hand, by projecting this virtual image from the CPU 20 of the HMD 16 via the image display device 24 so that it can be viewed by the observer P, real images or videos that are actually the scenery around the observer P can be viewed at once in one virtual image. Therefore, in this embodiment, without disposing a spherical surface W around the observer P, information on a 360-degree spherical image that is information on everything in front of and behind the observer P can be obtained by the front fisheye camera 32 and the rear fisheye camera 34.

[0061] Next, a flow diagram of the image generation and display process for the real space is shown in FIG. 12, and the following explanation will be given based on this diagram. Display software is installed in advance from an external personal computer 14 into the memory 18 of the HMD 16, and after the user, who is the observer P, wears the HMD 16, the CPU 20 of the HMD 16 reads the software from the memory 18.

[0062] Then, the operation starts. When the controller 22 is switched on, first, from the start, in step S31, "two fisheye images (original spherical images) acquired from two fisheye cameras or the like facing forward and backward from the user" are obtained, and the process proceeds to step S32, where it is determined "Should auxiliary lines be added?" If the determination is YES, the process proceeds to step S33, where "auxiliary line addition processing" is performed by the CPU 20, and the process proceeds to step S34. However, even if the determination is NO in step S32, the process proceeds directly to step S34.

[0063] In step S34, it is determined "Which image, a normal view or a spherical view, should be displayed?", and if the spherical view is selected, the process proceeds to step S35, where "processing for generating a spherical view image" is performed by CPU 20, and then the process proceeds to step S37. However, if it is determined in step S34 that the normal view has been selected, the process proceeds directly to step S36, where "processing for generating a normal view image" is performed by CPU 20, and then the process proceeds to step S37.

[0064] In step S37, the CPU 20 performs "image display processing" and the image is displayed on the image display device 24, and then the process proceeds to step S38, where it is determined whether "processing is to end?" If the determination is YES, the process ends. However, if the determination is NO in step S38, the process returns to step S31, and the process is executed again.

[0065] 7 , the present embodiment differs from the process of generating and displaying an image for a virtual space in that, in the virtual space, position information of the user in the virtual space held by personal computer 14 is acquired, whereas in the real space, such data is not stored in advance. For this reason, information on the entire celestial sphere is acquired in real time from two fisheye cameras 32, 34 attached to the top of the head or the like to capture images of the front and rear, respectively, and based on this, auxiliary lines are added and an image in a normal view or a celestial sphere view is generated and displayed.

[0066] Furthermore, in step S38, when the answer to the question "Do you want to end the process?" is NO, the user may move somewhere in the virtual space depending on the application, so the "user's viewpoint position" at the top is reacquired to reflect that result. In other words, in the case of real space, the image captured in each frame is from the updated camera position, and it is possible to add auxiliary lines centered on that point, so "user movement" processing is not performed.

[0067] In the above embodiment, the HMD 16 that can be viewed by the observer P is employed, but a virtual image may be projected onto a display of a desktop personal computer or the like, and viewed by the observer P. Also, instead of directly capturing real images with the front fisheye camera 32 and the rear fisheye camera 34 as in the second embodiment, a virtual image that is to be considered a fisheye image may be created using real images that have been captured in advance.

[0068] On the other hand, the auxiliary lines are four rings centered on the observer's viewpoint, each having a diameter twice the height from the observer's viewpoint to the ground and whose bottom ends touch the ground, but the number and diameter of the rings may be changed depending on the application. For example, instead of four rings arranged every 45 degrees, it is also possible to use n rings arranged every (180 / n) degrees, where n is an integer greater than or equal to 1. Furthermore, it is also possible to use rings centered on the observer's viewpoint and with a diameter of any finite value as auxiliary lines. Furthermore, auxiliary lines may be lines or shapes other than rings. On the other hand, although the center of the ring is set to be the observer's viewpoint, it is also possible to create a 360-degree spherical image seen from each of the observer's two eyes (both eyes). In this case, a ring with a diameter twice the height from the observer's viewpoint to the ground and centered on each eye is included in each spherical image as an auxiliary line.

[0069] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications are possible based on the spirit of the present invention, and these modifications are not excluded from the scope of the present invention. [Industrial Applicability]

[0070] The information presentation system according to the present invention can also be applied to games, amusement, e-sports, crime prevention and security, automobiles, space, aviation, and marine fields. [Explanation of symbols]

[0071] 10 Information presentation system 14. Computer 16 HMD 18 Memory 20 CPU (control means) 22 Controller 24 Image display device R1, R2 rings (auxiliary lines) P Observer

Claims

1. Prepare images in front of and behind the observer in virtual or real space, Using the equidistant projection method of a fisheye lens, which is proportional to the distance from the center of the screen, The control means combines the image in front of the observer and the image behind the observer into a single virtual image that is a fisheye image of a circular area in front of the observer's field of view, An information presentation system that projects this virtual image onto an image display device that is visible to the observer.

2. 2. The information presentation system according to claim 1, wherein the image display device is installed in a head-mounted display that can be worn by the observer.

3. 2. The information presentation system according to claim 1, wherein auxiliary lines effective for determining the position of an object in a three-dimensional space in the virtual image are arranged in the virtual image.

4. 4. The information presentation system according to claim 3, wherein the auxiliary line is a ring having a diameter twice the height from the observer's viewpoint to the ground and a lower end thereof in contact with the ground, the diameter being centered on the observer's viewpoint.

5. 5. The information presentation system according to claim 4, wherein the auxiliary lines are four rings arranged at 45-degree intervals with the observer at the center.

6. 6. The information presentation system according to claim 3, further comprising a function of displaying auxiliary lines, a function of resetting the display position of the auxiliary lines, and a function of switching to a normal view.

Images