A program for a floating image display device that displays 360-degree images

Abstract

We provide a program for an aerial image display device that can be viewed over a wide area of ​​360 degrees space by using a spherical surface as a screen. [Solution] A program executed on an aerial image display device (1) equipped with a computer and an imaging system controlled by the computer that projects an image onto an imaging surface in the air includes the steps of inputting 360-degree image data, defining a sphere (D) in an area in the air that includes the imaging surface, pasting the 360-degree image onto the surface of the sphere, and projecting and displaying the surface of the sphere with the pasted 360-degree image onto the imaging surface. The sphere can be freely rotated using hand gestures.

Description

[Technical Field]

[0001] The present invention relates to a program for an aerial image display device that displays 360-degree images. [Background technology]

[0002] In recent years, 360-degree video has rapidly become popular and is attracting attention in various fields. This technology allows users to freely move their line of sight and view an object from any angle they like.

[0003] 360-degree video is impactful, contains a lot of information, and is immersive, so it is widely used in corporate activities, such as room viewings in the real estate industry, providing simulated travel experiences in the tourism industry, and facility tours.

[0004] Furthermore, as 360-degree cameras have become readily available, it has become common for individuals to easily create and publish 360-degree videos. In fact, many 360-degree videos can be viewed in various fields on Internet video sites. Examples of such 360-degree cameras include those described in Patent Document 1. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-012099 Summary of the Invention [Problem to be solved by the invention]

[0006] 360-degree video is typically captured using a spherical 360-degree camera, capturing 360 degrees of vertical and horizontal space, and then recorded as a single flat image in a format such as panoramic, dome master, or cube map. When a user (viewer) views the recorded 360-degree video, the data is rendered into an area seen from a specific viewpoint (viewport), and then played back as a flat image. Therefore, the user is viewing a portion of the 360-degree space.

[0007] Therefore, an object of the present invention is to provide a program for an aerial image display device that can be viewed over a wide area of ​​360 degrees in space by using a spherical surface as a screen. [Means for solving the problem]

[0008] In order to solve the above problems, a program for an aerial image display device according to one aspect of the present invention includes a computer and an image system controlled by the computer that projects an image onto an imaging surface in the air. a motion sensor that detects the coordinates of a user's hand in the vicinity of the aerial image projected by the video system and transmits the detection results to the computer; a program executed by an aerial image display device comprising: a step of inputting data of a 360-degree image into the computer; a step of defining a sphere in an area in the air including the imaging surface; a step of pasting the 360-degree image onto the surface of the sphere; and a step of projecting and displaying the surface of the sphere with the pasted 360-degree image onto the imaging surface. The motion sensor detects a hand gesture made by a user and rotates the sphere according to the detection result. Execute In the step of rotating the sphere, the direction of the rotation vector changes according to the hand gesture made by the user. It is characterized by:

[0009] In one embodiment, the 360-degree video is a 360-degree video.

[0010] Furthermore, in one embodiment, The 360-degree video is a spherical video taken with a 360-degree camera. It is characterized by: [Effects of the Invention]

[0011] According to the program for the aerial image display device of the present invention, by using a spherical surface as a screen, 360-degree images can be viewed over a wide area. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a perspective view showing an aerial image display device 1 that executes an embodiment of a program according to the present invention. [Figure 2] FIG. 2 is a cross-sectional view showing the aerial image display device 1 taken along line AA in FIG. [Figure 3] FIG. 3 is a diagram illustrating the process of applying a 360-degree video image based on equirectangular projection onto a spherical surface in the program according to the present invention. [Figure 4] FIG. 4 is a diagram showing an example in which a 360-degree video is pasted only on the front side of a sphere in the program according to the present invention. [Figure 5] FIG. 5 is a diagram showing an example in which a 360-degree video is pasted only on the far side of a sphere in the program according to the present invention. [Figure 6] FIG. 6 is a diagram for explaining the program according to the present invention, showing how a sphere serving as a 360-degree video screen is rotated by touching it with one's hand. [Figure 7] FIG. 7 is a diagram for explaining the operation of an embodiment of the program according to the present invention, and is a flowchart showing the process of rotating a sphere serving as a 360-degree video screen by touching it with one's hand. [Figure 8] FIG. 8 is a diagram showing a state in which a light-shielding plate 50 is placed over liquid crystal display 10 in aerial image display device 1 of FIG. [Figure 9] FIG. 9 is a plan view showing the light-shielding plate 50 of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0013] An embodiment of a program for an aerial image display device according to the present invention will now be described with reference to the accompanying drawings. Fig. 1 is a perspective view showing an aerial image display device 1 for carrying out an embodiment of the present invention. Fig. 2 is a cross-sectional view taken along line A-A in Fig. 1, showing the aerial image display device 1 in use.

[0014] [Structure of the aerial image display device] 1 and 2, the aerial image display device 1 includes, as essential components, a liquid crystal display 10 and an optical plate 20. The aerial image display device 1 also includes an information processing device 30 that controls the liquid crystal display 10, a speaker 40, and the like. The liquid crystal display 10, the speaker 40, and the information processing device 30 are housed inside the lower housing 12, and are each connected by signal lines (not shown).

[0015] The information processing device 30 is essentially a small computer and is composed of a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), a storage device for storing various programs and data, an input / output interface, etc. Examples of input / output interfaces include a USB port and a wireless LAN such as Wi-Fi. The information processing device 30 outputs a video signal to the liquid crystal display 10, which displays the aerial image. The information processing device 30 also outputs an audio signal to the speaker 40, which generates voice guidance and sound effects. This information processing device 30 can also be a commercially available general-purpose small personal computer or a general-purpose tablet.

[0016] The liquid crystal display 10 is housed in a lower housing 12 that is rectangular in plan view and has an open top, and is supported almost horizontally with the display screen facing upward. The optical plate 20 is fitted into the upper housing 14 so that its incident surface 21 faces downward and faces diagonally to the display screen of the liquid crystal display 10. Here, the optical plate 20 and the liquid crystal display 10 are fixed at an angle of approximately 45 degrees.

[0017] An example of such an optical plate 20 is a retro-transmission optical imaging element (two-sided orthogonal reflector) described in Japanese Patent Laid-Open Publication No. 2011-175297. This optical imaging element is realized by arranging a large number of mutually orthogonal planar light reflecting portions at a fixed pitch. Alternatively, a two-sided corner reflector, in which reflective surfaces are formed on the side surfaces of a square hole, as described in Japanese Patent No. 4900618, may be used.

[0018] Furthermore, upper housing 14, which constitutes aerial image display device 1, can be easily separated from lower housing 12. Separating upper housing 14 makes it easier to perform maintenance and adjustments inside lower housing 12, and also reduces the height during transportation.

[0019] As shown in Figure 2, light from the display screen of the liquid crystal display 10 enters the optical plate 20, reflects twice inside the optical plate 20, and exits to the opposite side. As a result, a real aerial image is formed in space on the opposite side of the optical plate 20, with the optical plate 20 as the plane of symmetry. In this case, to achieve a clearer aerial image, it is desirable that external light not be added to the light from the liquid crystal display 10. In Figure 2, reference character G indicates the imaging area corresponding to the display surface of the liquid crystal display 10. This imaging area is the imaging plane for the display image on the liquid crystal display 10.

[0020] Furthermore, a three-dimensional motion sensor 7 consisting of an infrared LED 72 and a pair of infrared cameras 74 is provided on the front side of the lower housing 12. This motion sensor 7 accurately detects the movement of the user's hand in three dimensions. The detection range of the motion sensor 7 is determined by the emission angle of the infrared LED 72 and the angle of view of the infrared camera 74. Such a motion sensor 7 can be a commercially available product such as Ultraleap's Leap Motion Controller 2.

[0021] In particular, the three-dimensional motion sensor 7 is attached so that its tilt angle can be adjusted. That is, the motion sensor 7 is stored in a storage member that allows it to rotate within a certain angular range (here, ±20 degrees) around the rotation axis perpendicular to the paper surface of the cross-sectional view of FIG. 2. Here, the center of the detection range of the motion sensor 7 is adjusted so that it is shifted toward the user from the center of the imaging area G. By adjusting it in this way, the effective operation detection range can be maximized.

[0022] Although not shown here, it is recommended to place the aerial image display device 1 on a table or the like at an appropriate height so that the aerial image can be viewed at eye level. In particular, a device that comes with an elevating device and allows the height to be freely adjusted is suitable. An example of such an elevating device is the Mario N electric elevating device sold by Yamato Metal Works.

[0023] Typically, such aerial image displays are used as administrative input devices for reception systems and the like. That is, by displaying a non-contact interface screen as aerial images, users can operate the non-contact interface screen by touching it with their fingers or using gestures. The operation is detected by a motion sensor 7 consisting of an infrared LED 72 and an infrared camera 74, and a corresponding operation signal is sent to the information processing device 30, where the specified processing is performed. The operation interface includes controls such as buttons, check boxes, and drop-down menus. Therefore, a completely non-contact interface can be realized with the same ease of use as a conventional touch panel.

[0024] [360-degree spherical display] As described above, a floating image is displayed in imaging area G corresponding to the display surface of liquid crystal display 10. Although the floating image is two-dimensional, a three-dimensional object is virtually generated in three-dimensional space near imaging area G, and this three-dimensional object is perspectively projected onto the screen of liquid crystal display 10, resulting in an image with a sense of depth (stereoscopic effect). This allows the three-dimensional object to be visually perceived as floating in the air.

[0025] In this embodiment, a sphere D is displayed as this three-dimensional object. The center of the sphere D is aligned with the center of the imaging area G, and the diameter is approximately half the width of the imaging area G. The surface of the sphere D is used as a screen for displaying a spherical image (360-degree image). That is, the spherical image is displayed by being attached to the surface (spherical surface) of the sphere D.

[0026] Spherical video uses data captured by a 360-degree camera. Here, we will explain using 360-degree data showing the movement of stars in the night sky as an example. Therefore, the 360-degree video image will be rendered on the surface of a sphere D.

[0027] 360-degree videos are generated as MP4 format video files based on equirectangular projection, for example. In equirectangular projection, spherical images are expressed using rectangular image data whose vertical and horizontal axes represent latitude and longitude, respectively (see Figure 3(a)). Therefore, when pasting onto a spherical surface, the radius of sphere D is taken as r and the center of sphere D as the coordinate origin, and each point on the vertical and horizontal axes of the image data (latitude φ and longitude λ) can be converted to a point (x, y, z) on the spherical surface using the following equation (see Figure 3(b)). x = r cosφ cosλ y = r cosφ sinλ z = r sinφ

[0028] In this way, the spherical image with the 360-degree video pasted on it is projected onto the imaging surface using perspective projection transformation and displayed. This pasting can be easily implemented using 3D development software such as Unity.

[0029] When a 360-degree video is pasted onto a spherical surface in this way, the image of the front side as seen from viewpoint E outside sphere D is mirrored (see Figure 4). For example, when expressing a starry sky as in this example, all constellations and the like are mirror images. As will be explained later, in this embodiment, sphere D onto which the video is pasted can be freely rotated on any axis, allowing the viewer to observe the entire starry sky from a bird's-eye view, just as if using a celestial globe.

[0030] In this example, the starry sky background is black, so the back side is visible. Therefore, if the opposite side (the back side of sphere D) is displayed, the image on the front side will be difficult to see. For this reason, the opposite side of sphere D will be erased using culling processing.

[0031] In addition, since this example uses an actual photograph of the starry sky, the starry sky cannot be seen in the hemisphere facing the ground, but it is possible to display a corresponding starry sky (mainly the starry sky of the southern hemisphere) in the hemisphere facing the ground, creating a sphere equivalent to a celestial globe. In this case, multiple images, including a 360-degree image of the starry sky in the southern hemisphere, can be connected to create a spherical starry sky, or the arrangement of stars can be reconstructed on a computer like in a planetarium.

[0032] On the other hand, mirror images often create an unnatural feeling with ordinary video. In such cases, the image is not pasted onto the front spherical surface of sphere D as seen from viewpoint E, but is removed and pasted only onto the spherical surface at the back of sphere D. In this way, the user will be able to see the inner surface at the back from inside sphere D (see Figure 5). By doing this, the user can enjoy a 360-degree video on the spherical surface as a normal image. Of course, in the case of a starry sky, if the image is pasted onto the back side of the sphere and displayed as a normal image, the constellations can be seen in their actual arrangement.

[0033] Furthermore, in this embodiment, the sphere (image) can be enlarged or reduced by a predetermined gesture. For example, if you bring your index finger and thumb together (as if pinching) and move your hand forward, the display of the three-dimensional object will enlarge. Conversely, if you bring your index finger and thumb together and move your hand backward, the display of the three-dimensional object will shrink. These gestures correspond to pinching out and pinching in to enlarge and reduce the size of a touch panel. Of course, instead of bringing your index finger and thumb together, you can also use a gesture of moving your hand back and forth with your thumbs up.

[0034] Therefore, the user can rotate the sphere so that the part of the image they are interested in is directly in front of them, and then perform the gesture described above to enlarge and display the part in front of them. For example, when enlarged, it can be expected that the user feels as if they are inside the sphere and observing it.

[0035] [Rotating sphere] Sphere D with this 360-degree video pasted on it can be rotated using hand gestures. The rotation of sphere D is controlled by gestures, just like stroking the surface of a sphere floating in the air (see Figure 6). If the coordinates of the user's hand detected by motion sensor 7 are inside the coordinates of the surface of sphere D, it is determined that there is interference between the user's hand and sphere D, and sphere D is rotated in accordance with the movement of the user's hand.

[0036] When the system starts up, the 360-degree video is pasted onto the surface of the sphere D, and the coordinates of the user's hand detected by the motion sensor 7 are repeatedly acquired. Then, each time a coordinate is acquired, it is determined whether or not the coordinate is located inside the sphere D. If the hand coordinate is inside the sphere, the movement of the user's hand is calculated. Here, the movement of the user's hand is calculated as the difference between the coordinates acquired last time and the coordinates acquired this time. The calculated movement of the user's hand is converted into a rotation vector with the center (center of rotation) of the sphere D as the origin, and the sphere D is rotated to redraw the 360-degree video. In this way, the movement of the user's hand can be reflected in the rotation of the sphere D.

[0037] If the coordinates of the user's hand acquired by the motion sensor 7 are outside the sphere, it is determined whether the previously acquired coordinates were located inside the sphere. If the previously acquired coordinates were not located inside the sphere D, no special processing is performed here, and acquisition of the user's hand coordinates is repeated. If the previously acquired coordinates were located inside the sphere D, it is determined that the user touched the sphere D with their hand, rotated it, and then released their hand from the sphere D, maintaining the rotation.

[0038] That is, the rotation speed of sphere D is calculated from the difference between the coordinates acquired this time and the coordinates acquired previously. After the user's hand leaves sphere D, sphere D continues to rotate according to this rotation speed regardless of the movement of the user's hand. However, this rotation speed decays according to a certain decay coefficient.

[0039] If the user wants to stop sphere D in a specific position, the user pulls their hand straight out from sphere D in a radial direction from the center so as not to rotate sphere D. To detect this intention, the radial component of the difference between the coordinates acquired this time and the coordinates acquired last time is compared with the other component (circumferential component). If the radial component is greater than the circumferential component, the circumferential component is ignored (replaced with 0), and sphere D is fixed in its current position. Conversely, if the radial component is not greater than the circumferential component, sphere D is rotated according to the circumferential component.

[0040] The overall flow of the above process will be explained in more detail with reference to the flowchart in Figure 7. First, in step S1, the motion sensor 7 acquires the coordinates of the user's hand. Here, the coordinates of the fingertips are acquired. Therefore, if both hands are used, up to 10 coordinates are acquired. The center of the sphere D displayed here is the center of rotation, the center of the imaging area, and the origin of the coordinate system. The vectors from the origin of this coordinate system to each coordinate will be called coordinate vectors.

[0041] Next, in step S2, it is determined whether the acquired coordinates are located inside the sphere D. If it is determined that at least one coordinate is located inside the sphere D (YES in step S2), the movement of the user's hand is calculated as the difference between the previously acquired value and the currently acquired value for each coordinate located inside the sphere D, and a rotation vector v linked to the hand movement is obtained (step S3).

[0042] Specifically, the coordinate vector obtained this time is r i , the previously obtained coordinate vector is r i', the individual rotation vectors v i (Subscript i is 1 to n: maximum 10) is calculated. Here, X is the cross product, and simply v i =(r i Xr i ') / |r i | Calculate the rotation vector v i is very small, the rotation vector v we want is the rotation vector v i The average (Σv i / n).

[0043] Next, the sphere D is rotated using the rotation vector v thus obtained (step S4). That is, if the coordinates of each point on the sphere D are p, the point p' after rotation can be obtained using the following equation: Here, X means the cross product, and · means the dot product.

[0044] p'=pcos|v|+|v| -2 v(v·p)(1-cos|v|)+|v| -1 (vXp)sin|v|

[0045] If it is determined in step S2 that all of the acquired coordinates are located outside sphere D (NO in step S2), it is determined whether any of the previously acquired coordinates are located inside sphere D (step S5). If any of the previously acquired coordinates are located inside sphere D (YES in step S5), it is determined that the user touched sphere D with their hand, rotated it, and then released their hand from sphere D.

[0046] If the result of step S5 is YES, the difference between the coordinates acquired this time and the coordinates acquired last time (r i -r i The radial and circumferential components of the coordinate vector (′) are compared (step S6). To do this, the cross product and inner product of the currently acquired coordinate vector and the difference are calculated, and it is determined which is larger.

[0047] That is, (r i -r i ')·r i >|(r i -r i ')Xri If |, it is determined that the radial component is large (YES in step S6), and the rotation of sphere D is stopped and fixed at the current position (step S7). This allows the user to intentionally stop sphere D in a specific position by pulling their hand out from sphere D.

[0048] (r i -r i ')·r i ≦|(r i -r i ')Xr i If |, the circumferential component is determined to be large (NO in step S6), and it is determined that the user intends to freely rotate the sphere D even after releasing the hand. Therefore, the coordinate vector r i , r i ', the rotation vector v is calculated as above. Even after the user moves their hand out of the sphere D, the sphere D continues to rotate according to the rotation vector v.

[0049] The rotation vector v corresponds to an angular velocity ω (=|v| / d), where d is the coordinate acquisition interval. It is possible to continue rotating the sphere D at this angular velocity ω, but in the real world, even when floating in the air, the rotation gradually decays due to air resistance. Therefore, a decay simulation is also performed here (Step S8).

[0050] Since air resistance is proportional to angular velocity ω, the equation of motion is Cω=-Mdω / dt, where C is the coefficient and M is the moment of inertia. To implement this, we can update the display of sphere D at coordinate acquisition intervals of d, and multiply the angular velocity ω by a constant damping coefficient at each interval d. For example, if the damping coefficient is 0.9, the angular velocity will be roughly halved every 7d. Increasing this damping coefficient will result in the sphere stopping more slowly.

[0051] In the above embodiment, operation using both hands is also possible. For example, if the left hand is stopped inside the sphere D and the right hand is used to rotate the sphere, the movements of the left and right hands are averaged, making it possible to apply the brake with the left hand while controlling finer movements with the right hand.

[0052] In the above embodiment, the sphere (image) is also programmed to be enlarged or reduced by a predetermined gesture. If this enlargement or reduction function is not implemented, the size of the sphere will be fixed, and only the circular area in the center of the screen of the liquid crystal display 10 (the display area of ​​the sphere) will always be used for display.

[0053] Furthermore, due to its structure, LCD displays cannot achieve complete black, and a small amount of light inevitably leaks from the backlight in areas other than the circular area that are supposed to emit no light (set to black). This is reflected in the imaging area G, and there is a possibility that it may slightly reduce the floating and three-dimensional effect of the sphere.

[0054] Therefore, when the sphere size is fixed, it is effective to cover the display screen of the liquid crystal display 10 except for the circular area with a light-shielding plate 50, as shown in Fig. 8. That is, the light-shielding plate 50 is large enough to cover the entire display screen of the liquid crystal display 10, and has an opening 52 in the center that is the same size as the circular area (see Fig. 9). When the light-shielding plate 50 is installed on the display screen of the liquid crystal display 10, the circular area displayed on the display screen (the display area of ​​the sphere) and the opening 52 in the light-shielding plate 50 exactly match. As a result, the area outside the circular area corresponding to the sphere is completely shielded from light.

[0055] Therefore, only the display of the sphere on the liquid crystal display 10 is imaged, and no image is formed in the other areas. In other words, the image formation area is limited to the narrower area G'. This gives the sphere a distinct floating feeling, and when a gesture of rotating it is made, it feels more realistic that you are touching the sphere. [Industrial Applicability]

[0056] According to the floating image display device for displaying 360-degree images of the present invention, by using a spherical surface as a screen, it is possible to provide a floating image display device that allows viewing over a wide area of ​​360-degree space. Such a floating image display device can provide users with a video experience never seen before.

[0057] The above describes the aerial image display device according to the present invention based on an embodiment, but the present invention is not limited to this, and modifications may be made within the scope of the spirit of the present invention, and if possible, the techniques described in each embodiment may be combined, or publicly known techniques may be combined, etc.

[0058] For example, in the above embodiment, the video to be pasted onto the spherical surface is input as a two-dimensional image expressed by equirectangular projection, but this is just one example, and other formats may be used. For example, any format that expresses an image captured from a single viewpoint over the entire circumference, such as a combination of two images in the Dome Master format, may be used in the same way.

[0059] The 360-degree images used here include not only 360-degree videos but also 360-degree still images. By displaying 360-degree still images on a sphere, you can slowly observe the entire view from a certain position.

[0060] Furthermore, although the above embodiment employs an aerial image display device using a retro-transmissive optical imaging element, the present invention is not limited to this, and other types of aerial image display devices, such as an aerial image display device using a retro-reflective optical imaging element, may also be employed. [Explanation of symbols]

[0061] 1. Aerial image display device 7. Three-dimensional motion sensor 10 LCD display 12 Lower housing 14 Upper housing 20 Optical Plate 21 Entrance plane 30 Information processing equipment 40 speakers 72 infrared LEDs 74 Infrared Camera D sphere E viewpoint G Imaging area

Claims

1. A program executed on an aerial image display device including a computer, an image system controlled by the computer and projecting an image onto an imaging plane in the air, and a motion sensor that detects the coordinates of a user's hand moving near the aerial image projected by the image system and transmits the detection results to the computer, the program including: inputting data of the 360-degree image; defining a sphere in an area in the air that includes the imaging plane; pasting the 360-degree image onto a surface of the sphere; a step of projecting and displaying the surface of the sphere onto which the 360-degree image is pasted onto the imaging surface; the motion sensor detects a hand gesture made by a user and rotates the sphere according to the detection result; In the step of rotating the sphere, the direction of the rotation vector changes according to a hand gesture made by the user.

2. 2. The program according to claim 1, wherein the 360-degree image is a 360-degree video.

3. The program described in Claim 1, characterized in that the 360-degree image is a spherical image captured by a 360-degree camera.

Images