Program for aerial image display device

Abstract

This invention provides an aerial image display device that allows users to freely move three-dimensional objects floating in the air using their hands. [Solution] A spherical determination area encompassing a three-dimensional object projected into the air by an aerial image display device is defined. A motion sensor installed in this aerial image display device detects the user's hand movements, and if the user's hand movements occur within this determination area, the three-dimensional object is rotated in conjunction with the hand movements. If the user's hand movements occur outside the determination area, those hand movements are not reflected in the movement of the three-dimensional object.

Description

Technical Field

[0001] The present invention relates to a program for an aerial image display device capable of moving a three-dimensional object floating in air without discomfort by direct physical touch. Background Art

[0002] An aerial image display device is a device that displays objects in empty air. For example, Patent Document 1 discloses an aerial image display device that displays an image in air by utilizing light reflection. When displayed as an aerial image by an aerial image display device, a three-dimensional effect can be perceived because the image appears floating from the surroundings. In particular, when displaying an object in a 3D space by perspective projection, it is possible to create the illusion that a three-dimensional object actually exists there.

[0003] Many currently practical aerial image display devices are provided with a three-dimensional motion sensor composed of an infrared LED and an infrared camera. By detecting the movement of a user's hand in the vicinity of the aerial image with this three-dimensional motion sensor, an input interface via the aerial image is implemented. This allows the aerial image of the aerial image display device to be operated by directly touching the aerial image. For example, it is possible to manually move and rotate a three-dimensional object floating in air. Prior Art Documents Patent Documents

[0004] Patent Document 1 Re-published Publication No. 2018 / 139141 Summary of the Invention Problems to be Solved by the Invention

[0005] However, when attempting to implement a system that allows users to directly manipulate floating 3D objects as if touching them, it often proves that the objects don't move as intended. While 3D motion sensors can detect hand movements with sufficient accuracy, the user's perceived stereoscopic vision is not always accurate. This is likely because the aerial images projected onto a flat surface do not adequately simulate depth perception. Furthermore, even when the hand is positioned to actually touch the 3D object, tactile sensation may not be perceived.

[0006] Therefore, you might think you're touching a part of a 3D object when you're not, or conversely, you might accidentally touch a protruding part of a 3D object and move it significantly when you didn't intend to.

[0007] Therefore, the objective of the present invention is to create an aerial image display device that minimizes the feeling of discomfort when manipulating a three-dimensional object floating in the air by directly touching it with one's hands. Program for The objective is to provide. [Means for solving the problem]

[0008] To solve the above problems, an aerial image display device according to one aspect of the present invention A program for an aerial image display device comprising a computer, an image system controlled by the computer that projects an image into the air, and a motion sensor that detects the coordinates of a user's hand in the vicinity of the aerial image projected by the image system and transmits the detection result to the computer, wherein the program is executed on an aerial image display device comprising the steps of controlling the image system to project a three-dimensional object having a convex portion as an aerial image, The steps include: defining a three-dimensional determination area that is larger than the three-dimensional object and encompasses the entire three-dimensional object; projecting the movement of the three-dimensional object linked to the hand movement as an aerial image if the coordinates of the user's hand acquired by the motion sensor represent hand movement within the determination area but outside the three-dimensional object; and projecting the movement of the three-dimensional object unaffected by the hand movement as an aerial image if the coordinates of the user's hand acquired by the motion sensor represent hand movement outside the determination area. It is characterized by causing this.

[0009] In one embodiment, The determination region is a spherical region or a substantially ellipsoidal region that includes the three-dimensional object. It is characterized by the following:

[0010] Furthermore, in one embodiment, In the step of projecting the movement of the three-dimensional object, which is linked to the hand movement, as an aerial image, the three-dimensional object is rotated around its center point in accordance with the user's hand movement. It is characterized by the following:

[0011] Furthermore, in one embodiment, the user's hand movements are detected as movements of multiple fingertips on both hands, and the average of the rotations corresponding to these fingertip movements is reflected in the rotation of the three-dimensional object. [Effects of the Invention]

[0012] According to the aerial image display device of the present invention, three-dimensional objects floating in the air can be freely manipulated without any sense of incongruity. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a perspective view showing an embodiment of an aerial image display device that executes a program according to the present invention. [Figure 2] Figure 2 is a cross-sectional view along line A-A in Figure 1, showing the aerial image display device 1 in use. [Figure 3] Figure 3 illustrates an embodiment of an aerial image display device that executes a program according to the present invention, showing how a three-dimensional object displayed in the air is rotated by touching it with a hand. [Figure 4] Figure 4 is a diagram illustrating the operation of an embodiment of the program according to the present invention, and is a flowchart showing the process of rotating a three-dimensional object displayed in the air as the object to be manipulated by touching it with a hand. [Modes for carrying out the invention]

[0014] Hereinafter, an embodiment of the program for the aerial image display device according to the present invention will be described with reference to the attached drawings. Figure 1 is a perspective view showing an aerial image display device 1 for executing an embodiment of the present invention. Figure 2 is a cross-sectional view along line A-A in Figure 1 showing the aerial image display device 1 in use.

[0015] As shown in Figures 1 and 2, the aerial image display device 1 includes a liquid crystal display 10 and an optical plate 20 as essential components. The aerial image display device 1 also includes an information processing device 30 that controls the liquid crystal display 10, and a speaker 40. Here, the liquid crystal display 10, speaker 40, and information processing device 30 are housed inside the lower housing 12 and are connected to each other by signal lines (not shown in the figures).

[0016] The information processing device 30 is substantially a small computer, and is constituted by a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a storage device that stores various programs, data and the like, an input / output interface, and the like. As the input / output interface, for example, a USB port, a wireless LAN such as WiFi, or the like is mounted. The information processing device 30 outputs a video signal to the liquid crystal display 10, and performs display that serves as a source of an aerial image. Further, the information processing device 30 outputs an audio signal to the speaker 40, and generates guidance voice, sound effects, and the like. For the information processing device 30, a commercially available general-purpose small personal computer, a general-purpose tablet, or the like can also be used as-is.

[0017] The liquid crystal display 10 is housed in a lower housing 12 whose upper rectangular portion in plan view is open, and is supported substantially horizontally with its display screen facing upward. Further, the optical plate 20 is fitted into the upper housing 14 such that its incident surface 21 faces downward and obliquely faces the display screen of the liquid crystal display 10. Here, the optical plate 20 and the liquid crystal display 10 are fixed at an angle of approximately 45 degrees.

[0018] As such an optical plate 20, for example, a retrotransmissive optical imaging element (two-surface orthogonal reflector) described in Japanese Patent Application Laid-Open No. 2011-175297 can be used. This optical imaging element is realized by arranging a large number of planar light reflecting portions orthogonal to each other at a constant pitch. In addition, a structure such as a two-surface corner reflector in which a reflecting surface is formed on a side surface of a square hole as described in Japanese Patent No. 4900618 may also be used.

[0019] It should be noted that the upper housing 14 constituting the aerial image display device 1 can be easily separated from the lower housing 12. If the upper housing 14 is separated, maintenance and adjustment of the interior of the lower housing 12 can be easily performed, and the height during transportation can be reduced.

[0020] As shown in FIG. 2, light from the display screen of a liquid crystal display 10 enters an optical plate 20, is reflected twice inside the optical plate 20, and exits to the opposite side. As a result, with the optical plate 20 serving as a plane of symmetry, an aerial image as a real image is formed in the space on the opposite side. In this case, to achieve a clearer aerial image, it is preferable that external light is not added to the light from the liquid crystal display 10. In FIG. 2, reference sign G indicates an image forming region corresponding to the display surface of the liquid crystal display 10.

[0021] Furthermore, a three-dimensional motion sensor 7 consisting of an infrared LED 72 and a pair of infrared cameras 74 is provided on the front side of the lower housing 12. By means of this motion sensor 7, the movement of a user's hand is precisely detected three-dimensionally. The detection range of the motion sensor 7 is defined by the emission angle of the infrared LED 72 and the angle of view of the infrared cameras 74. As such a motion sensor 7, a commercially available product, for example, the Leap Motion Controller 2 from Ultraleap, can be diverted for use.

[0022] In particular, this three-dimensional motion sensor 7 is attached such that its inclination angle can be adjusted. That is, the motion sensor 7 is housed in a housing member that can rotate within a certain angular range (±20 degrees in this example) about a rotation axis extending in a direction perpendicular to the plane of the cross-sectional view of FIG. 2. Here, adjustment is performed such that the center of the detection range of the motion sensor 7 is shifted closer to the user than the center of the image forming region G. By performing adjustment in this manner, the effective operation detection range can be maximized.

[0023] Although not shown here, it is preferable to place the aerial image display device 1 on a table or the like of an appropriate height so that the aerial image can be viewed at eye level. In particular, a device equipped with a lifting device that allows the height to be freely adjusted is suitable. As such a lifting device, for example, the electric lifting device Mario N sold by Yamato Metal Works is available.

[0024] Typically, such aerial image display devices are used as administrative input devices, such as in reception systems. That is, by displaying a contactless interface screen as an aerial image, users can operate the contactless interface screen using gestures, such as touching it with their fingers. This operation is detected by a motion sensor 7, consisting of an infrared LED 72 and an infrared camera 74, and a corresponding operation signal is sent to an information processing device 30 for predetermined processing. The operation interface can include controls such as buttons, checkboxes, and drop-down menus. Therefore, a completely contactless interface can be achieved with the same ease of use as a conventional touch panel.

[0025] In this invention, aerial image display devices are used to display 3DCG-based three-dimensional objects for entertainment or educational purposes. The three-dimensional objects are created by projecting a computer-defined three-dimensional object onto the screen of the liquid crystal display 10 using perspective projection, resulting in an image with a sense of depth (three-dimensionality). Furthermore, because the screen of the liquid crystal display 10 is projected in front of the user, this computer-defined three-dimensional space corresponds to the three-dimensional space in front of the user.

[0026] Here, a tetrapod is used as an example of a three-dimensional object. However, although Figure 2 depicts a tetrapod viewed from the side, in reality, the aerial image display device 1 would not appear this way when viewed from the side. The original liquid crystal display 10 is flat, and the perspective projection of the tetrapod is projected onto the flat image-forming surface. Here, a tetrapod in three-dimensional space is virtually depicted as a perspective projection.

[0027] This three-dimensional object's orientation can be controlled by hand gestures directed at it. The three-dimensional object T is displayed at the center of the imaging region G. In other words, by fixing the center point (center of gravity) of the three-dimensional object T at the center of the imaging region G, the degrees of freedom of the three-dimensional object T's movement are limited to three-dimensional rotation.

[0028] Controlling the rotation of the 3D object T using gestures is done by stroking and moving the surface of the 3D object T as if it were floating in the air (see Figure 3). If the coordinates of the user's hand detected by the motion sensor 7 are inside the coordinates of the surface of the 3D object T, interference between the user's hand and the 3D object T is recognized, and the 3D object T is rotated in accordance with the movement of the user's hand. However, in this case, interference recognition may occur discontinuously due to the unevenness of the surface of the 3D object T. As a result, the user may not be able to perform the intended rotation smoothly.

[0029] Therefore, in this invention, a spherical determination region D containing the three-dimensional object T is defined so that the user can intuitively rotate the three-dimensional object T without being aware of its shape. This spherical determination region D is used instead of the three-dimensional object T to determine interference with the user's hand.

[0030] The center of the spherical determination region D coincides with the center (center of rotation) of the three-dimensional object T, and its radius is sufficient to encompass the entire three-dimensional object T. For example, the distance from the center to the furthest point on the three-dimensional object T can be used as the radius of the spherical determination region D.

[0031] When the system starts up, the coordinates of the user's hand detected by the motion sensor 7 are repeatedly acquired. Each time coordinates are acquired, it is determined whether or not these coordinates are located within the judgment area D. If the hand coordinates are within the judgment area D, the movement of the user's hand is calculated. Here, the movement of the user's hand is calculated as the difference between the previously acquired coordinates and the currently acquired coordinates. The calculated movement of the user's hand is converted into a rotation vector with the center of the 3D object T (center of rotation) as the origin, and the 3D object T is rotated and redrawn. In this way, the movement of the user's hand can be reflected in the rotation of the 3D object T.

[0032] If the user's hand coordinates obtained by motion sensor 7 are outside the judgment area D, it is determined whether the previously obtained coordinates were located inside the judgment area D. If the previously obtained coordinates were not located inside the judgment area D, no special processing is performed here, and the acquisition of the user's hand coordinates is repeated. If the previously obtained coordinates were located inside the judgment area D, it is determined that the user touched the 3D object T with their hand, rotated it, and then released their hand from the 3D object T, maintaining the rotation.

[0033] In other words, the rotation speed of the 3D object T is calculated from the difference between the coordinates obtained this time and the coordinates obtained last time. After the user's hand leaves the judgment area D, the 3D object T continues to rotate according to this rotation speed, regardless of the movement of the user's hand. However, this rotation speed decreases according to a certain damping coefficient.

[0034] If a user wants to stop a 3D object T in a specific pose, they should pull their hand straight out of the detection area D radially from the center to prevent the object T from rotating. To detect this intention, the radial component of the difference between the currently acquired coordinates and the previously acquired coordinates is compared with the other component (circumferential component). If the radial component is greater than the circumferential component, the circumferential component is ignored (replaced with 0), and the 3D object T is fixed in its current position. Conversely, if the radial component is not greater than the circumferential component, the 3D object T is rotated according to the circumferential component.

[0035] The above process will be explained in more detail by referring to the flowchart in Figure 4. First, in step S1, the motion sensor 7 acquires the coordinates of the user's hand. Here, the coordinates of the fingertips are acquired. Therefore, if both hands are used, a maximum of 10 coordinates are acquired. The center of the 3D object T displayed here is the center of rotation, the center of the image formation region, and the origin of the coordinate system. The vectors from this origin of the coordinate system to each coordinate will be called coordinate vectors.

[0036] Next, in step S2, it is determined whether the acquired coordinates are located inside the determination area D. If it is determined that at least one coordinate is located inside the determination area D (YES in step S2), the user's hand movement is calculated as the difference between the previously acquired value and the currently acquired value for each coordinate located inside the determination area D, and a rotation vector v linked to the hand movement is obtained (step S3).

[0037] Specifically, we calculate individual rotation vectors vi (with subscript i from 1 to n, up to 10) using the coordinate vector obtained this time as ri and the coordinate vector obtained last time as ri'. Here, we use X as the cross product and calculate vi simply as vi = (riXri') / |ri|. Assuming that the rotation vector vi is very small, the rotation vector v we are looking for is the average of the rotation vectors vi (Σvi / n).

[0038] Next, the 3D object T is rotated using the rotation vector v obtained (step S4). That is, if the coordinates of each point of the 3D object T are p, the rotated point p' can be found using the following formula. Here, X represents the cross product and · represents the dot product. p'=pcos|v|+|v|-2v(v·p)(1-cos|v|)+|v|-1(vXp)sin|v|

[0039] If, in step S2, it is determined that all acquired coordinates are located outside the determination area D (NO in step S2), then it is determined whether any of the previously acquired coordinates are located inside the determination area D (step S5). If any of the previously acquired coordinates are located inside the determination area D (YES in step S5), it is determined that the user touched the 3D object T with their hand, rotated it, and then released their hand from the 3D object T.

[0040] If the answer in step S5 is YES, then the radial and circumferential components of the difference (ri' - ri) between the coordinates obtained this time and the coordinates obtained last time are compared (step S6). To do this, the cross product and dot product of the coordinate vector obtained this time and the difference are calculated, and it is determined which one is larger.

[0041] In other words, if (ri'-ri)·ri > |(ri'-ri)Xri|, it is determined that the radial component is large (YES in step S6), and the rotation of the 3D object T is stopped and fixed in its current position (step S7). This allows the user to intentionally stop the 3D object T in a specific pose by withdrawing their hand outside the judgment area D.

[0042] If (ri'-ri)·ri≦|(ri'-ri)Xri|, then assuming the circumferential component is large (NO in step S6), it is determined that the user intends to freely rotate the 3D object T even after releasing their hand. Therefore, the rotation vector v is calculated from the coordinate vectors ri and ri' as described above. Even after the user moves their hand out of the judgment area D, the rotation of the 3D object T continues according to the rotation vector v.

[0043] The rotation vector v corresponds to the angular velocity ω (=|v| / d), where d is the interval for acquiring coordinates. While it is possible to continuously rotate the 3D object T with this angular velocity ω, in the real world, even when floating in the air, the rotation gradually dampens due to air resistance. Therefore, a simulation of damping is performed here as well (step S8).

[0044] Since air resistance is proportional to angular velocity ω, the equation of motion is Cω = -Mdω / dt, where C is the coefficient and M is the moment of inertia. To implement this, the display of the 3D object T is updated at coordinate acquisition intervals d, and the angular velocity ω is multiplied by a constant damping coefficient at each interval d. For example, if the damping coefficient is 0.9, the angular velocity will be approximately halved every 7d. Increasing this damping coefficient will result in a slower stopping time.

[0045] As described above, in the aerial image display device according to the present invention, by setting a determination area D that encompasses the displayed three-dimensional object, the user will no longer feel any unintended discomfort when actually manipulating this three-dimensional object with their hands.

[0046] In the above embodiment, operation using both hands is also possible. For example, if the left hand is stopped within the judgment area D and the right hand rotates the 3D object, the movements of the left and right hands are averaged out, making it possible to apply the brakes with the left hand while controlling finer movements with the right hand. [Industrial applicability]

[0047] According to the aerial image display device of the present invention, it becomes possible to move a three-dimensional object floating in the air by directly touching it with one's hands without any sense of discomfort.

[0048] Although the aerial image display device according to the present invention has been described above based on embodiments, the present invention is not limited thereto, and modifications may be made without departing from the spirit of the invention, and if possible, the technologies described in each embodiment or known technologies may be combined.

[0049] In the above embodiment, the determination region D is a spherical region, but the present invention is not limited to this. For example, it may be an ellipsoidal region, or a region of an oblong or oblate sphere, and generally, any region that is larger than the three-dimensional object to be displayed and completely encloses this object is acceptable. Specifically, it can be set as a region of any polyhedron or the like, according to the shape of the three-dimensional object to be displayed.

[0050] Furthermore, although the aerial image display device 1 is equipped with an information processing device 30 in this example, the present invention is not limited to this. For example, the information processing device 30 may be omitted, and the aerial image display device 1 may be provided with an external input terminal for inputting video signals to the liquid crystal display 10 and an external audio input terminal for inputting audio signals to a speaker, and the necessary video and audio signals may be supplied from an external computer or the like.

[0051] Furthermore, although the above embodiment employs a three-dimensional motion sensor consisting of an infrared LED and an infrared camera, the present invention is not limited thereto, and tracking may be performed using a conventional camera that detects visible light.

[0052] Furthermore, the above embodiment employs an aerial image display device using a retrotransmissive optical imaging element. However, the present invention is not limited to this, and for example, an aerial image display device using a retroreflective optical imaging element may also be employed.

[0053] Furthermore, although a liquid crystal display is used in the above embodiment, the present invention is not limited thereto, and an organic EL display or backlit electronic paper may also be used. [Explanation of Symbols]

[0054] 1. Aerial Image Display Device 7. Three-dimensional motion sensor 10 LCD displays 12 Lower enclosure 14 Upper chassis 20 Optical Plates 21 Entrance plane 30 Information Processing Devices 40 Speakers 72 Infrared LEDs 74 Infrared Cameras G imaging region

Claims

1. A program executed on an aerial image display device comprising a computer, an image system controlled by the computer that projects images into the air, and a motion sensor that detects the coordinates of a user's hand in the vicinity of the aerial image projected by the image system and transmits the detection result to the computer, wherein the computer... The steps include controlling the aforementioned video system to project a three-dimensional object having a convex portion as an aerial image, The steps include defining a three-dimensional determination region that is larger than the aforementioned three-dimensional object and encompasses the entirety of the aforementioned three-dimensional object, The steps include determining whether the coordinates of the user's hand acquired by the motion sensor are located inside the determination area, A program that, if the coordinates of the user's hand obtained by the motion sensor are located inside the determination area, causes the program to execute the step of moving the three-dimensional object according to the coordinates of the hand.

2. The program according to claim 1, characterized in that the determination region is a spherical region or a substantially ellipsoidal region including the three-dimensional object.

3. The program according to claim 1, wherein in the step of moving the three-dimensional object, the three-dimensional object is rotated around its center point in accordance with the movement of the user's hand.

4. The program according to claim 1, characterized in that the user's hand movements are detected as movements of multiple fingertips on both hands, and the average of the rotations corresponding to these fingertips is reflected in the rotation of the three-dimensional object.

Images