A program to visualize contact with 3D objects displayed in the air

Abstract

To provide an aerial image display device that enables a user to visually recognize whether or not a hand is touching a three-dimensional object floating in the air when the user operates the object with the hand. [Solution] When a user manipulates a three-dimensional object projected in the air by a floating image display device with their hand, the display of the area near the point of contact between the three-dimensional object and their hand changes when their hand comes into contact with the three-dimensional object. The change in the display of the three-dimensional object can be, for example, a change in display color or brightness. This allows the user to visually recognize whether their hand is actually touching the three-dimensional object, allowing them to experience and manipulate the three-dimensional object more realistically.

Description

[Technical Field]

[0001] The present invention relates to a program for visualizing contact with a three-dimensional object displayed in the air by an aerial image display device. [Background technology]

[0002] An aerial image display device is a device that displays objects in the air where there is nothing. For example, Patent Document 1 shows an aerial image display device that displays images in the air by using the reflection of light. When an aerial image is displayed on an aerial image display device, it appears to float above the surroundings, creating a sense of three-dimensionality. In particular, when an object in 3D space is displayed using perspective projection, it is possible to create the illusion that the three-dimensional object actually exists there.

[0003] In other words, the fact that it is a floating image display device is the cause of this illusion. For example, with a floating image display device, the image displayed has no frame, so you feel as if you are observing a normal object, or because it is in the air and it is difficult to accurately grasp depth, you perceive a thickness in the depth direction of the image, which makes it appear three-dimensional. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Re-tabled publication 2018 / 139141 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in terms of physiological factors related to stereoscopic vision, such as binocular disparity, convergence, and accommodation, aerial images give the same perception as flat images. Therefore, even if you recognize them as 3D images, when you manipulate the 3D object with your hands, there is a tendency for the sense of position in the depth direction to be different from what you would expect.

[0006] Therefore, when touching a three-dimensional object and attempting to perform a gesture such as rotating it, it may be unclear whether the hand is actually touching the three-dimensional object.

[0007] Therefore, an object of the present invention is to provide an aerial image display device that allows a user to visually recognize whether or not their hand is touching a three-dimensional object floating in the air when operating the object with their hand. [Means for solving the problem]

[0008] In order to solve the above problem, one aspect of the present invention provides a program executed on an aerial image display device including a computer, a video system controlled by the computer that projects an image into the air, and a motion sensor that detects the coordinates of a user's hand near the aerial image projected by the video system and transmits the detection result to the computer, the program including the steps of: controlling the video system to project a three-dimensional object as an aerial image; determining whether the coordinates of the user's hand acquired by the motion sensor are located inside the three-dimensional object; and, if it is determined that the coordinates of the user's hand acquired by the motion sensor are located inside the three-dimensional object, The coordinates are regarded as the contact position between the user's hand and the three-dimensional object, and the display of a local area that is a part of the three-dimensional object and is in the vicinity of the contact position is changed to visualize the contact position between the user's hand and the three-dimensional object. Steps and execute In an area of ​​the three-dimensional object other than the local area, the display change for visualizing the contact position is not performed, thereby allowing the user to perceive where the user's hand is in contact with the three-dimensional object. .

[0009] In one embodiment, the change in the display of the three-dimensional object is a change in display color or brightness.

[0010] Furthermore, in one embodiment, the coordinates of the user's hand acquired by the motion sensor are coordinates of the user's fingertips. [Effects of the Invention]

[0011] According to the program for the aerial image display device of the present invention, when manipulating a three-dimensional object floating in the air with one's hand, it is possible to visually recognize whether or not one's hand is touching the three-dimensional object. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a perspective view showing an embodiment of an aerial image display device that executes a program according to the present invention. [Figure 2] FIG. 2 is a cross-sectional view showing the aerial image display device 1 taken along line AA in FIG. [Figure 3] FIG. 3 is a diagram illustrating an embodiment of an aerial image display device that executes a program according to the present invention, showing a display change area that is displayed when a three-dimensional object displayed in the air is touched with a hand. [Figure 4] FIG. 4 is a diagram illustrating an embodiment of an aerial image display device that executes a program according to the present invention, showing a display change area that is displayed when a three-dimensional object displayed in the air is touched and then released by a hand. [Figure 5] FIG. 5 is a diagram for explaining the operation of an embodiment of the program according to the present invention, and is a flowchart showing the process of touching and rotating a three-dimensional object displayed in the air as an object to be operated. DETAILED DESCRIPTION OF THE INVENTION

[0013] An embodiment of a program for an aerial image display device according to the present invention will now be described with reference to the accompanying drawings. Fig. 1 is a perspective view showing an aerial image display device 1 for carrying out an embodiment of the present invention. Fig. 2 is a cross-sectional view of the aerial image display device 1 taken along line A-A in Fig. 1.

[0014] 1 and 2, the aerial image display device 1 includes, as essential components, a liquid crystal display 10 and an optical plate 20. The aerial image display device 1 also includes an information processing device 30 that controls the liquid crystal display 10, a speaker 40, and the like. The liquid crystal display 10, the speaker 40, and the information processing device 30 are housed inside the lower housing 12, and are each connected by signal lines (not shown).

[0015] The information processing device 30 is essentially a small computer and is composed of a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), a storage device for storing various programs and data, an input / output interface, etc. Examples of input / output interfaces include a USB port and a wireless LAN such as Wi-Fi. The information processing device 30 outputs a video signal to the liquid crystal display 10, which displays the aerial image. The information processing device 30 also outputs an audio signal to the speaker 40, which generates voice guidance and sound effects. This information processing device 30 can also be a commercially available general-purpose small personal computer or a general-purpose tablet.

[0016] The liquid crystal display 10 is housed in a lower housing 12 that is rectangular in plan view and has an open top, and is supported almost horizontally with the display screen facing upward. The optical plate 20 is fitted into the upper housing 14 so that its incident surface 21 faces downward and faces diagonally to the display screen of the liquid crystal display 10. Here, the optical plate 20 and the liquid crystal display 10 are fixed at an angle of approximately 45 degrees.

[0017] An example of such an optical plate 20 is a retro-transmission optical imaging element (two-sided orthogonal reflector) described in Japanese Patent Laid-Open Publication No. 2011-175297. This optical imaging element is realized by arranging a large number of mutually orthogonal planar light reflecting portions at a fixed pitch. Alternatively, a two-sided corner reflector, in which reflective surfaces are formed on the side surfaces of a square hole, as described in Japanese Patent No. 4900618, may be used.

[0018] Furthermore, upper housing 14, which constitutes aerial image display device 1, can be easily separated from lower housing 12. Separating upper housing 14 makes it easier to perform maintenance and adjustments inside lower housing 12, and also reduces the height during transportation.

[0019] As shown in Figure 2, light from the display screen of the liquid crystal display 10 enters the optical plate 20, reflects twice inside the optical plate 20, and exits on the opposite side. As a result, a real aerial image is formed in space on the opposite side of the optical plate 20, with the optical plate 20 as the plane of symmetry. In this case, to achieve a clearer aerial image, it is desirable that external light not be added to the light from the liquid crystal display 10. In Figure 2, reference character G indicates the imaging area corresponding to the display surface of the liquid crystal display 10.

[0020] Furthermore, a three-dimensional motion sensor 7 consisting of an infrared LED 72 and a pair of infrared cameras 74 is provided on the front side of the lower housing 12. This motion sensor 7 precisely detects the user's hand movements in three dimensions. For example, the three-dimensional motion sensor 7 can detect the positions of the fingertips of each hand and the center position of the palm using dedicated tracking software. The detection range of the motion sensor 7 is determined by the emission angle of the infrared LED 72 and the angle of view of the infrared camera 74. Such a motion sensor 7 can be used as a commercially available product, such as Ultraleap's Leap Motion Controller 2.

[0021] In particular, the three-dimensional motion sensor 7 is attached so that its tilt angle can be adjusted. That is, the motion sensor 7 is stored in a storage member that allows it to rotate within a certain angular range (here, ±20 degrees) around the rotation axis perpendicular to the paper surface of the cross-sectional view of FIG. 2. Here, the center of the detection range of the motion sensor 7 is adjusted so that it is shifted toward the user from the center of the imaging area G. By adjusting it in this way, the effective operation detection range can be maximized.

[0022] Although not shown here, it is recommended to place the aerial image display device 1 on a table or the like at an appropriate height so that the aerial image can be viewed at eye level. In particular, a device that comes with an elevating device and allows the height to be freely adjusted is suitable. An example of such an elevating device is the Mario N electric elevating device sold by Yamato Metal Works.

[0023] Typically, such aerial image displays are used as administrative input devices for reception systems and the like. That is, by displaying a non-contact interface screen as aerial images, users can operate the non-contact interface screen using gestures such as touching it with their fingers. The operation is detected by a motion sensor 7 consisting of an infrared LED 72 and an infrared camera 74, and a corresponding operation signal is sent to the information processing device 30, where the specified processing is performed. The operation interface includes controls such as buttons, check boxes, and drop-down menus. Therefore, a completely non-contact interface can be realized with the same ease of use as a conventional touch panel.

[0024] In the present invention, this aerial image display device is used to display a three-dimensional object (3D object) using 3DCG for entertainment or educational purposes. The three-dimensional object is an image with a sense of depth (stereoscopic effect) created by perspectively projecting a three-dimensional object in a computer-defined three-dimensional space onto the screen of the LCD display 10. Furthermore, when the screen of the LCD display 10 is projected in front of the user, the three-dimensional space defined by the computer corresponds to the three-dimensional space in front of the user.

[0025] Here, a sphere S is shown as an example of a three-dimensional object. However, although Figure 2 depicts a sphere as seen from the side, the sphere does not actually appear like this when viewed from the side of the aerial image display device 1. The original liquid crystal display 10 is flat, and a perspective projection of the tetrapods is projected onto the flat imaging surface G. Here, the sphere in three-dimensional space, expressed as a perspective projection, is depicted virtually.

[0026] The rotation of this three-dimensional object S can be controlled by hand gestures on the three-dimensional object S. The three-dimensional object S is displayed at the center of the imaging area G. In other words, by fixing the central point (center of gravity) of the three-dimensional object S at the center of the imaging area G, the degree of freedom of movement of the three-dimensional object S is limited to three-dimensional rotation.

[0027] The rotation of the three-dimensional object S is controlled by a gesture, just like stroking the surface of the three-dimensional object S floating in the air. If the coordinates of the user's hand detected by the motion sensor 7 are inside the coordinates of the surface of the three-dimensional object S, it is determined that there is interference between the user's hand and the three-dimensional object S, and the three-dimensional object S is rotated in accordance with the movement of the user's hand.

[0028] Due to psychological factors such as shadows and the fact that it is a mid-air image, the user perceives the three-dimensional object S as three-dimensional, but binocular disparity, convergence, and focal accommodation indicate that the three-dimensional object S is located on the plane of the imaging area G. Naturally, the user cannot rely on the sense of touch to confirm the position of the surface.

[0029] Therefore, in the present invention, when the user's hand touches the three-dimensional object S, specifically when the coordinates of each fingertip reach the surface of the three-dimensional object S, or more practically, when the coordinates enter inside the surface of the three-dimensional object S, a visual change occurs near the coordinates. This allows the user to recognize that they are actually touching the three-dimensional object S and the position where they have touched.

[0030] The visual change is achieved by changing the color (brightness) of the surface of the three-dimensional object S near the fingertip. In the example of Figure 1, a circular display change area V is created around the index finger. For example, the brightness of the surface of the three-dimensional object S is maximized in this display change area V. If the display change area V is too wide, the three-dimensional display of the three-dimensional object S will be impaired, so a circular area of ​​maximum brightness with a diameter of, for example, 2 to 3 cm is displayed at the center of the fingertip, slightly extending beyond the periphery of the fingertip.

[0031] This allows the user to visually know that the tip of the finger has reached the surface of the three-dimensional object S, but if the user further removes the finger from the surface of the three-dimensional object S, the display change area V disappears, the surface of the three-dimensional object S returns to its original state, and the user realizes that the tip of the finger has left the surface of the three-dimensional object S.

[0032] It is also possible to use a display that enhances the visual effect more than a circular area of ​​maximum brightness. For example, instead of making the display change area V uniformly maximum brightness, it is also possible to overlay a color complementary to the color of the three-dimensional object S at that position. For example, if the three-dimensional object S is blue, the display change area V should be orange. It is also possible to use a concentric wave pattern extending from the center of the circular area to the outside.

[0033] To give an example of one implementation, the circle of the display change area V gradually expands from the point of contact between the user's hand and the 3D object S to the periphery. In this case, as shown in Figure 3, the change gradually fades from the center to the periphery and does not expand beyond a certain size. Here, when the user removes their hand from the 3D object S, the change gradually disappears from the center to the periphery as shown in Figure 4.

[0034] In any case, the position of the contact point between the user's hand and the three-dimensional object S in the depth direction changes depending on which position of the three-dimensional object S is touched. When touching various places on the three-dimensional object S and the actually touched position can be understood by the display change area, a sense of depth can be felt by touching here and there. Thereby, it is possible to create an illusion as if the three-dimensional object actually exists there.

[0035] Although only one fingertip is shown in the figure, when multiple fingertips are in contact with the surface of the three-dimensional object S, circular display change areas are displayed at the positions of each fingertip. Also, when touching the three-dimensional object S with both hands, circular display change areas are displayed for each fingertip of both hands.

[0036] Also, when the user's hand touches the three-dimensional object S, a visual change is caused in the vicinity of its coordinates, and a sound effect is generated by the speaker 40. Thereby, not only visually but also auditorily, the contact with the three-dimensional object S can be felt.

[0037] Here, a sphere is exemplified as the three-dimensional object S, but a general 3D model obtained by a polyhedron, 3D scan, etc. may also be used. In that case, the coordinates with the center of the three-dimensional object S as the origin are converted into polar coordinates where the contact point between the user's hand and the three-dimensional object S has a latitude θ = 0, and the surface area where θ < d (for example, d is π / 6, etc.) is set as the display change area.

[0038] The overall flow will be described in more detail by referring to the flowchart of FIG. 5 for the above processing. First, in step S1, the coordinates of the user's hand are acquired by the motion sensor 7. Here, the coordinates of the fingertip are acquired. Therefore, when using both hands, a maximum of 10 coordinates are acquired. Note that the center of the three-dimensional object S displayed here is the center of rotation, the center of the imaging area, and the origin of the coordinate system. The vector from the origin of this coordinate system to each coordinate will be called the coordinate vector.

[0039] Next, in step S2, it is determined whether the acquired coordinates are located inside the three-dimensional object S. If it is determined that at least one coordinate is located inside the three-dimensional object S (YES in step S2), the coordinates on the surface of the three-dimensional object S near that coordinate are selected as the center point (the point of contact between the user's hand and the three-dimensional object S), and a visual display change area is displayed around this center point as described above (step S3). If a display change area is already displayed, the position of the display change area is updated to match the current coordinates.

[0040] Alternatively, for each coordinate located inside the three-dimensional object S, the movement of the user's hand is calculated as the difference between the value previously acquired and the value currently acquired, and a rotation vector v linked to the hand movement is obtained (step S4).

[0041] Specifically, the coordinate vector obtained this time is r i , the previously obtained coordinate vector is r i ', the individual rotation vectors v i (Subscript i is 1 to n: maximum 10) is calculated. Here, X is the cross product, and simply v i =(r i Xr i ') / |r i | Calculate the rotation vector v i is very small, the rotation vector v we want is the rotation vector v i The average of (Σv i / n).

[0042] Next, the three-dimensional object S is rotated using the obtained rotation vector v (step S5). The visual display change area also rotates in synchronization. That is, if the coordinates of each point on the three-dimensional object S are p, the point p' after rotation can be obtained using the following equation. Here, X means the cross product, and · means the dot product.

[0043] p'=pcos|v|+|v| -2 v(v·p)(1-cos|v|)+|v| -1 (vXp)sin|v|

[0044] If it is determined in step S2 that all of the acquired coordinates are located outside the three-dimensional object S (NO in step S2), it is determined whether any of the previously acquired coordinates is located inside the three-dimensional object S (step S6). If any of the previously acquired coordinates is located inside the three-dimensional object S (YES in step S6), it is determined that the user touched the three-dimensional object S with their hand, rotated it, and then released their hand from the three-dimensional object S.

[0045] If the answer is YES in step S6, the above visual change at the point of contact between the user's hand and the three-dimensional object S is terminated. For example, the display change area is made to disappear the moment the user's hand leaves the three-dimensional object S, and the three-dimensional object S is returned to its original state and displayed. Alternatively, as shown in FIG. 4, the display change area is gradually returned to its original state from the center to the periphery.

[0046] Then, the difference between the coordinates obtained this time and the coordinates obtained last time (r i 'ーr i ) are compared with each other (step S8). This can be done by calculating the cross product and inner product of the currently acquired coordinate vector and the difference, and determining which is larger.

[0047] That is, (r i 'ーr i )·r i >|(r i 'ーr i )Xr i If |, it is determined that the radial component is large (YES in step S8), and the rotation of the three-dimensional object S is stopped and fixed at the current position (step S9). This allows the user to intentionally stop the three-dimensional object S in a specific posture by pulling their hand out from the three-dimensional object S.

[0048] (r i 'ーr i )·r i ≦|(r i 'ーri )Xr i If |, the component in the circumferential direction is determined to be large (NO in step S8), and it is determined that the user intends to freely rotate the three-dimensional object S even after releasing the hand. Therefore, the coordinate vector r i , r i ', the rotation vector v is calculated as described above. Even after the user moves their hand out of the 3D object S, the 3D object S continues to rotate according to the rotation vector v.

[0049] The rotation vector v corresponds to an angular velocity ω (=|v| / d), where d is the coordinate acquisition interval. It is possible to continue rotating the three-dimensional object S at this angular velocity ω, but in the real world, even if the object is floating in the air, the rotation gradually decays due to air resistance. Therefore, a decay simulation is also performed here (Step S10).

[0050] Since air resistance is proportional to angular velocity ω, the equation of motion is Cω=-Mdω / dt, where C is the coefficient and M is the moment of inertia. To implement this, the display of the three-dimensional object S is updated at coordinate acquisition intervals of d, and the angular velocity ω is multiplied by a constant damping coefficient at each interval d. For example, if the damping coefficient is 0.9, the angular velocity will be reduced by approximately half every 7d. Increasing this damping coefficient will result in the object stopping more slowly. [Industrial Applicability]

[0051] According to the aerial image display device of the present invention, when a user operates a three-dimensional object floating in the air with his or her hand, the user can visually recognize whether or not his or her hand is touching the three-dimensional object, thereby providing the user with an application that gives a more realistic feel.

[0052] The above describes the aerial image display device according to the present invention based on an embodiment, but the present invention is not limited to this, and modifications may be made within the scope of the spirit of the present invention, and if possible, the techniques described in each embodiment may be combined, or publicly known techniques may be combined, etc.

[0053] In the above embodiment, the color and brightness of the surface of the three-dimensional object are changed near the coordinates of the hand to visualize the contact of the user's hand with the three-dimensional object, but the present invention is not limited to this. For example, the contact may be visualized by generating distortion on the surface of the three-dimensional object near the contact point. Another effective display method is to display the distortion as a surface wave propagating from the contact point in a concentric circle outward while attenuating.

[0054] Furthermore, in the above embodiment, the three-dimensional object does not deform, but it may deform appropriately due to contact with the user's hand. Even in this case, the contact between the user's hand and the three-dimensional object is visualized by changing the color and brightness of the contact area (deformed area). For example, when a user makes a gesture of pinching and pulling a part of the three-dimensional object, that part deforms and stretches. The color of the deformed part then changes. Furthermore, when the user releases the deformed part, the shape of the three-dimensional object returns to its original shape.

[0055] Furthermore, in the above embodiment, the information processing device 30 is mounted on the aerial image display device 1, but the present invention is not limited to this. For example, the information processing device 30 may be omitted, and the aerial image display device 1 may be provided with an external input terminal for inputting a video signal to the liquid crystal display 10 and an external audio input terminal for inputting an audio signal to the speaker, and the necessary video and audio signals may be supplied from an external computer or the like.

[0056] Furthermore, in the above embodiment, a three-dimensional motion sensor consisting of an infrared LED and an infrared camera is used, but the present invention is not limited to this, and tracking may also be performed using a normal camera that detects visible light.

[0057] Furthermore, in the above embodiment, an aerial image display device using a retro-transmissive optical imaging element is used, but the present invention is not limited to this, and for example, an aerial image display device using a retro-reflective optical imaging element may also be used.

[0058] Furthermore, although a liquid crystal display is used in the above embodiment, the present invention is not limited to this, and an organic EL display, electronic paper with a backlight, or the like may also be used. [Explanation of symbols]

[0059] 1. Aerial image display device 7. Three-dimensional motion sensor 10 LCD display 12 Lower housing 14 Upper housing 20 Optical Plate 21 Entrance plane 30 Information processing equipment 40 speakers 72 infrared LEDs 74 Infrared Camera G Imaging area V Display change area

Claims

1. A program executed on an aerial image display device including a computer, an image system controlled by the computer and projecting an image into the air, and a motion sensor that detects the coordinates of a user's hand moving near the aerial image projected by the image system and transmits the detection results to the computer, the program including: controlling the imaging system to project a three-dimensional object as an aerial image; determining whether the coordinates of the user's hand acquired by the motion sensor are located inside the three-dimensional object; and when it is determined that the coordinates of the user's hand acquired by the motion sensor are located inside the three-dimensional object, the coordinates are regarded as a contact position between the user's hand and the three-dimensional object, and the contact position between the user's hand and the three-dimensional object is visualized by changing the display of a local area that is a part of the three-dimensional object and is in the vicinity of the contact position. A program that allows a user to perceive where on the three-dimensional object the user's hand is touching by not making the display change to visualize the contact position in areas of the three-dimensional object other than the local area.

2. 2. The program according to claim 1, wherein the change in the display of the three-dimensional object is a change in display color or brightness.

3. 2. The program according to claim 1, wherein the coordinates of the user's hand acquired by the motion sensor are coordinates of the user's fingertips.

Images