Interface device and interface system

Abstract

An interface device (2) is provided with: a detection unit (21) that detects the three-dimensional position of a detection target in a virtual space (K); and a projection unit (20) that projects the aerial image (S) into a virtual space that is divided into a plurality of operation spaces in which operations that can be executed by a user when the three-dimensional position of the object to be detected detected by the detection unit is included are identified, the projection unit (20) projecting the aerial image (S) into a virtual space that is divided into a plurality of operation spaces in which the three-dimensional position of the object to be detected by the detection unit is included. The boundary position of each operation space in the virtual space is shown by the aerial image projected by the projection unit.

Description

Technical Field

[0001] The present disclosure relates to an interface device and an interface system. Background Art

[0002] In the past, as an operation input technology for electronic devices, a technology has been proposed in which a user can perform non-contact operation input by operating a virtual space set in space. Regarding such a technology, Patent Document 1 discloses a display device having a function of controlling operation input on a display screen based on a user's remote operation.

[0003] The display device includes two cameras for capturing a range including a user viewing a display screen, and detects a second point representing a user reference position relative to a first point representing a camera reference position and a third point representing a position of a user's finger from the captured images of the cameras, and sets a virtual surface space at a position of a predetermined length along a first direction from the second point in the space, and determines and detects a predetermined operation of the user based on the degree of entry of the user's finger relative to the virtual surface space. Then, the display device generates operation input information based on the results of the determination and detection, and controls the operation of the display device based on the generated information.

[0004] Here, the virtual plane space is a space without a physical entity and is set as a position coordinate of a three-dimensional space by calculation based on a processor or the like of the display device. The virtual plane space is constituted as a roughly rectangular parallelepiped or flat plate-shaped space sandwiched between two virtual planes. The two virtual planes are a first virtual plane located in front of the user and a second virtual plane located in the back thereof.

[0005] For example, when the point of the finger position reaches the first virtual surface from the first space in front of the first virtual surface and further enters the second space on the inner side behind the first virtual surface, the display device automatically switches to a state of accepting a prescribed operation and displays a cursor on the display screen. In addition, when the point of the finger position passes through the second space and reaches the second virtual surface and further enters the third space on the inner side behind the second virtual surface, the display device determines and detects a prescribed operation (such as touching, tapping, sliding, and pinching the second virtual surface). When the prescribed operation is detected, the display device controls the operation of the display device including the display control of the GUI of the display screen based on the position coordinates of the detected finger position point and the operation information indicating the prescribed operation.

[0006] Prior Art Literature

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent Application Publication No. 2021-15637 Summary of the invention

[0009] Problems to be solved by the invention

[0010] In the display device described in Patent Document 1 (hereinafter also referred to as "conventional device"), a mode for accepting a prescribed operation and a mode for determining and detecting a prescribed operation are switched according to the point of the user's finger position in the virtual plane space. However, in the conventional device, it is difficult for the user to visually confirm at which position in the virtual plane space each of the modes is switched, in other words, it is difficult to visually confirm the boundary positions of the spaces constituting the virtual plane space (the boundary position between the first space and the second space and the boundary position between the second space and the third space).

[0011] The present disclosure has been made to solve the above-mentioned problems, and an object of the present disclosure is to provide a technology capable of visually confirming the boundary positions of a plurality of operation spaces constituting a virtual space that is an operation target of a user.

[0012] Means for solving problems

[0013] The interface device disclosed in the present invention is characterized in that it comprises: a detection unit, which detects the three-dimensional position of a detection object in a virtual space; and a projection unit, which projects an aerial image into the virtual space, wherein the virtual space is divided into a plurality of operation spaces, wherein the plurality of operation spaces determine operations that a user can perform while including the three-dimensional position of the detection object detected by the detection unit, and the boundary position of each operation space in the virtual space is shown by using the aerial image projected by the projection unit.

[0014] In addition, the interface device disclosed in the present invention is an interface device capable of executing operations of an application displayed on a display, and is characterized in that the interface device comprises: a detection unit, which detects the three-dimensional position of a detection object in a virtual space divided into multiple operation spaces; at least one boundary defining unit, which shows the boundary of each operation space and is composed of lines or surfaces; and a boundary display unit, which is composed of points, lines or surfaces and is provided with at least one boundary of each operation space that can be visually confirmed. When the three-dimensional position of the detection object detected by the detection unit is contained in the virtual space, the detection object can execute multiple operations on the application that have established corresponding relationships with each operation space.

[0015] In addition, the interface system disclosed in the present invention is characterized in that it comprises: a detection unit, which detects the three-dimensional position of a detection object in a virtual space; a projection unit, which projects an aerial image into the virtual space; and a display, which displays image information, wherein the virtual space is divided into a plurality of operation spaces, and the plurality of operation spaces determine the operations that a user can perform while containing the three-dimensional position of the detection object detected by the detection unit, and the boundary position of each operation space in the virtual space is shown by using an aerial image projected by the projection unit, and the aerial image projected by the projection unit can be visually confirmed by the user together with the image information displayed on the display.

[0016] In addition, the interface system disclosed in the present invention is characterized in that it comprises: a detection unit, which detects the three-dimensional position of a detection object in a virtual space divided into a plurality of operation spaces; an acquisition unit, which acquires the three-dimensional position of the detection object detected by the detection unit; a projection unit, which projects an aerial image showing the boundary positions of each operation space in the virtual space; a determination unit, which determines the operation space containing the three-dimensional position of the detection object based on the three-dimensional position of the detection object acquired by the acquisition unit and the boundary positions of each operation space in the virtual space; and an operation information output unit, which outputs operation information for executing a prescribed operation on an application displayed on a display device using at least a determination result based on the determination unit, wherein each operation space corresponds to at least any one of a plurality of operations performed on the application using a mouse or a touch panel, and a correspondence is established between continuous and different operations performed on the application and adjacent operation spaces in each operation space.

[0017] In addition, the interface system disclosed in the present invention is characterized in that it comprises: a detection unit, which detects the three-dimensional position of a detection object in a virtual space divided into a plurality of operation spaces; an acquisition unit, which acquires the three-dimensional position of the detection object detected by the detection unit; a projection unit, which projects an aerial image showing the boundary positions of each operation space in the virtual space; a determination unit, which determines the operation space containing the three-dimensional position of the detection object based on the three-dimensional position of the detection object acquired by the acquisition unit and the boundary positions of each operation space in the virtual space; and an operation information output unit, which outputs operation information for executing a prescribed operation on an application displayed on a display device using at least a determination result based on the determination unit, the operation information output unit determines the action of the detection object based on the three-dimensional position of the detection object, establishes a correspondence between the action of the detection object within each operation space or across each operation space and at least any one of a plurality of operations performed on the application using a mouse or a touch panel, and links the prescribed operation on the application with the action of the detection object.

[0018] Effects of the Invention

[0019] According to the present disclosure, since it is configured as described above, it is possible to visually check the boundary positions of a plurality of operation spaces constituting a virtual space that is an operation target of a user. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1A is a perspective view showing a structural example of the interface system according to Embodiment 1, Figure 1B This is a side view showing a configuration example of the interface system according to the first embodiment.

[0021] Figure 2A 1 is a perspective view showing a configuration example of a projection device in Embodiment 1. Figure 2B It is a side view showing a configuration example of the projection device in Embodiment 1.

[0022] Figure 3 This is a diagram showing a basic operation example of the interface system in the first embodiment.

[0023] Figure 4 This is a perspective view showing an example of the arrangement structure of the projection device and the detection device in the interface device according to the first embodiment.

[0024] Figure 5 This is a plan view showing an example of the arrangement structure of the projection device and the detection device in the interface device according to the first embodiment.

[0025] Figure 6 This is a perspective view showing an example of the arrangement structure of the projection device and the detection device in the interface device according to the second embodiment.

[0026] Figure 7 It is a plan view showing an example of the arrangement structure of the projection device and the detection device in the interface device according to the second embodiment.

[0027] Figure 8 It is a side view showing an example of the arrangement structure of the projection device and the detection device in the interface device of the third embodiment.

[0028] Fig. 9 It is a side view showing an example of the arrangement structure of the projection device and the detection device in the interface device of the fourth embodiment.

[0029] Fig.10 This is a diagram showing a configuration example of a conventional aerial image display system.

[0030] Fig.11 This is a diagram showing an example of functional blocks of the interface system of embodiment 5.

[0031] Fig.12This is a flowchart showing an example of the operation in “A. Aerial image projection phase” of the interface system according to the fifth embodiment.

[0032] Fig.13 This is a flowchart showing an example of operation in "B. Control execution phase" of the interface system according to the fifth embodiment.

[0033] Fig.14 This is a flowchart showing an example of actions in "spatial processing A" of the interface system according to the fifth embodiment.

[0034] Fig.15 This is a flowchart showing an example of actions in "spatial processing B" of the interface system of embodiment 5.

[0035] Fig.16 This is a diagram illustrating cursor movement in Implementation Example 5.

[0036] Fig.17 This is a diagram illustrating cursor movement in Implementation Example 5.

[0037] Fig.18 This is a diagram illustrating the fixing of the cursor in Implementation Example 5.

[0038] Fig.19 This is a diagram for explaining the left click in the fifth embodiment.

[0039] Fig. 20 This is a diagram for explaining right-clicking in Implementation Example 5.

[0040] Fig.21 This is a diagram for explaining a double left click in Implementation Example 5.

[0041] Figures 22A to 22D This is a diagram illustrating the continuous movement operation of the pointer in Implementation Example 5.

[0042] Fig.23A is a diagram for explaining the continuous movement operation of a pointer of a conventional device. Fig. 23B This is a diagram illustrating the continuous movement operation of the pointer in Implementation Example 5.

[0043] Fig.24A , Fig. 24B This is a diagram for explaining the scroll operation in the fifth embodiment.

[0044] Fig.25 This is a flowchart showing another example of operation in "B. Control execution phase" of the interface system according to the fifth embodiment.

[0045] Fig.26 This is a flowchart showing an example of actions in the "spatial processing AB" of the interface system according to the fifth embodiment.

[0046] Fig.27Ais a diagram illustrating a left drag operation in Implementation Example 5, Fig.27B This is a diagram illustrating the right drag operation in Implementation Example 5.

[0047] Fig.28A , Fig.28B This is a diagram showing a hardware configuration example of a device control device in the fifth embodiment.

[0048] Fig.29 It is a perspective view showing an example of the arrangement structure of the projection device and the detection device in the interface device of the sixth embodiment.

[0049] Fig.30 It is a plan view showing an example of the arrangement structure of the projection device and the detection device in the interface device of the sixth embodiment.

[0050] Fig.31 This is a front view showing an example of the arrangement structure of the projection device and the detection device in the interface device of the sixth embodiment.

[0051] Fig.32 This is a diagram used to supplement the configuration relationship between the light source and the aerial image in Implementation Example 6.

[0052] Fig.33 This is a perspective view showing a structural example of an interface device according to a seventh embodiment.

[0053] Fig.34 This is a side view showing a structural example of an interface device according to Embodiment 7.

[0054] Fig.35 This is a perspective view showing a structural example of a boundary display unit in the eighth embodiment. DETAILED DESCRIPTION

[0055] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

[0056] Implementation method 1.

[0057] Figure 1A and Figure 1B 1 is a diagram showing a configuration example of the interface system 100 according to the first embodiment. Figure 1A and Figure 1B As shown, the interface system 100 is configured to include a display device 1 and an interface device 2. In addition, Figure 1A is a perspective view showing a structural example of the interface system 100, Figure 1B It is a side view showing a configuration example of the interface device 2.

[0058] <Display device 1>

[0059] For example Figure 1AAs shown, the display device 1 is configured to include a display 10 and a display control device 11 .

[0060] The display 10 displays various screens, typified by a predetermined operation screen R displaying a pointer P operable by a user, under the control of a display control device 11. The display 10 is constituted by, for example, a liquid crystal display or a plasma display.

[0061] The display control device 11 performs control for displaying various screens on the display 10. The display control device 11 is constituted by, for example, a PC (Personal Computer) and a server.

[0062] In Embodiment 1, the user uses the interface device 2 described below to perform various operations on the display device 1. For example, the user uses the interface device 2 described below to operate the pointer P on the operation screen displayed on the display 10 or to execute various instructions to the display device 1.

[0063] <Interface device 2>

[0064] The interface device 2 is a non-contact type device that enables the user to input operations to the display device 1 without direct contact. Figure 1A and Figure 1B As shown, the interface device 2 is configured to include a projection device 20 and a detection device 21 disposed inside the projection device 20 .

[0065] <Projection device 20>

[0066] The projection device 20 projects one or more aerial images S into the virtual space K using, for example, an imaging optical system. The imaging optical system is, for example, an optical system having a light bending surface constituting a plane that bends the optical path of light emitted from a light source.

[0067] For example Figure 1B As shown, the virtual space K is a space without a physical entity that is set within a detectable range that can be detected by the detection device 21 and is divided into a plurality of operation spaces. Figure 1B 2 shows an example in which the virtual space K is set to a posture along the detection direction of the detection device 21, but the virtual space K is not limited to this and can be set to an arbitrary posture.

[0068] In the following description, for easy understanding, the virtual space K is divided into two operation spaces (operation space A and operation space B). Figure 1BAs shown, the boundary position between the operation space A and the operation space B constituting the virtual space K is shown by the aerial image S projected by the projection device 20 .

[0069] Next, refer to Figure 2A and Figure 2B , a specific structural example of the projection device 20 is described. Figure 2A and Figure 2B The example in which the imaging optical system mounted on the projection device 20 is configured to include a beam splitter 202 and a retroreflector 203 is shown. In addition, reference numeral 201 is a light source. Figure 2A 2 is a perspective view showing a structural example of the projection device 20. Figure 2B 2 is a side view showing a structural example of the projection device 20. Figure 2B In the figure, the description of the detection device 21 is omitted.

[0070] The light source 201 is composed of a display device that emits incoherent diffuse light. For example, the light source 201 is composed of a display device including a liquid crystal element and a backlight such as a liquid crystal display, a display device using a self-luminous device such as an organic EL element and an LED element, or a projection device using a projector and a screen.

[0071] The beam splitter 202 is an optical element that separates incident light into transmitted light and reflected light, and is an element whose element surface functions as the above-mentioned light bending surface. The beam splitter 202 is composed of, for example, an acrylic plate and a glass plate. In the case where the beam splitter 202 is composed of an acrylic plate and a glass plate, the intensity of their transmitted light is generally higher than that of reflected light. Therefore, the beam splitter 202 can also be composed of a half mirror in which metal is added to the acrylic plate and the glass plate to increase the reflection intensity.

[0072] In addition, the beam splitter 202 may be configured using a reflective polarizing plate whose reflection and transmission actions vary according to the polarization state of incident light by a liquid crystal element or a thin film element. In addition, the beam splitter 202 may be configured using a reflective polarizing plate whose ratio of transmittance to reflectance varies according to the polarization state of incident light by a liquid crystal element or a thin film element.

[0073] The retroreflective member 203 is a sheet-shaped optical element having a retroreflective property of reflecting incident light in the incident direction as it is. Optical elements that realize retroreflection include bead-type optical elements formed by covering small glass beads in a mirror-like manner, micro-triangular pyramids with convex shapes formed by covering each surface with mirrors, or micro-prism-type optical elements formed by cutting off the center of a triangular pyramid.

[0074] In the projection device 20 having the imaging optical system configured as described above, for example, the light (diffused light) emitted from the light source 201 is specularly reflected on the surface of the beam splitter 202, and the reflected light is incident on the retroreflector 203. The retroreflector 203 retroreflects the incident light and makes it incident on the beam splitter 202 again. The light incident on the beam splitter 202 passes through the beam splitter 202 and reaches the user. In addition, by advancing along the above-mentioned optical path, the light emitted from the light source 201 is re-converged and re-diffused at a position symmetrical to the light source 201 with the beam splitter 202 as the boundary. As a result, the user can perceive the aerial image S in the virtual space K.

[0075] In addition, Figure 2A and Figure 2B 2 shows an example in which the aerial image S is projected in a star shape, but the shape of the aerial image S is not limited thereto and may be any shape.

[0076] In the above description, an example is described in which the imaging optical system provided in the projection device 20 is configured to include the beam splitter 202 and the retroreflective member 203 , but the structure of the imaging optical system is not limited to the above example.

[0077] For example, the imaging optical system may be configured to include a dihedral corner reflector array element. The dihedral corner reflector array element is an element configured by arranging a plurality of two orthogonal mirror elements (reflecting mirrors) on a flat plate (substrate), for example.

[0078] The dihedral corner reflector array element has the following functions: using one of the two mirror elements to reflect the light incident from the light source 201 arranged on one side of the plate, and then using the mirror element on the other side to reflect the reflected light so that it passes through the other side of the plate. If the path of the light is observed from the side, the light entry path and the light exit path are plane-symmetrical across the plate. That is, the element surface of the dihedral corner reflector array element functions as the above-mentioned light bending surface, and the real image generated by the light source 201 located on one side of the plate is imaged as an aerial image S at a plane-symmetrical position on the other side.

[0079] When the imaging optical system is composed of a dihedral corner reflector array element, in the configuration using the above-mentioned retroreflector 203, the dihedral corner reflector array element is arranged at the position where the beam splitter 202 is arranged. In this case, the retroreflector 203 is omitted.

[0080] In addition, the imaging optical system may also be configured to include a lens array element, for example. The lens array element is, for example, an element configured by arranging a plurality of lenses on a flat plate (substrate). In this case, the element surface of the lens array element functions as the light bending surface, and the real image generated by the light source 201 arranged on one side of the plate is imaged as an aerial image S at a surface-symmetrical position on the other side. In addition, in this case, the distance from the light source 201 to the element surface is roughly proportional to the distance from the element surface to the aerial image S.

[0081] In addition, the imaging optical system may also be configured to include a holographic element, for example. In this case, the element surface of the holographic element functions as the light bending surface. By projecting light from the light source 201 as reference light onto the holographic element, the holographic element outputs light in a manner that reproduces the phase information of the light stored in the element. Thus, the holographic element forms an image as an aerial image S at a surface-symmetrical position on the other side of the holographic element, using the real image generated by the light source 201 disposed on one side of the element.

[0082] <Detection device 21>

[0083] The detection device 21 detects the three-dimensional position of a detection target (eg, a user's hand) existing in the virtual space K, for example.

[0084] As a detection method of the detection device 21 for the detection object, for example, the following method can be cited: irradiating infrared rays toward the detection object, and calculating the position of the detection object in the depth direction within the shooting angle of view of the detection device 21 based on the detection of its time of flight (ToF) and infrared pattern. In embodiment 1, the detection device 21 is composed of, for example, a three-dimensional camera sensor or a two-dimensional camera sensor that can also detect infrared wavelengths. In this case, the detection device 21 can calculate the position of the detection object in the depth direction within the shooting angle of view, and can detect the three-dimensional position of the detection object.

[0085] In addition, the detection device 21 may be composed of a device that detects the position in the depth direction in one dimension, such as a line sensor. In addition, when the detection device 21 is composed of a line sensor, a plurality of line sensors are arranged according to the detection range, so that the three-dimensional position of the detection object can be detected. In addition, in Embodiment 4, an example in which the detection device 21 is composed of the above-mentioned line sensor is described in detail.

[0086] In addition, for example, the detection device 21 may also be composed of a stereo camera device composed of a plurality of cameras. In this case, the detection device 21 performs triangulation based on feature points detected within the shooting angle of view and detects the three-dimensional position of the detection object.

[0087] <Virtual Space K>

[0088] Next, refer to Figure 3 , a specific structural example of the virtual space K is explained.

[0089] As described above, the virtual space K is a space without a physical entity that is set within a detectable range that can be detected by the detection device 21, and is divided into an operation space A and an operation space B. Figure 3 As shown, the virtual space K is set to a rectangular parallelepiped shape as a whole and is divided into two operation spaces (operation space A and operation space B). In the following description, the operation space A is also referred to as the "first operation space" and the operation space B is also referred to as the "second operation space".

[0090] In this case, the aerial image S projected by the projection device 20 into the virtual space K shows the boundary position between the two operation spaces, namely, the operation space A and the operation space B. Figure 3 Two aerial images S are projected in the image. These aerial images S are projected on a closed plane (hereinafter, this plane is also particularly referred to as a "boundary surface") that divides the operation space A and the operation space B. Figure 3 In FIG. 1 , an example in which two aerial images S are projected is shown, but the number of aerial images S is not limited thereto, and may be one, or may be three or more. Figure 3 As shown, the short side direction of the boundary surface is defined as the X-axis direction, the long side direction is defined as the Y-axis direction, and the direction orthogonal to the X-axis direction and the Y-axis direction is defined as the Z-axis direction.

[0091] In addition, in the operation space A and the operation space B, a correspondence is established with the operation that the user can perform when the three-dimensional position of the detection object detected by the detection device 21 is included in each operation space. In addition, in the following description, for easy understanding, the case where the detection object of the detection device 21 is the user's hand is described as an example. In this case, it is assumed that the detection device 21 detects the three-dimensional position of the user's hand in the virtual space K, especially the three-dimensional positions of the five fingers of the user's hand in the virtual space K.

[0092] For example, as an operation that can be performed by the user, the operation of the pointer P is associated with the operation space A. Specifically, for example, when the user places his hand in the operation space A, that is, when the three-dimensional positions of the five fingers of the user's hand detected by the detection device 21 are all included in the operation space A, if the user moves his hand in the operation space A, the pointer P ( Figure 3 In addition, Figure 3 On the left side of , as a conceptual diagram, the pointer P is displayed on the operation space A, but in fact, the pointer P displayed on the operation screen R of the display 10 moves.

[0093] In the following description, “the three-dimensional position of the user's hand is included in the operation space A” means “the three-dimensional positions of the five fingers of the user's hand are all included in the operation space A”. In the following description, “the user operates the operation space A” means “the user moves the hand in a state where the three-dimensional position of the user's hand is included in the operation space A”.

[0094] In addition, when the user moves his hand from the operation space A across the boundary position (boundary surface) and into the operation space B, that is, when the three-dimensional positions of the five fingers of the user's hand detected by the detection device 21 are all included in the operation space B, the movement of the pointer P displayed on the operation screen R in the display 10 is fixed ( Figure 3 In addition, Figure 3 , square brackets displayed at four corners of the pointer P indicate that the movement of the pointer P is fixed.

[0095] At this time, even if the user moves the hand in the operation space B, the pointer P will not move. On the other hand, if the user moves the hand in a predetermined pattern in the operation space B, the command (left click, right click, etc.) corresponding to the action (gesture) can be executed. That is, for example, as an operation that can be executed by the user, the input (execution) of the command is associated with the operation space B.

[0096] In the following description, “the three-dimensional position of the user's hand is included in the operation space B” means “the three-dimensional positions of the five fingers of the user's hand are all included in the operation space B”. In the following description, “the user operates the operation space B” means “the user moves the hand in a state where the three-dimensional position of the user's hand is included in the operation space B”.

[0097] In this way, the user moves the pointer P displayed on the operation screen R of the display 10 by operating the operation space A, and then executes the command corresponding to the hand movement by operating the operation space B. In other words, the operation performed by the user, especially the continuous operation, establishes a correspondence relationship with the adjacent operation spaces A and B. Here, "continuous operation" refers to an operation that is usually assumed to be performed continuously in time, such as the user moving the pointer P displayed on the operation screen R of the display 10 and then executing a prescribed command.

[0098] In addition, the operation with continuity may be associated with all adjacent operation spaces in each operation space, or the operation with continuity may be associated with a portion of adjacent operation spaces. In other words, the operation without continuity may be associated with other adjacent operation spaces.

[0099] In addition, for example, Figure 3 The two aerial images S shown are projected onto a closed plane (boundary surface) that divides the adjacent operation space A and the adjacent operation space B. That is, these aerial images S show the adjacent boundary of the two adjacent operation spaces.

[0100] In addition, the range of the operating space A is, for example, Figure 3 The range of the operation space B is from the position of the boundary surface of the projected aerial image S to the upper limit position of the detectable range of the detection device 21 in the Z-axis direction. Figure 3 The range in the Z-axis direction from the position of the boundary surface of the projected aerial image S to the lower limit position of the detectable range of the detection device 21.

[0101] In addition, Figure 3 On the right side of , the aerial image SC is an aerial image projected by the projection device 20 when the user crosses the boundary position (boundary surface) from the operation space A and puts the hand into the operation space B. The aerial image SC is an aerial image that shows the lower limit position of the detectable range of the detection device 21 and shows the reference position for dividing the operation space B into left and right spaces when observed from the user side. The aerial image SC is projected by the projection device 20 near the lower limit position of the detectable range of the detection device 21 and near the approximate center of the operation space B in the X-axis direction. The aerial image S exists on a plane (boundary surface) whose coordinate position in the Z-axis direction is 0, while the aerial image SC exists in an area where the coordinate position in the Z-axis direction is negative. As a result, the user can easily understand to what extent the hand can be lowered in the operation space B, and can execute instructions that require left and right instructions such as left and right clicks. The input method of instructions such as left and right clicks will be described later.

[0102] Next, refer to Figure 4 and Figure 5 , an example of the configuration structure of the projection device 20 and the detection device 21 in the interface device 2 is described. Figure 4 is a perspective view showing an example of the configuration structure of the projection device 20 and the detection device 21 in the interface device 2, Figure 5 It is a plan view showing an example of the arrangement structure of the projection device 20 and the detection device 21 in the interface device 2.

[0103] In the following description, for easy understanding, the imaging optical system of the projection device 20 is configured to include Figure 2A and Figure 2B The case of the beam splitter 202 and the retro-reflective element 203 shown will be described as an example.

[0104] In addition, in the following description, the following situation is taken as an example: the projection device 20 is constructed to include two rod-shaped light sources 201a and 201b, and the light emitted from the two light sources 201a and 201b is re-converged and re-diffused at positions symmetrical to the light sources 201a and 201b with the beam splitter 202 as the boundary, thereby projecting two aerial images Sa and Sb composed of line (straight) graphics in the virtual space K.

[0105] In addition, in the following description, the following situation is taken as an example: the detection device 21 is composed of a camera device, which can detect the three-dimensional position of the user's hand by irradiating infrared light as detection light and receiving infrared light reflected by the user's hand as the detection object.

[0106] like Figure 4 and Figure 5 As shown, the detection device 21 is arranged inside the projection device 20. More specifically, the detection device 21 is arranged inside the imaging optical system provided in the projection device 20, and is particularly arranged inside the beam splitter 202 constituting the imaging optical system.

[0107] In addition, at this time, the shooting angle of view (hereinafter also referred to as "angle of view") of the detection device 21 is set to a range that does not reflect the aerial images Sa and Sb projected by the projection device 20. Figure 4 and Figure 5 , the viewing angle of the detection device 21 is set to a range that does not enter the aerial images Sa and Sb projected by the projection device 20, and is set to fall within the internal area U determined by the two aerial images Sa and Sb. In other words, the projection device 20 images the aerial images Sa and Sb in the virtual space K in such a way that the aerial images Sa and Sb include the viewing angle of the detection device 21. And, if the point is observed from the side of the aerial images Sa and Sb, the aerial images Sa and Sb are imaged at a position where the detection device 21 can suppress the decrease in the detection accuracy of the three-dimensional position of the user's hand (detection object).

[0108] Here, the "internal area determined by the two aerial images Sa and Sb" refers to a rectangular area drawn on the boundary surface of the projected two aerial images Sa and Sb using these connecting lines and the two aerial images Sa and Sb when connecting the ends of one side of the aerial images Sa and Sb facing each other and connecting the ends of the other side of the aerial images Sa and Sb facing each other.

[0109] In addition, here, the case where two aerial images are projected is explained as an example, but the same is true for the case where three or more aerial images composed of linear (straight) graphics are projected. For example, "the internal area determined by the three aerial images Sa, Sb, and Sc" refers to the area drawn on the boundary surface of the three aerial images Sa, Sb, and Sc using these connecting lines and the three aerial images Sa, Sb, and Sc when the ends of the adjacent aerial images Sa, Sb, and Sc are connected to each other. In addition, the projection device 20 images the three aerial images in the virtual space K in such a way that the three aerial images contain the perspective of the detection device 21. And, if the point is observed from the aerial image, the three aerial images are imaged at positions where the detection device 21 can suppress the decrease in the detection accuracy of the three-dimensional position of the user's hand (detection object).

[0110] In addition, when the aerial image S is not composed of a line (straight line)-shaped figure but is composed of a figure having a closed area such as a frame-shaped figure or a circular figure, the "internal area determined by the aerial image S" refers to, for example, the closed area surrounded by the frame line of the frame-shaped figure or the closed area surrounded by the circumference of the circular figure. In addition, the projection device 20 images the aerial image in the virtual space K in such a way that the closed area of ​​the aerial image composed of the figure having a closed area includes the viewing angle of the detection device 21. And, if the point is observed from the aerial image, the aerial image is imaged at a position where the detection device 21 can suppress the decrease in the detection accuracy of the three-dimensional position of the user's hand (detection object).

[0111] In this way, by configuring the detection device 21 inside the imaging optical system of the projection device 20, particularly at a position closer to the inside than the beam splitter 202 constituting the imaging optical system, in the detection device 21 that requires a specified detection distance to the hand of the user as the detection object, the specified detection distance can be ensured, and the size of the projection device 20 including the structure of the imaging optical system can be miniaturized.

[0112] Furthermore, by arranging the detection device 21, in particular, at a position further inside than the beam splitter 202 constituting the imaging optical system, it also contributes to stabilizing the detection accuracy of the user's hand by the detection device 21.

[0113] For example, when the detection device 21 is exposed to the outside of the projection device 20, it is conceivable that the detection accuracy of the three-dimensional position of the user's hand will be reduced due to external factors such as dust, dirt, and water. In addition, when the detection device 21 is exposed to the outside of the projection device 20, since external light such as sunlight or illumination light is incident on the sensor part of the detection device 21, it is conceivable that the external light will become noise when the three-dimensional position of the user's hand is detected.

[0114] In this regard, in Embodiment 1, since the detection device 21 is arranged at a position closer to the inside than the beam splitter 202 constituting the imaging optical system, it is possible to suppress the decrease in the detection accuracy of the three-dimensional position of the user's hand caused by external factors such as dust, dirt, and water. In addition, for example, by adding an optical material such as a phase polarization plate to the surface (the surface facing the user side) of the beam splitter 202 that absorbs light other than the infrared light emitted by the detection device 21 and the light emitted from the light sources 201a and 201b, it is also possible to suppress the decrease in detection accuracy caused by external light such as sunlight or illumination light.

[0115] Furthermore, when a phase polarization plate is added to the surface of the beam splitter 202 (the surface facing the user side) as described above, in the interface device 2, the phase polarization plate makes it difficult to visually confirm the detection device 21 itself from the outside of the projection device 20. Therefore, in the interface device 2, the user will not be given the impression that they are being photographed by a camera, and an effect in terms of appearance can be expected.

[0116] In the interface device 2, the viewing angle of the detection device 21 is set to a range where the aerial images Sa and Sb projected by the projection device 20 are not reflected. Figure 4 and Figure 5 As described above, the viewing angle of the detection device 21 is set to a range that does not enter the aerial images Sa and Sb projected by the projection device 20, and is set to fall within the internal area U defined by the two aerial images Sa and Sb. As a result, in the interface device 2, it is possible to suppress a decrease in the resolution of the aerial images Sa and Sb. This will be described in detail below.

[0117] For example, International Publication No. 2018-78777 discloses an aerial image display system (hereinafter also referred to as a “conventional system”) having a structure similar to the interface device 2 of Implementation Example 1.

[0118] The aerial image display system comprises: an image display device, which displays an image on a screen; an imaging component, which forms an image of image light including the displayed image as a real image in the air; a wavelength selective reflection component, which is arranged on the incident surface side of the image light in the imaging component and has the characteristics of transmitting visible light and reflecting invisible light; and a camera, which receives invisible light reflected by a subject that performs input operation on the real image and captures an image of the subject composed of an invisible light image.

[0119] In addition, the above-mentioned image display device includes: an input operation determination unit, which obtains an image of the subject from a camera, analyzes the image of the subject and analyzes the input operation content of the subject; a main control unit, which outputs an action control signal based on the input operation content analyzed by the input operation determination unit; and an image generation unit, which generates an image signal reflecting the input operation content according to the action control signal and outputs it to an image display, and the above-mentioned wavelength selection reflection component is configured at a position where the above-mentioned real image enters the viewing angle of the camera.

[0120] exist Fig.10 An example of the structure of an aerial image display system constructed as described above is shown in FIG. Fig.10 In the figure, reference numeral 600 is an image display device, reference numeral 604 is an image display, reference numeral 605 is a light irradiator, and reference numeral 606 is a camera. In addition, reference numeral 610 is a wavelength selective imaging device, reference numeral 611 is an imaging member, and reference numeral 612 is a wavelength selective reflection member. In addition, reference numeral 701 is a half mirror, and reference numeral 702 is a retro-reflective sheet. In addition, reference numeral 503 is a real image.

[0121] exist Fig.10 In the conventional system shown in FIG. 1 , in addition to the display device 604 for emitting image light for forming a real image 503 visually recognized by the user, the image display device 600 also includes a light irradiator 605 for emitting infrared light for detecting the three-dimensional position of the fingers of the user's hand and a camera 606 composed of a visible light camera. Fig.10 In the conventional system shown, a wavelength selective reflection component 612 for reflecting infrared light is added to the surface of the retroreflective sheet 702, so that the infrared light irradiated from the light irradiator 605 is reflected by the wavelength selective reflection component 612 so as to irradiate the position of the user's hand, and a part of the infrared light diffused on the user's fingers, etc. is reflected by the wavelength selective reflection component 612 so as to be incident on the camera 606, thereby enabling the user's position detection, etc.

[0122] However, in the conventional system configured as described above, since the user contacts and operates the real image 503, in other words, since the position of the user's hand to be detected coincides with the position of the real image (air image) 503, the wavelength selective reflection member 612 that reflects infrared light needs to be arranged in the optical path of the image light starting from the display device 604 that irradiates the image light for imaging the real image 503. That is, in the conventional system described above, it is necessary to replace a part of the image light irradiated from the display device 604 with infrared light, as a result, the resolution of the real image 503 may be reduced. In addition, since the wavelength selective reflection member 612 added to the surface of the retroreflective sheet 702 also has an influence on the optical path for imaging the real image 503, it may cause a reduction in the brightness and resolution of the real image 503.

[0123] In contrast, in the interface device 2 of embodiment 1, since the aerial image S shows the boundary position of the operating space A and the operating space B that constitute the virtual space K, it can be said that it is used as a guide. Therefore, the user does not necessarily need to contact the aerial image S. In addition, the detection device 21 does not need to detect the three-dimensional position of the user's hand that contacts the aerial image S.

[0124] Therefore, in the interface device 2 of the first embodiment, the viewing angle of the detection device 21 only needs to be set to a range that does not fall into the aerial images Sa and Sb projected by the projection device 20 and, for example, is set to fall into the internal area U defined by the two aerial images Sa and Sb so that the three-dimensional position of the user's hand in the internal area U can be detected. In this way, in the interface device 2 of the first embodiment, since the viewing angle of the detection device 21 is set to a range that does not fall into the aerial images Sa and Sb projected by the projection device 20, the optical path of the infrared light irradiated from the detection device 21 does not hinder the optical path for imaging the aerial image S as in the conventional system. Therefore, in the interface device 2 of the first embodiment, the reduction in the resolution of the aerial image S can be suppressed.

[0125] In addition, in the interface device 2 of the first embodiment, the viewing angle of the detection device 21 only needs to be set within a range that does not reflect the aerial images Sa and Sb projected by the projection device 20. Therefore, when configuring the detection device 21, it is not necessary to consider the positional relationship with other components constituting the imaging optical system as in the conventional system. Therefore, in the interface device 2 of the first embodiment, the detection device 21 can be configured at a position close to other components constituting the imaging optical system, and as a result, the interface device 2 as a whole can be miniaturized.

[0126] In addition, in the interface device 2, the projection device 20 images the aerial images Sa and Sb in the virtual space K in such a manner that the aerial images Sa and Sb include the viewing angle of the detection device 21. That is, the aerial images Sa and Sb are imaged at a position where the detection device 21 can suppress the decrease in the detection accuracy of the three-dimensional position of the user's hand (detection object). More specifically, for example, the aerial images Sa and Sb are imaged at least outside the viewing angle of the detection device 21. Thus, in the interface device 2, the aerial images Sa and Sb projected into the virtual space K do not hinder the detection of the three-dimensional position of the user's hand by the detection device 21. Therefore, in the interface device 2, the decrease in the detection accuracy of the three-dimensional position of the user's hand caused by the aerial images Sa and Sb being reflected in the viewing angle of the detection device 21 can be suppressed.

[0127] In addition, in the above description, an example is described in which the detection device 21 is arranged inside the projection device 20 (inward of the beam splitter 202), but the detection device 21 only needs to set the viewing angle to a range that does not reflect the aerial images Sa and Sb projected by the projection device 20, and it may not necessarily be arranged inside the projection device 20. However, in this case, the size of the entire interface device 2 including the projection device 20 and the detection device 21 may be enlarged. Therefore, it is preferable that the detection device 21 is arranged inside the projection device 20 as described above and the viewing angle is set to a range that does not reflect the aerial images Sa and Sb projected by the projection device 20.

[0128] In addition, in the above description, the imaging optical system provided by the projection device 20 is configured to include the beam splitter 202 and the retroreflector 203, and the detection device 21 is arranged at a position closer to the inside than the beam splitter 202 constituting the imaging optical system. However, the imaging optical system may also be a structure other than the above. In this case, the detection device 21 only needs to be arranged at a position closer to the inside than the light bending surface included in the imaging optical system. The inner side of the light bending surface refers to the side of the light bending surface and the side where the light source is arranged relative to the light bending surface.

[0129] For example, when the imaging optical system is configured to include a dihedral corner reflector array element, since the element surface of the dihedral corner reflector array element functions as the above-mentioned light bending surface, the detection device 21 only needs to be arranged at a position closer to the inside than the element surface of the dihedral corner reflector array element.

[0130] In addition, for example, when the imaging optical system is configured to include a lens array element, since the element surface of the lens array element functions as the above-mentioned light bending surface, the detection device 21 only needs to be arranged at a position closer to the inside than the element surface of the lens array element.

[0131] In addition, in the above description, an example is shown in which the viewing angle of the detection unit 21 is set to a range that does not reflect the aerial images Sa and Sb showing the boundary positions of the operation space A and the operation space B in the virtual space K. However, in a case where the aerial images of the boundary positions of the operation spaces in the virtual space K are not shown and are projected into the virtual space K, it is not always necessary to prevent the aerial images from being reflected into the viewing angle of the detection unit 21.

[0132] For example, in the operation space B, an aerial image SC showing the lower limit position of the detectable range of the detection unit 21 may be projected by the projection unit 20 (see FIG. 1 ). Figure 3 ). In addition, the aerial image SC is projected near the center position of the X-axis direction in the operation space B and shows the above-mentioned lower limit position, and sometimes also becomes a reference for left and right indications when the user moves the hand in the operation space B in an action corresponding to the instruction of left and right indications such as left click and right click. Regarding such an aerial image SC, since it is not an aerial image showing the boundary position of each operation space in the virtual space K, it is not always necessary to prevent it from being reflected in the field of view of the detection device 21. That is, an aerial image other than the aerial image showing the boundary position of each operation space in the virtual space K can be projected into the field of view of the detection device 21.

[0133] In addition, in the interface device 2, as described above, one or more aerial images are projected by the projection device 20. In this case, the one or more aerial images may show the outer frame or outer surface of the virtual space K to the user.

[0134] For example, in the interface device 2, an aerial image showing the boundary position of each operation space in the virtual space K and an aerial image not showing the boundary position can be projected by the projection device 20. The former aerial image, i.e., the aerial image showing the boundary position of each operation space in the virtual space K, can be an aerial image showing the boundary position of each operation space in the virtual space K and the outer frame or outer surface of the virtual space K by setting its projection position to a position along the outer edge of the virtual space K, for example. In this case, the user can easily grasp not only the boundary position of each operation space in the virtual space K but also the outer edge of the virtual space K by visually confirming the aerial image.

[0135] As described above, according to the first embodiment, the interface device 2 includes: the detection unit 21 that detects the three-dimensional position of the detection object in the virtual space K; and the projection unit 20 that projects the air image S into the virtual space K, the virtual space K being divided into a plurality of operation spaces, the plurality of operation spaces defining operations that can be performed by the user when the three-dimensional position of the detection object detected by the detection unit 21 is included, and the boundary position of each operation space in the virtual space K is shown by the air image S projected by the projection unit 20. Thus, in the interface device 2 of the first embodiment, the boundary positions of the plurality of operation spaces constituting the virtual space that is the operation target of the user can be visually confirmed.

[0136] The projection unit 20 images the aerial images Sa and Sb in the virtual space K so that the aerial images Sa and Sb include the view angle of the detection unit 21. Thus, in the interface device 2 of the first embodiment, the detection unit 21 can suppress a decrease in the detection accuracy of the three-dimensional position of the detection object.

[0137] In addition, the projection unit 20 includes an imaging optical system having a light bending surface, the light bending surface constituting a plane that bends the optical path of light emitted from the light source, and the imaging optical system forms an image as an aerial image Sa, Sb on the opposite side of the light bending surface, of a real image generated by a light source arranged on one side of the light bending surface. Thus, in the interface device 2 of the first embodiment, the aerial images Sa, Sb can be projected using the imaging optical system.

[0138] In addition, the imaging optical system is configured to include: a beam splitter 202 having a light bending surface, which separates the light emitted from the light source 201 into transmitted light and reflected light; and a retroreflector 203, which, when incident with the reflected light from the beam splitter 202, reflects the reflected light in the incident direction. Thus, in the interface device 2 of the first embodiment, it is possible to project the aerial images Sa and Sb using the retroreflection of light.

[0139] In addition, the imaging optical system is configured to include a dihedral corner reflector array element having a light bending surface. Thus, in the interface device 2 of the first embodiment, it is possible to project aerial images Sa and Sb using specular reflection of light.

[0140] In addition, the detection unit 21 is arranged in the internal area of ​​the imaging optical system and is arranged on one side of the light bending surface of the imaging optical system. Therefore, in the interface device 2 of embodiment 1, the miniaturization of the device as a whole can be achieved. In addition, the decrease in the detection accuracy of the three-dimensional position of the detection object caused by external factors such as dust, dirt and water can be suppressed.

[0141] In addition, the aerial images Sa and Sb projected into the virtual space K are formed at positions where the detection unit 21 can suppress the decrease in the detection accuracy of the three-dimensional position of the detection object. Therefore, in the interface device 2 of the first embodiment, the detection unit 21 can suppress the decrease in the detection accuracy of the three-dimensional position of the detection object.

[0142] The viewing angle of the detection unit 21 is set to a range that does not reflect the aerial images Sa and Sb projected by the projection unit 20. Thus, in the interface device 2 of the first embodiment, a decrease in the resolution of the aerial images Sa and Sb can be suppressed.

[0143] In addition, one or more aerial images are projected in the virtual space K, and the one or more aerial images show the user the outer frame or outer surface of the virtual space K. Thus, in the interface device 2 of the first embodiment, the user can easily grasp the outer edge of the virtual space K.

[0144] Furthermore, at least one of the plurality of projected aerial images is projected within the viewing angle of the detection unit 21. Thus, in the interface device 2 of the first embodiment, for example, the degree of freedom of the projection position of the aerial image indicating the lower limit position of the detectable range of the detection unit 21 can be increased.

[0145] Implementation method 2.

[0146] In Embodiment 1, an interface device 2 is described which can suppress a decrease in the resolution of aerial images Sa and Sb and can reduce the size of the entire device. In Embodiment 2, an interface device 2 is described which can suppress a decrease in the resolution of aerial images Sa and Sb and can further reduce the size of the entire device.

[0147] Figure 6 2 is a perspective view showing an example of the configuration structure of the projection device 20 and the detection device 21 in the interface device 2 of the second embodiment. Figure 7 It is a plan view showing an example of the arrangement structure of the projection device 20 and the detection device 21 in the interface device 2 according to the second embodiment.

[0148] In the interface device 2 of the second embodiment, with respect to Figure 4 and Figure 5In the interface device 2 of the illustrated embodiment 1, the beam splitter 202 is divided into two beam splitters 202a and 202b, and the retroreflective element 203 is divided into two retroreflective elements 203a and 203b.

[0149] In addition, the first imaging optical system including the beam splitter 202a and the retro-reflector 203a is used to form a virtual space K( Figure 6 The aerial image Sa is projected in the virtual space K by using the second imaging optical system including the beam splitter 202b and the retroreflector 203b. That is, the two split beam splitters are in a corresponding relationship with the two retroreflectors, the beam splitter 202a corresponds to the retroreflector 203a, and the beam splitter 202b corresponds to the retroreflector 203b.

[0150] In addition, the projection (imaging) principle of the aerial image based on the first imaging optical system and the second imaging optical system is the same as that of Embodiment 1. For example, the retroreflector 203a reflects the reflected light from the corresponding beam splitter 202a along the incident direction, and the retroreflector 203b reflects the reflected light from the corresponding beam splitter 202b along the incident direction.

[0151] In addition, in the interface device 2 of embodiment 2, similarly to the interface device 2 of embodiment 1, the detection device 21 is also arranged inside the projection device 20. More specifically, the detection device 21 is arranged inside the first imaging optical system and the second imaging optical system of the projection device 20, and in particular, is arranged in the area sandwiched between the light source 201 and the two beam splitters 202a and 202b.

[0152] In this case, similarly to the first embodiment, the viewing angle of the detection device 21 is set to a range that does not reflect the aerial images Sa and Sb projected by the projection device 20, and in particular, is set to fall within the internal area U defined by the two aerial images Sa and Sb.

[0153] Thus, in the interface device 2 of the second embodiment, by using two imaging optical systems including the split beam splitters 202a and 202b and the retroreflectors 203a and 203b, respectively, it is possible to project the aerial images Sa and Sb that can be visually recognized by the user into the virtual space K, and the overall size of the interface device 2 can be further reduced compared to the first embodiment. In addition, in this case, by arranging the detection device 21 inside the two imaging optical systems, the overall size of the interface device 2 can be further reduced.

[0154] In addition, in the interface device 2 of embodiment 2, the viewing angle of the detection device 21 is also set to a range that does not reflect the aerial images Sa and Sb projected by the projection device 20. Therefore, as with the interface device 2 of embodiment 1, the reduction in resolution of the aerial images Sa and Sb can be suppressed.

[0155] In addition, in the above description, an example is described in which the light source 201 is set to one and the beam splitter 202 and the retro-reflective member 203 are respectively divided into two, but the interface device 2 is not limited to this, and the light source 201 may be increased to two, and different light sources may be used in the first imaging optical system and the second imaging optical system. In addition, the number of additional light sources 201 and the number of divisions of the beam splitter 202 and the retro-reflective member 203 are not limited to the above case, and may be set to n (n is an integer greater than 2).

[0156] In the above description, an example is described in which the imaging optical system is configured to include a beam splitter and a retroreflector. However, the imaging optical system is not limited to this. For example, as described in Embodiment 1, the imaging optical system may also be configured to include a dihedral corner reflector array element. In this case, in the interface device 2, as long as Figure 6 The retro-reflective elements 203a and 203b may be omitted and dihedral corner reflector array elements may be arranged at the positions where the beam splitters 202a and 202b are arranged, respectively.

[0157] In addition, in the above description, an example in which the beam splitter 202 and the retro-reflector 203 are respectively divided into two in one imaging optical system is described, but the interface device 2 is not limited to this, and for example, it can also include more than one imaging optical system and more than two light sources 201. In this case, the number of imaging optical systems and the number of light sources 201 may not necessarily be the same, and each imaging optical system and each light source may not necessarily correspond to each other. In addition, in this case, each of the more than two light sources 201 can form an image of a real image as an aerial image using more than one imaging optical system.

[0158] For example, in the case where an imaging optical system is provided and two light sources 201 (first light source and second light source) are provided, the first light source may form an image of a real image as an aerial image using the imaging optical system, and the second light source may form an image of a real image as an aerial image using the imaging optical system. Figure 4 and Figure 5 The structure shown.

[0159] In addition, for example, when three imaging optical systems (the first imaging optical system to the third imaging optical system) and four light sources 201 (the first light source to the fourth light source) are provided, the first light source may image the real image as an aerial image using only any one imaging optical system (for example, the first imaging optical system), or may image the real image as an aerial image using any two imaging optical systems (for example, the first imaging optical system and the second imaging optical system), or may image the real image as an aerial image using all the imaging optical systems (the first imaging optical system to the third imaging optical system).

[0160] Similarly, the second light source may form an image of the real image as an aerial image S using only one imaging optical system (e.g., the second imaging optical system), or may form an image of the real image as an aerial image S using any two imaging optical systems (e.g., the second imaging optical system and the third imaging optical system), or may form an image of the real image as an aerial image S using all imaging optical systems (the first imaging optical system to the third imaging optical system). The same is true for the third light source and the fourth light source. Thus, in the interface device 2, it is easy to adjust the brightness of the aerial image S and the imaging position of the aerial image S.

[0161] As described above, according to Embodiment 2, the beam splitter 202 and the retroreflector 203 are respectively divided into n (n is an integer greater than 2), the n beam splitters correspond to the n retroreflectors one-to-one, and the n retroreflectors respectively reflect the reflected light from the corresponding beam splitters along the incident direction. Thus, in addition to the effects of Embodiment 1, the interface device 2 of Embodiment 2 can further reduce the size of the entire interface device 2 compared with Embodiment 1.

[0162] In addition, the interface device 2 has two or more light sources 201 and one or more imaging optical systems, and each light source uses one or more imaging optical systems to form an image of a real image as an aerial image. Therefore, in addition to the effects of the first embodiment, the interface device 2 of the second embodiment can also easily adjust the brightness and imaging position of the aerial image.

[0163] Implementation method 3.

[0164] In the first embodiment, the interface device 2 is described which can suppress the reduction in resolution of the aerial images Sa and Sb and can reduce the size of the entire device. In the third embodiment, the interface device 2 is described which can extend the detection path from the detection device 21 to the detection object in addition to suppressing the reduction in resolution of the aerial images Sa and Sb and reducing the size of the entire device.

[0165] Figure 8 FIG. 2 is a side view showing an example of the configuration of the projection device 20 and the detection device 21 in the interface device 2 according to the third embodiment. Figure 4 and Figure 5 The interface device 2 of the embodiment 1 shown changes the configuration of the detection device 21 to a position near the light sources 201a and 201b. More specifically, the configuration of the detection device 21 is changed to a position sandwiched between the light sources 201a and 201b when viewed from above, and is changed to a position slightly in front of the light sources 201a and 201b (close to the beam splitter 202) when viewed from the side. In addition, Figure 8 The diagram shows the interface device 2 according to the third embodiment as viewed from the light source 201b and the aerial image Sb side.

[0166] In addition, at this time, the viewing angle of the detection device 21 is set to be oriented in a direction substantially the same as the emission direction of the light emitted from the light sources 201a and 201b in the imaging optical system. In addition, at this time, similarly to the first embodiment, the viewing angle of the detection device 21 is set to a range that does not reflect the aerial images Sa and Sb projected by the projection device 20.

[0167] In this way, by configuring the detection device 21 near the light sources 201a and 201b, and setting the viewing angle of the detection device 21 to a direction substantially the same as the emission direction of the light emitted from the light sources 201a and 201b, the infrared light emitted when the detection device 21 detects the three-dimensional position of the user's hand passes through the beam splitter 202 after reflection based on the beam splitter 202 and retro-reflection based on the retro-reflector 203, and moves along a path to reach the user's hand at the transmission destination.

[0168] That is, the infrared light emitted from the detection device 21 travels along a path substantially the same as the light emitted from the light sources 201a and 201b when the imaging optical system forms images of the aerial images Sa and Sb. Thus, in the interface device 2 of the third embodiment, it is possible to suppress the reduction in resolution of the aerial image S and to miniaturize the overall size of the device, and, compared with the interface device 2 of the first embodiment in which the paths of the light from both sides are different, it is possible to extend the distance (detection distance) from the detection device 21 to the hand of the user as the detection object.

[0169] In particular, when the detection device 21 is composed of a camera device that can detect the three-dimensional position of the user's hand, the camera device is set with a minimum distance (shortest detectable distance) that must be separated from the detection object in order to perform appropriate detection. In addition, in order to perform appropriate detection, the detection device 21 needs to ensure the shortest detectable distance. On the other hand, there is also a demand for miniaturization of the overall size of the interface device 2.

[0170] In this regard, in the interface device 2 of embodiment 3, by configuring the detection device 21 as described above, the overall size of the interface device 2 can be miniaturized, and the detection distance of the detection device 21 can be extended to ensure the shortest detectable distance, thereby suppressing the decrease in detection accuracy.

[0171] Thus, according to Embodiment 3, the detection unit 21 is arranged at a position and a viewing angle such that the detection path when detecting the three-dimensional position of the detection object is substantially the same as the optical path of light from the light sources 201a and 201b in the imaging optical system through the beam splitter 202 and the retroreflector 203 to reach the aerial images Sa and Sb. Thus, in the interface device 2 of Embodiment 3, in addition to the effects of Embodiment 1, the overall size of the interface device 2 can be miniaturized, and the shortest detectable distance in the detection device 21 can be ensured, and the decrease in detection accuracy can be suppressed.

[0172] Implementation method 4.

[0173] In Embodiment 1, an example is described in which the detection device 21 is composed of a camera device that can detect the three-dimensional position of the user's hand by irradiating detection light (infrared light). In Embodiment 4, an example is described in which the detection device 21 is composed of a device that detects the position in the one-dimensional depth direction.

[0174] Fig. 9 FIG. 2 is a side view showing an example of the configuration of the projection device 20 and the detection device 21 in the interface device 2 according to the fourth embodiment. Figure 4 and Figure 5 In the interface device 2 of the illustrated first embodiment, the detection device 21 is changed into detection devices 21 a , 21 b , and 21 c , and the three detection devices 21 a , 21 b , and 21 c are arranged at the upper end of the beam splitter 202 .

[0175] The detection devices 21a, 21b, and 21c are composed of, for example, line sensors that detect the position of the user's hand in the one-dimensional depth direction by emitting detection light (infrared light) to the user's hand as the detection target. Fig. 9 The diagram shows the interface device 2 according to the fourth embodiment as viewed from the light source 201b and the aerial image Sb side.

[0176] In addition, at this time, the viewing angle of the detection device 21b is set to face the direction where the aerial images Sa and Sb are projected, and the surface (scanning surface) formed by the detection light (infrared light) is set to roughly overlap with the boundary surface where the aerial images Sa and Sb are projected. That is, the detection device 21b detects the position of the user's hand in the area near the boundary surface where the aerial images Sa and Sb are projected. However, similarly to the interface device 2 of the first embodiment, the viewing angle of the detection device 21b is set to a range where the aerial images Sa and Sb are not reflected.

[0177] In addition, the detection device 21a is arranged above the detection device 21b, and its viewing angle is set to face the direction in which the aerial images Sa and Sb are projected, and the surface (scanning surface) formed by the detection light is set to be substantially parallel to the above-mentioned boundary surface. That is, the detection device 21a sets the area inside the scanning surface in the space (operation space A) above the above-mentioned boundary surface as the detectable range, and detects the position of the user's hand in this area.

[0178] In addition, the detection device 21c is arranged at a position lower than the detection device 21b, and its viewing angle is set to face the direction in which the aerial images Sa and Sb are projected, and the surface (scanning surface) formed by the detection light is set to be substantially parallel to the above-mentioned boundary surface. That is, the detection device 21c sets the area inside the scanning surface in the space (operation space B) below the above-mentioned boundary surface as the detectable range, and detects the position of the user's hand in this area. In addition, similarly to the interface device 2 of the first embodiment, the viewing angles of the detection devices 21a and 21c are also set to a range where the aerial images Sa and Sb are not reflected.

[0179] Thus, in the interface device 2 of the fourth embodiment, the detection devices 21a, 21b, and 21c formed of line sensors are used as the detection device 21, and the viewing angles of the respective detection devices are set so that the planes (scanning planes) formed by the detection lights from the respective detection devices are parallel to each other and the planes are arranged in the space in the up-down direction (front-back direction) centered on the above-mentioned boundary plane. Thus, in the interface device 2 of the fourth embodiment, the three-dimensional position of the user's hand in the virtual space K can be detected using the line sensors.

[0180] In addition, since the line sensor is small and inexpensive compared to the camera device that can detect the three-dimensional position of the user's hand as described in Implementation Example 1, by using a line sensor as the detection device 21, the size of the entire device can be miniaturized compared to the interface device 2 of Implementation Example 1, and the cost can also be reduced.

[0181] In addition, in the above description, an example of using three detection devices composed of line sensors is described, but the number is not limited to this. However, as described above, in order to be able to detect the position of the user's hand in a space including a surface in the up-down direction (front-back direction) centered on the above boundary surface, it is preferred that at least three detection devices composed of line sensors are provided.

[0182] Thus, according to the fourth embodiment, the detection unit 21 is composed of three or more line sensors, and the three or more line sensors set at least the area inside the boundary surface, which is the surface on which the aerial images Sa and Sb are projected in the virtual space K, and the area inside the surface sandwiching the boundary surface in the virtual space K as the detectable range. Therefore, in the interface device 2 of the fourth embodiment, in addition to the effects of the first embodiment, the size of the entire device can be reduced compared with the interface device 2 of the first embodiment, and the cost can also be reduced.

[0183] Implementation method 5.

[0184] In the first to fourth embodiments, the configuration examples of the interface device 2 included in the interface system 100 are mainly described. In the fifth embodiment, an example of the functional blocks included in the interface system 100 is described. Fig.11 An example of a functional block diagram of the interface system 100 in the fifth embodiment is shown.

[0185] like Fig.11 As shown, the interface system 100 includes an aerial image projection unit 31, a position detection unit 32, a position acquisition unit 41, a boundary position recording unit 42, an operation space determination unit 43, a pointer operation information output unit 44, a pointer position control unit 45, an instruction determination unit 46, an instruction recording unit 47, an instruction output unit 48, an instruction generation unit 49 and an aerial image generation unit 50.

[0186] The aerial image projection unit 31 acquires data representing the aerial image S generated by the aerial image generation unit 50, and projects the aerial image S based on the acquired data to the virtual space K. The aerial image projection unit 31 is constituted by, for example, the above-mentioned projection device 20. In addition, the aerial image projection unit 31 may acquire data representing the above-mentioned aerial image SC generated by the aerial image generation unit 50, and project the aerial image SC based on the acquired data to the virtual space K.

[0187] The position detection unit 32 detects the three-dimensional position of the detection object (here, the user's hand) in the virtual space K. The position detection unit 32 is composed of, for example, the above-mentioned detection device 21. The position detection unit 32 outputs the detection result of the three-dimensional position of the detection object (hereinafter also referred to as "position detection result") to the position acquisition unit 41.

[0188] Alternatively, the position detection unit 32 may detect the three-dimensional position of the aerial image S projected into the virtual space K, and record data indicating the detected three-dimensional position of the aerial image S in the boundary position recording unit 42 .

[0189] In addition, when the aerial image projector 31 is configured by the projection device 20 and the position detector 32 is configured by the detection device 21 , the functions of the aerial image projector 31 and the position detector 32 are realized by the interface device 2 .

[0190] The position acquisition unit 41 acquires the position detection result output from the position detection unit 32. The position acquisition unit 41 outputs the acquired position detection result to the operation space determination unit 43.

[0191] The boundary position recording unit 42 records data indicating the boundary position between the operation space A and the operation space B constituting the virtual space K, that is, the three-dimensional position of the aerial image S. The boundary position recording unit 42 is composed of, for example, a HDD (Hard Disc Drive) or an SSD (Solid State Drive).

[0192] For example, in the aerial image S, Figure 3 In the case of a line (straight line)-shaped graphic structure as shown in FIG. 1 , the boundary position recording unit 42 records data indicating the three-dimensional position of at least one point among the points (pixels) of the aerial image S constituting the line. For example, the boundary position recording unit 42 may record data indicating the three-dimensional positions of any three points among the points constituting the aerial image S of the line, or may record data indicating the three-dimensional positions of all points among the points constituting the aerial image S of the line. In addition, since the aerial image S is projected on Figure 3 Therefore, the coordinate position of each point in the Z-axis direction recorded by the boundary position recording unit 42 is the same coordinate position on the boundary surface shown.

[0193] The operation space determination unit 43 obtains the position detection result output from the position acquisition unit 41. In addition, the operation space determination unit 43 determines the operation space where the user's hand exists based on the obtained position detection result and the boundary position of each operation space in the virtual space K. The operation space determination unit 43 outputs the result of the above determination (hereinafter also referred to as "space determination result") to the aerial image generation unit 50. In addition, the operation space determination unit 43 outputs the space determination result together with the position detection result obtained from the position acquisition unit 41 to the operation information output unit 51.

[0194] The operation information output unit 51 outputs operation information for executing a predetermined operation on the display device 1 using at least the space determination result by the operation space determination unit 43. The operation information output unit 51 includes the pointer operation information output unit 44, the command determination unit 46, and the command output unit 48.

[0195] The pointer operation information output unit 44 obtains the space determination result and the position detection result output from the operation space determination unit 43. When the space determination result obtained above indicates that the user's hand is present in the operation space A, the pointer operation information output unit 44 generates information (hereinafter also referred to as "movement control information") for moving the pointer P displayed on the operation screen R of the display 10 in correspondence with the movement of the user's hand in the operation space A. In addition, "the movement of the user's hand" includes information related to the movement, such as the amount of movement of the user's hand. For example, the pointer operation information output unit 44 calculates the amount of movement of the user's hand based on the position detection result output from the operation space determination unit 43. The amount of movement of the user's hand includes information related to the direction in which the user's hand moves and the distance the user's hand moves in that direction.

[0196] Then, based on the calculated movement amount, the pointer operation information output unit 44 generates information (movement control information) for moving the pointer P displayed on the operation screen R of the display 10 in accordance with the movement of the user's hand in the operation space A. The pointer operation information output unit 44 outputs the above-mentioned operation information including the generated movement control information to the pointer position control unit 45.

[0197] In addition, when the spatial determination result obtained above indicates that the user's hand is present in the operation space B, the pointer operation information output unit 44 generates information (hereinafter also referred to as "fixing control information") for fixing the pointer P displayed on the operation screen R of the display 10. The pointer operation information output unit 44 outputs the above operation information including the generated fixing control information to the pointer position control unit 45.

[0198] In addition, the pointer operation information output unit 44 may include the following information in the operation information and output it, the meaning of which is: the direction ( Figure 3 The distance in the Z-axis direction) can be changed to make the movement amount or movement speed of the pointer P displayed on the screen of the display device 1 variable.

[0199] The pointer position control unit 45 obtains the operation information output from the pointer operation information output unit 44. When the operation information obtained from the pointer operation information output unit 44 includes movement control information, the pointer position control unit 45 moves the pointer P on the operation screen R displayed on the display 10 in accordance with the movement of the user's hand based on the movement control information. For example, the pointer position control unit 45 moves the pointer P by an amount corresponding to the amount of the user's hand movement, in other words, moves the pointer P by a distance included in the direction included in the movement amount.

[0200] Furthermore, when the operation information acquired from the pointer operation information output unit 44 includes the fixing control information, the pointer position control unit 45 fixes the pointer P on the operation screen R displayed on the display 10 based on the fixing control information.

[0201] The command determination unit 46 obtains the space determination result and the position detection result output from the operation space determination unit 43. When the space determination result obtained above indicates that the user's hand is present in the operation space B, the command determination unit 46 determines the user's hand movement (gesture) based on the position detection result output from the operation space determination unit 43.

[0202] The command recording unit 47 records the command information in advance. The command information is information that establishes a correspondence between the user's hand movements (gestures) and the commands that the user can execute. The command recording unit 47 is composed of, for example, a HDD (Hard Disc Drive), an SSD (Solid State Drive), or the like.

[0203] The command determination unit 46 determines the command corresponding to the above-determined user's hand motion (gesture) based on the command information recorded in the command recording unit 47. The command determination unit 46 outputs the determined command to the command output unit 48 and the aerial image generation unit 50.

[0204] The command output unit 48 acquires the command output from the command determination unit 46. The command output unit 48 outputs the above-mentioned operation information including information indicating the acquired command to the command generation unit 49.

[0205] The command generation unit 49 receives the operation information output from the command output unit 48 and generates a command included in the received operation information. Thus, the command corresponding to the hand movement (gesture) of the user is executed in the interface system 100 .

[0206] The aerial image generation unit 50 generates data representing the aerial image S projected by the aerial image projection unit 31 onto the virtual space K. The aerial image generation unit 50 outputs the generated data representing the aerial image S to the aerial image projection unit 31 .

[0207] In addition, the aerial image generation unit 50 may obtain the space determination result output from the operation space determination unit 43, and regenerate data representing the aerial image S projected in a form corresponding to the obtained space determination result. In addition, the aerial image generation unit 50 may output the regenerated data representing the aerial image S to the aerial image projection unit 31.

[0208] For example, when the spatial determination result indicates that the user's hand is present in the operation space A, the aerial image generation unit 50 may generate data representing the aerial image S projected in blue again. Also, when the spatial determination result indicates that the user's hand is present in the operation space B, the aerial image generation unit 50 may generate data representing the aerial image S projected in red again. Also, when the spatial determination result indicates that the user's hand is present in the operation space B, the aerial image generation unit 50 may generate data representing the aerial image SC and output the generated data representing the aerial image SC to the aerial image projection unit 31.

[0209] The aerial image generation unit 50 may also obtain the command output from the command determination unit 46 and regenerate data representing the aerial image S projected in a form corresponding to the obtained command. The aerial image generation unit 50 may also output the regenerated data representing the aerial image S to the aerial image projection unit 31.

[0210] For example, when the instruction obtained from the instruction determination unit 46 is a left click, the aerial image generation unit 50 may again generate data representing the aerial image S that flashes once. Alternatively, when the instruction obtained from the instruction determination unit 46 is a left-double click, the aerial image generation unit 50 may again generate data representing the aerial image S that flashes twice in succession.

[0211] Furthermore, the operation information output unit 51 may include a sound information output unit (not shown) which generates information to output a sound corresponding to the fixation of the pointer P (a sound notifying the fixation of the pointer P) when the operation information including the fixation control information is output from the pointer operation information output unit 44 to the pointer position control unit 45, and outputs the generated information by including the generated information in the operation information. In this case, when the pointer position control unit 45 fixes the pointer P based on the fixation control information, the sound corresponding to the fixation of the pointer P is output. Therefore, the user can easily understand that the pointer P is fixed by hearing the sound.

[0212] Alternatively, the sound information output unit may generate information indicating the sound corresponding to the instruction determined by the instruction determination unit 46, and output the generated information by including it in the operation information. In this case, when the instruction generation unit 49 generates an instruction, the sound corresponding to the instruction is output. Therefore, the user can easily understand that the instruction has been generated by hearing the sound.

[0213] In addition, the sound information output unit may generate information to output a sound corresponding to the three-dimensional position of the user's hand in the operation space A or a sound corresponding to the movement of the user's hand in the operation space A, and output the generated information by including it in the operation information. For example, the sound information output unit may generate information to output a sound corresponding to the three-dimensional position based on the three-dimensional position of the user's hand in the operation space A detected by the position detection unit 32, and output the generated information by including it in the operation information. In this case, for example, when the user brings the hand close to the boundary surface in the operation space A, a sound is output whose volume increases as the user's hand approaches the boundary surface. By hearing the sound, the user can easily understand that the hand is close to the boundary surface.

[0214] In addition, for example, the sound information output unit generates information to output a sound corresponding to the amount of hand movement calculated by the pointer operation information output unit 44, and outputs the generated information by including it in the operation information. In this case, for example, the more the user moves his hand in the operation space A (the greater the amount of hand movement), the louder the sound will be output. By hearing the sound, the user can easily grasp that the hand has made a large movement. In this way, the user can easily grasp the three-dimensional position of the hand or the movement of the hand in the operation space A by hearing the sound.

[0215] In the fifth embodiment, the position acquisition unit 41, the boundary position recording unit 42, the operation space determination unit 43, the pointer operation information output unit 44, the pointer position control unit 45, the command determination unit 46, the command recording unit 47, the command output unit 48, the command generation unit 49, and the aerial image generation unit 50 are mounted on the display control device 11. In this case, the device control device 12 includes the position acquisition unit 41, the boundary position recording unit 42, the operation space determination unit 43, the pointer operation information output unit 44, the command determination unit 46, the command recording unit 47, the command output unit 48, and the aerial image generation unit 50. The device control device 12 controls the interface device 2.

[0216] In the above description, an example is described in which the boundary position recording unit 42 and the command recording unit 47 are mounted in the device control device 12 . However, the boundary position recording unit 42 and the command recording unit 47 are not limited thereto and may be provided outside the device control device 12 .

[0217] Next, refer to Figure 12 to Figure 15 The flowchart shown in the figure describes an operation example of the interface system 100 according to Embodiment 5. Here, for easy understanding, the operation example of the interface system 100 is divided into "A. aerial image projection phase" and "B. control execution phase" for description.

[0218] <A. Aerial image projection stage>

[0219] First, refer to Fig.12 The flowchart shown in FIG. 1 illustrates the aerial image projection phase. In the aerial image projection phase, the aerial image S is projected into the virtual space K. In addition, the aerial image projection phase is executed at least once when the interface system 100 is started.

[0220] First, the aerial image generation unit 50 generates data representing the aerial image S to be projected by the aerial image projection unit 31 into the virtual space K (step A001 ). The aerial image generation unit 50 outputs the generated data representing the aerial image S to the aerial image projection unit 31 .

[0221] Next, the aerial image projecting unit 31 acquires data representing the aerial image S generated by the aerial image generating unit 50 , and projects the aerial image S based on the acquired data into the virtual space K (step A002 ).

[0222] Next, the position detection unit 32 detects the three-dimensional position of the aerial image S projected into the virtual space K, and records data indicating the detected three-dimensional position of the aerial image S in the boundary position recording unit 42 (step A003 ).

[0223] In addition, in the above description, the following example is described: first, the aerial image projection unit 31 projects the aerial image S, and then the position detection unit 32 detects the three-dimensional position of the aerial image S, and records the data indicating the detected three-dimensional position of the aerial image S in the boundary position recording unit 42. However, step A003 is not a necessary process and can be omitted. For example, in the interface system 100, first, the user records the data indicating the three-dimensional position of the aerial image S in the boundary position recording unit 42 in advance, and the aerial image projection unit 31 projects the aerial image S to the three-dimensional position indicated by the data. In this case, step A003 can also be omitted.

[0224] <B. Control execution phase>

[0225] Next, refer to Fig.13 The flowchart shown in FIG. 1 illustrates the control execution phase. In the control execution phase, the interface device 2 is used by the user and performs control by the display control device 11 and the device control device 12. After the above-mentioned aerial image projection phase is completed, the control execution phase is repeatedly executed at predetermined intervals.

[0226] First, when the user places his hand in the virtual space K, the position detection unit 32 detects the three-dimensional position of the user's hand in the virtual space K (step B001 ). The position detection unit 32 outputs the detection result of the three-dimensional position of the user's hand (position detection result) to the position acquisition unit 41 .

[0227] Next, the position acquisition unit 41 acquires the position detection result output from the position detection unit 32 (step B002 ). The position acquisition unit 41 outputs the acquired position detection result to the operation space determination unit 43 .

[0228] Next, the operation space determination unit 43 acquires the detection result output from the position acquisition unit 41 , and determines the operation space where the user's hand exists based on the acquired position detection result and the boundary position of each operation space in the virtual space K.

[0229] For example, the operation space determination unit 43 Figure 3 The position coordinates of the five fingers of the user's hand in the Z-axis direction shown are compared with the position coordinates of the boundary position of the operation space A and the operation space B in the Z-axis direction. And if the former is equal to the latter or the former is located above the latter (in the +Z direction), the operation space determination unit 43 determines that the user's hand exists in the operation space A. On the other hand, if the former is located below the latter (in the -Z direction), the operation space determination unit 43 determines that the user's hand exists in the operation space B.

[0230] Next, the operation space determination unit 43 confirms whether it is determined that the user's hand exists in the operation space A (step B003). When it is determined that the user's hand exists in the operation space A (step B003; "Yes"), the operation space determination unit 43 outputs the result of the determination (space determination result) to the aerial image generation unit 50 (step B004). In addition, the operation space determination unit 43 outputs the space determination result together with the position detection result obtained from the position acquisition unit 41 to the pointer operation information output unit 44 (step B004). After that, the processing is transferred to step B005 (space processing A).

[0231] On the other hand, in step B003, when it is determined that the user's hand does not exist in the operation space A (step B003; "No"), the operation space determination unit 43 confirms whether it is determined that the user's hand exists in the operation space B (step B006). When it is determined that the user's hand exists in the operation space B (step B006; "Yes"), the operation space determination unit 43 outputs the result of the determination (space determination result) to the aerial image generation unit 50 (step B007). In addition, the operation space determination unit 43 outputs the space determination result together with the position detection result obtained from the position acquisition unit 41 to the pointer operation information output unit 44 and the instruction determination unit 46 (step B007). Thereafter, the processing is transferred to step B008 (space processing B).

[0232] On the other hand, in step B006 , when it is determined that the user's hand is not present in the operation space B (step B006 ; No), the interface system 100 ends the processing.

[0233] <Space Processing A>

[0234] Next, refer to Fig.14 The flowchart shown illustrates the spatial processing A of step B005.

[0235] First, the aerial image generation unit 50 obtains the spatial determination result indicating that the user's hand is present in the operation space A outputted from the operation space determination unit 43, and regenerates data representing the aerial image S projected in a form corresponding to the obtained spatial determination result (step C001). For example, the aerial image generation unit 50 regenerates data representing the aerial image S projected in blue as the aerial image S indicating that the user's hand is present in the operation space A. The aerial image generation unit 50 outputs the regenerated data representing the aerial image S to the aerial image projection unit 31.

[0236] Next, the aerial image projection unit 31 obtains the data representing the aerial image S regenerated by the aerial image generation unit 50, and re-projects the aerial image S based on the obtained data to the virtual space K (step C002). That is, the aerial image projection unit 31 updates the aerial image S projected to the virtual space K. As a result, for example, the color of the aerial image S changes to blue, and the user can easily understand that the hand has entered the operation space A (the pointer operation mode has been reached). In addition, steps C001 and C002 are not necessary processing and can be omitted.

[0237] Next, the pointer operation information output unit 44 determines whether the user's hand is moving based on the position detection result output from the operation space determination unit 43 (step C003). As a result, if it is determined that the user's hand is not moving (step C003; "No"), the process returns. On the other hand, if it is determined that the user's hand is moving (step C003; "Yes"), the process is transferred to step C004.

[0238] In step C004, the pointer operation information output unit 44 determines the user's hand movement based on the position detection result output from the operation space determination unit 43. Then, the pointer operation information output unit 44 generates information (movement control information) for moving the pointer P displayed on the operation screen R of the display 10 in accordance with the user's hand movement in the operation space A (step C004). In addition, the pointer operation information output unit 44 outputs the operation information including the generated movement control information to the pointer position control unit 45 (step C005).

[0239] Next, the pointer position control unit 45 controls the pointer P based on the movement control information included in the operation information output from the pointer operation information output unit 44 (step C006). Specifically, based on the movement control information, the pointer position control unit 45 moves the pointer P on the operation screen R displayed on the display 10 in correspondence with the movement of the user's hand. In more detail, the pointer position control unit 45 moves the pointer P on the operation screen R displayed on the display 10 by an amount corresponding to the amount of the user's hand movement, in other words, moves the pointer P in the direction included in the amount of movement by a distance included in the amount of movement. Thus, the pointer P moves in conjunction with the movement of the user's hand. Thereafter, the process is returned.

[0240] <Spatial Processing B>

[0241] Next, refer to Fig.15 The flowchart shown illustrates the spatial processing B of step B008.

[0242] First, the aerial image generation unit 50 obtains the spatial determination result indicating that the user's hand is present in the operation space B output from the operation space determination unit 43, and regenerates data representing the aerial image S projected in a form corresponding to the obtained spatial determination result (step D001). For example, the aerial image generation unit 50 regenerates data representing the aerial image S projected in red as the aerial image S indicating that the user's hand is present in the operation space B. The aerial image generation unit 50 outputs the regenerated data representing the aerial image S to the aerial image projection unit 31.

[0243] Next, the aerial image projection unit 31 obtains the data representing the aerial image S regenerated by the aerial image generation unit 50, and re-projects the aerial image S based on the obtained data to the virtual space K (step D002). That is, the aerial image projection unit 31 updates the aerial image S projected to the virtual space K. As a result, for example, the color of the aerial image S changes to red, and the user can easily understand that the hand has entered the operation space B (into the command execution mode). In addition, steps D001 and D002 are not necessary processing and can be omitted.

[0244] Next, the pointer operation information output unit 44 generates control information (fixing control information) for fixing the pointer P displayed on the operation screen R of the display 10 (step D003). In addition, the pointer operation information output unit 44 outputs operation information including the generated fixing control information to the pointer position control unit 45 (step D004).

[0245] Next, the pointer position control unit 45 fixes the pointer P on the operation screen R displayed on the display 10 based on the fixation control information included in the operation information output from the pointer operation information output unit 44 (step D005 ).

[0246] Next, the command determination unit 46 determines whether the user's hand is moving based on the position detection result output from the operation space determination unit 43 (step D006). As a result, if it is determined that the user's hand is not moving (step D006; "No"), the process returns. On the other hand, if it is determined that the user's hand is moving (step D006; "Yes"), the process is transferred to step D007.

[0247] In step D007 , command determination unit 46 determines the motion (gesture) of the user's hand based on the position detection result output from operation space determination unit 43 (step D007 ).

[0248] Next, the instruction determination unit 46 refers to the instruction information recorded in the instruction recording unit 47 to determine whether the action corresponding to the above-determined hand action exists in the instruction information (step D008). As a result, when it is determined that the action corresponding to the above-determined hand action does not exist in the instruction information (step D008; "No"), the process is returned. On the other hand, when it is determined that the action corresponding to the above-determined hand action exists in the instruction information (step D008; "Yes"), the instruction determination unit 46 determines the instruction that establishes a corresponding relationship with the action in the instruction information (step D009). The instruction determination unit 46 outputs the determined instruction to the instruction output unit 48.

[0249] Next, the command output unit 48 outputs the operation information including the information indicating the command acquired from the command determination unit 46 to the command generation unit 49 (step D010 ).

[0250] Next, the command generation unit 49 receives the operation information output from the command output unit 48 and generates a command included in the received operation information (step D011 ). Thus, the command corresponding to the user's hand movement (gesture) is executed in the interface system 100 .

[0251] In addition, although not shown in the above flowchart, in step D009, the command determination unit 46 may output the determined command to the aerial image generation unit 50. Furthermore, the aerial image generation unit 50 may obtain the command output from the command determination unit 46 and regenerate data representing the aerial image S projected in a form corresponding to the obtained command. In addition, the aerial image generation unit 50 may output the regenerated data representing the aerial image S to the aerial image projection unit 31.

[0252] In addition, the aerial image projection unit 31 may obtain data representing the aerial image S regenerated by the aerial image generation unit 50, and re-project the aerial image S based on the obtained data to the virtual space K. That is, the aerial image projection unit 31 may update the aerial image S projected to the virtual space K. Thus, for example, the aerial image S flashes once, and the user can easily understand that the left-click instruction has been executed.

[0253] Next, refer to Fig.16 24 , a control example of the interface system 100 according to the fifth embodiment will be described. The interface system 100 according to the fifth embodiment can perform the following control by operating as described above, for example.

[0254] (1) Pointer movement

[0255] When the user's hand is present in the operation space A, the pointer P moves on the operation screen R of the display 10 according to the movement amount of the user's hand in the virtual space K (XYZ coordinate system) (see Fig.16 ). In addition, Fig.16 In FIG. 1 , as a conceptual diagram, the pointer P is displayed on the operation space A, but actually, the pointer P displayed on the operation screen R of the display 10 moves.

[0256] In addition, in the above case, the pointer operation information output unit 44 can also generate the following movement control information: even if the amount of movement of the user's hand is the same, the movement amount or movement speed of the pointer P will change according to how far the three-dimensional position of the user's hand is away from the boundary surface (XY plane) of the virtual space shown by the aerial image S in the direction orthogonal to the boundary surface (i.e., the Z-axis direction).

[0257] For example, Fig.17 As shown, if the three-dimensional position of the user's hand is far away from the boundary surface (XY plane) in the Z-axis direction, the pointer operation information output unit 44 generates movement control information ( Fig.17 On the other hand, even if the amount of movement of the user's hand is the same as described above, the pointer operation information output unit 44 may generate movement control information to the effect that if the three-dimensional position of the user's hand is close to the boundary surface (XY plane) in the Z-axis direction, the pointer P is moved at a distance about half the distance moved by the user's hand or at a speed about half the speed moved by the user's hand ( Fig.17 Figure 1 shows the reference numeral W2).

[0258] That is, the pointer operation information output unit 44 can also generate movement control information by multiplying the movement amount or movement speed of the user's hand projected onto the boundary surface (XY plane) on which the aerial image S is projected by a coefficient corresponding to the distance in the Z-axis direction between the three-dimensional position of the user's hand and the boundary surface (XY plane).

[0259] In this case, if the user moves the hand at a position farther in the Z-axis direction from the boundary surface (XY plane) on which the aerial image S is projected, the pointer P can be moved by an amount equivalent to the amount of hand movement or at the same speed as the hand movement. On the other hand, if the user moves the hand at a position closer in the Z-axis direction to the boundary surface (XY plane) on which the aerial image S is projected, the pointer P can be moved slightly (slightly) or slowly. In particular, when switching from the pointer movement mode to the instruction execution mode, it is assumed that the user moves the hand near the boundary surface on which the aerial image S is projected. At this time, since the user can move the pointer P slightly or slowly, the position of the pointer P when executing the instruction can be precisely specified, and the convenience is improved.

[0260] In addition, the following example is described here: if the three-dimensional position of the user's hand is far away from the boundary surface (XY plane) in the Z-axis direction, the pointer operation information output unit 44 generates movement control information that causes the pointer P to move a distance that is the same as the distance moved by the user's hand or at a speed that is the same as the speed of the user's hand movement; if the three-dimensional position of the user's hand is close to the boundary surface (XY plane) in the Z-axis direction, movement control information that causes the pointer P to move a distance approximately half of the distance moved by the user's hand or at a speed approximately half of the speed of the user's hand movement is generated. However, contrary to the above, if the three-dimensional position of the user's hand is far away from the boundary surface (XY plane) in the Z-axis direction, the pointer operation information output unit 44 generates movement control information to move the pointer P at a distance approximately half of the distance moved by the user's hand or at a speed approximately half of the speed of the user's hand movement, and if the three-dimensional position of the user's hand is close to the boundary surface (XY plane) in the Z-axis direction, the pointer operation information output unit 44 generates movement control information to move the pointer P at a distance equal to the distance moved by the user's hand or at a speed equal to the speed of the user's hand movement.

[0261] (2) Fixed pointer

[0262] When the user's hand crosses the position (boundary position) of the aerial image from the operation space A and enters the operation space B, the pointer P is fixed on the operation screen R of the display 10 (see Fig.18 ). After that, even if the user's hand moves in the operation space B, the pointer P will remain fixed on the operation screen R of the display 10. In addition, at this time, the aerial image S can also be updated, for example, the color of the aerial image S can be changed from blue to red. In this way, the user can easily understand that the hand has entered the operation space B (changed to the instruction execution mode). In addition, at this time, the projection device 20 can also be used to project the aerial image SC to a position near the lower limit of the detectable range of the detection device 21 and near the approximate center of the virtual space K in the X-axis direction.

[0263] (3)Left click

[0264] For example, in the operation space B, when the user moves the hand in the -Y direction and the hand reaches a preset left click generation area, the hand movement (gesture) is determined by the command determination unit 46. The left click generation area is, for example, a predetermined area in the operation space B that is on the left side (-X direction side) of the aerial image SC and on the inner side (-Y direction side) when viewed from the user.

[0265] This action (gesture) establishes a correspondence relationship with the "left click" instruction in the instruction information. Therefore, the instruction determination unit 46 determines the "left click" instruction and executes the left click (refer to Fig.19 ). In addition, at this time, the aerial image generation unit 50 may, for example, regenerate data representing the aerial image S that flashes once, and the aerial image projection unit 31 projects the aerial image S based on the regenerated data. Thus, in the interface system 100, the aerial image S flashes once, and the user can easily understand that a left click has been performed. In addition, at this time, the interface system 100 may output a sound such as "click" as a sound corresponding to the left click. Thus, by hearing the sound, the user can more easily understand that a left click has been performed.

[0266] (4) Right-click

[0267] For example, in the operation space B, when the user moves the hand in the -Y direction and the hand reaches a preset right-click generation area, the hand movement (gesture) is determined by the command determination unit 46. The right-click generation area is, for example, a predetermined area in the operation space B that is to the right (+X direction side) of the aerial image SC and to the back (-Y direction side) when viewed from the user.

[0268] This action (gesture) establishes a correspondence relationship with the "right click" instruction in the instruction information. Therefore, the instruction determination unit 46 determines the "right click" instruction and executes the right click (refer to Fig. 20 ). In addition, at this time, the aerial image generation unit 50 may regenerate data representing the aerial image S that flashes once, for example, and the aerial image projection unit 31 projects the aerial image S based on the regenerated data. Thus, in the interface system 100, the aerial image S flashes once, and the user can easily understand that a right click has been performed.

[0269] (5) Double-click left

[0270] For example, in operation space B, when the user moves the hand in the -Y direction and the hand reaches the pre-set left-click generation area, when the user moves the hand continuously in the +Y direction and the -Y direction, the instruction determination unit 46 determines the action (gesture) of the hand. The action (gesture) establishes a correspondence relationship with the "left double-click" instruction in the instruction information. Therefore, the instruction determination unit 46 determines the "left double-click" instruction, and executes the left double-click (refer to Fig.21 ). In addition, at this time, the aerial image generation unit 50 may, for example, regenerate data representing the aerial image S that flashes twice in succession, and the aerial image projection unit 31 projects the aerial image S based on the regenerated data. Thus, in the interface system 100, the aerial image S flashes twice in succession, and the user can easily understand that a left double-click has been performed. In addition, at this time, the interface system 100 may output a continuous sound such as "click" and "click" as a sound corresponding to the left double-click. Thus, by hearing the sound, the user can more easily understand that a left double-click has been performed.

[0271] (6) Continuous movement of the pointer

[0272] When the user moves the hand in the +Y direction in the operation space A, the pointer P also moves in the +Y direction in conjunction with this movement (see Fig.22A Here, once the user moves his hand to the operation space B, the pointer P is fixed (refer to Fig. 22B ). In this state, when the user moves the hand in the -Y direction, the pointer P remains fixed (see Fig. 22C ).

[0273] Then, when the user moves his hand from the operation space B across the boundary position (boundary surface) to the operation space A, the pointer P moves again in conjunction with the movement of the user's hand (see Fig.22D ). By repeating the above operation, the user can move the pointer P continuously only by moving the hand within a limited space such as the operation space A and the operation space B.

[0274] In this regard, in the above-mentioned conventional devices, for example, Fig.23A As shown in FIG. 1 , in the case of continuous operations such as long-distance movement and scrolling of the pointer P, the user's hand moves a large amount, and a space wide enough to allow for such a large movement is required. In contrast, in Embodiment 5, for example, Fig. 23B As shown in the figure, by moving the user's hand back and forth at the boundary position (boundary surface), the correlation between the pointer P and the user's hand can be reset. Therefore, by repeatedly performing hand movements of a relatively short distance, the user can achieve continuous operations such as long-distance movement and scrolling of the pointer P even in limited spaces such as the operation space A and the operation space B.

[0275] (7) Scrolling operation

[0276] When the user starts to perform an action (gesture) such as rotating the hand in the operation space B without reaching the left click generation area or the right click generation area, the command determination unit 46 determines the action (gesture) of the hand. The action (gesture) is associated with the command of "scroll operation" in the command information. Therefore, in the interface system 100, the command determination unit 46 determines the command of "scroll operation" and executes the scroll operation (see Fig.24A ). In addition, at this time, the aerial image generating unit 50 may, for example, regenerate data representing an aerial image SE obtained by adding a predetermined graphic to the current aerial image S, and the aerial image projecting unit 31 projects the aerial image S and SE based on the regenerated data (see Fig. 24B). Thus, the aerial images S and SE to which predetermined graphics are added are projected, and the user can easily understand that the scroll operation can be performed.

[0277] Next, refer to Fig.25 The flowchart shown in the figure describes an application operation example in the control execution phase of the interface system 100 according to Embodiment 5. In this application operation example, an example in which the user operates both the operation space A and the operation space B using the left and right hands is described.

[0278] First, when the user places his hand in the virtual space K, the position detection unit 32 detects the three-dimensional position of the user's hand in the virtual space K (step E001 ). The position detection unit 32 outputs the detection result of the three-dimensional position of the user's hand (position detection result) to the position acquisition unit 41 .

[0279] Next, the position acquisition unit 41 acquires the position detection result output from the position detection unit 32 (step E002 ). The position acquisition unit 41 outputs the acquired position detection result to the operation space determination unit 43 .

[0280] Next, the operation space determination unit 43 acquires the detection result output from the position acquisition unit 41 , and determines the operation space where the user's hand exists based on the acquired position detection result and the boundary position of each operation space in the virtual space K.

[0281] Next, the operation space determination unit 43 checks whether the user's hand is determined to be present in both the operation space A and the operation space B (step E003). If it is determined that the user's hand is not present in both the operation space A and the operation space B (step E003; "No"), the process is transferred to the above Fig.13 Step B003 of the flowchart.

[0282] On the other hand, when it is determined that the user's hand is present in both the operation space A and the operation space B (step E003; "Yes"), the operation space determination unit 43 outputs the result of the determination (space determination result) to the aerial image generation unit 50. In addition, the operation space determination unit 43 outputs the space determination result together with the position detection result obtained from the position acquisition unit 41 to the pointer operation information output unit 44 and the instruction determination unit 46 (step E004). After that, the process is transferred to step E005 (space processing AB).

[0283] <Space Processing AB>

[0284] Next, refer to Fig.26 The flowchart shown illustrates the spatial processing AB of step E005.

[0285] First, the aerial image generation unit 50 obtains the spatial determination result outputted from the operation space determination unit 43 indicating that the user's hand exists in both the operation space A and the operation space B, and regenerates data representing the aerial image S projected in a form corresponding to the obtained spatial determination result (step F001). For example, as the aerial image S indicating that the user's hand exists in both the operation space A and the operation space B, the aerial image generation unit 50 regenerates data representing the aerial image S projected in green. The aerial image generation unit 50 outputs the regenerated data representing the aerial image S to the aerial image projection unit 31.

[0286] Next, the aerial image projection unit 31 obtains the data representing the aerial image S regenerated by the aerial image generation unit 50, and re-projects the aerial image S based on the obtained data to the virtual space K (step F002). That is, the aerial image projection unit 31 updates the aerial image S projected to the virtual space K. As a result, for example, the color of the aerial image S changes to green, and the user can easily understand that the hand has entered both the operation space A and the operation space B. In addition, steps F001 and F002 are not essential processes and can be omitted.

[0287] Next, the pointer operation information output unit 44 determines whether the user's hand is moving based on the position detection result output from the operation space determination unit 43 (step F003). As a result, if it is determined that the user's hand is not moving (step F003; "No"), the process returns. On the other hand, if it is determined that the user's hand is moving (step F003; "Yes"), the process is transferred to step F004.

[0288] In step F004, the instruction determination unit 46 determines the user's hand movement (gesture) based on the position detection result output from the operation space determination unit 43 (step F004). In this case, the user's hand movement (gesture) is a combination of the hand movement in the operation space A and the hand movement in the operation space B.

[0289] Next, the command determination unit 46 refers to the command information recorded in the command recording unit 47 to determine whether the action corresponding to the hand action determined above is present in the command information (step F005). As a result, if it is determined that the action corresponding to the hand action determined above is not present in the command information (step F005; "No"), the process returns.

[0290] On the other hand, when it is determined that the action corresponding to the hand action identified above is present in the instruction information (step F005; "Yes"), the instruction identification unit 46 identifies the instruction corresponding to the action in the instruction information (step F006). The instruction identification unit 46 outputs the identified instruction to the instruction output unit 48.

[0291] Next, the command output unit 48 outputs the above-mentioned operation information including the information indicating the command acquired from the command determination unit 46 to the command generation unit 49 (step F007 ).

[0292] Next, the command generation unit 49 receives the operation information output from the command output unit 48 and generates a command included in the received operation information (step F008 ). Thus, the command corresponding to the user's hand movement (gesture) is executed in the interface system 100 .

[0293] The interface system 100 according to the fifth embodiment can perform the following control by operating as described above, for example.

[0294] (8) Left drag operation

[0295] The user moves his left hand to the left-click generation area in the operation space B, and moves his right hand in the operation space A. In this way, in the interface system 100, the instruction determination unit 46 determines the movements (gestures) of the left and right hands. The movement (gesture) is associated with the instruction of "left drag operation" in the instruction information. Therefore, in the interface system 100, the instruction determination unit 46 determines the instruction of "left drag operation", and performs the left drag operation linked to the movement of the user's right hand (see Fig.27A ).

[0296] (9) Right drag operation

[0297] The user moves his right hand to the right-click generation area in the operation space B, and moves his left hand in the operation space A. In this way, in the interface system 100, the instruction determination unit 46 determines the movements (gestures) of the left and right hands. The movement (gesture) is associated with the instruction of "right drag operation" in the instruction information. Therefore, in the interface system 100, the instruction determination unit 46 determines the instruction of "right drag operation", and executes the right drag operation linked to the movement of the user's left hand (see Fig.27B ).

[0298] In addition, in the above description, an example is described in which the user performs a left drag operation and a right drag operation by moving the left and right hands, but these are only examples, and the instructions executed by the combination of the left and right hand movements of the user are not limited to the above examples. In this way, by establishing a correspondence between the combination of the left and right hand movements of the user and the instructions in advance, the variations of the instructions that the user can execute can be increased in the interface system 100.

[0299] In addition, in the above description, in order to facilitate the description, the action examples in the spatial processing AB and the action examples in the above spatial processing B are described separately, but these processes can also be performed continuously. For example, it is also possible that in the interface system 100, first, in the spatial processing B, the pointer position control unit 45 fixes the pointer P on the operation screen R based on the fixed control information generated by the pointer operation information output unit 44, and then performs the above spatial processing AB. That is, it is also possible that the user, for example, puts one of the left and right hands into the operation space B and fixes the pointer P on the operation screen R, and maintains this state and moves the left and right hands in the operation space A and the operation space B, thereby performing the above-mentioned left drag operation and right drag operation. In this case, the spatial processing B and the spatial processing AB are continuously performed in the interface system 100. As a result, in the interface system 100, it is possible to take into account both the accurate indication operation based on the user and the deformation and expansion of the instructions that the user can execute.

[0300] As described above, in the interface system 100 of the fifth embodiment, the aerial image S showing the boundary position between the operation space A and the operation space B constituting the virtual space K is projected onto the virtual space K. Thus, the user can visually confirm the boundary position between the operation space A and the operation space B in the virtual space K, and can easily understand at which position the operation space (mode) is switched.

[0301] In this regard, in the above-mentioned conventional devices, it is difficult for the user to visually confirm at which position in the virtual plane space the mode is switched, in other words, it is difficult to visually confirm the boundary positions of the spaces constituting the virtual plane space (the boundary position between the first space and the second space and the boundary position between the second space and the third space), and the user needs to grasp these positions while moving the hand to a certain extent. In addition, if the user does not move the hand to a certain extent, it is impossible to grasp the correlation between the pointer and the hand, and sometimes it takes time to start the operation.

[0302] On the other hand, in Embodiment 5, as described above, the user can visually check the boundary position between the operation space A and the operation space B in the virtual space K, and can easily understand which position is the boundary for switching the operation space (mode). In addition, the user does not need to move his hand to understand the boundary position for switching the operation space, and can start the operation more quickly than in the conventional device.

[0303] In addition, in the conventional non-contact pointing system represented by the conventional device, it is difficult for the user to know the position in the virtual space corresponding to the pressing of the button in the operation screen displayed on the display, so sometimes it is necessary to add an auxiliary display on the operation screen. Or, in order to reliably press the button on the operation screen according to the touch operation in the virtual space, sometimes it is necessary to make changes such as increasing the size of the button on the operation screen. That is, in the conventional non-contact pointing system, sometimes it is necessary to reorganize the existing software for displaying the operation screen.

[0304] Furthermore, in conventional non-contact pointing systems, even if the user holds the hand still in the air and performs an action (gesture) such as pressing, it is sometimes difficult to specify an accurate position on the operation screen due to the position of the pointer shifting when pressing. Furthermore, in conventional non-contact pointing systems, in operations involving continuity such as long-distance movement and scrolling of the pointer, the user's hand moves a large amount, and a wide space is sometimes required.

[0305] In this regard, in the fifth embodiment, as described above, the virtual space K is divided into the operation space A and the operation space B, and in the operation space A, the pointer P can be moved in conjunction with the user's hand movement, while in the operation space B, the pointer P is fixed, and the user's hand movement (gesture) that generates the instruction is recognized in the state where the pointer P is fixed. Thus, in the fifth embodiment, the position of the pointer P is prevented from being shifted during the execution of the hand movement (gesture) that generates the instruction. Therefore, the user can not only perform accurate instruction operations when executing instructions, but also directly operate an operation screen with small buttons created for PC mouse operations, and there is no need to reorganize the software for displaying the operation screen.

[0306] In addition, in embodiment 5, since the user can operate the display device represented by the operation of the pointer P in a non-contact manner, the user can operate in a non-contact manner even in a working environment where hygiene is important, such as when the user's hands are dirty or the user does not want to dirty his or her hands.

[0307] In addition, in Embodiment 5, since the user can execute instructions by hand movements regardless of the shape of the fingers, there is no need to remember specific finger gestures. In addition, in Embodiment 5, since the detection object of the detection device 21 is not limited to the user's hand, if the detection object is set to an object other than the user's hand, for example, even if the user holds an object in his hand, the operation can be performed.

[0308] In addition, regarding the means of an interface described in the present disclosure for providing the user with a feeling of mouse operation (close to the feeling of mouse operation) by using an aerial image as a guide, as long as the area for operation can be shown to the user by utilizing the guidance of the aerial image, it does not depend on the imaging optical system constructed by combining the beam splitter 202 and the retroreflective reflector 203. Other structures may also be used as the imaging optical system for imaging the aerial image.

[0309] Next, referring to FIG. 28 , an example of the hardware configuration of the device control device 12 included in the interface system 100 of Embodiment 5 will be described. The functions of the position acquisition unit 41, the operation space determination unit 43, the pointer operation information output unit 44, the instruction determination unit 46, the instruction output unit 48, and the aerial image generation unit 50 in the device control device 12 are implemented by a processing circuit. The processing circuit may be as follows: Fig.28A As shown in the dedicated hardware, it can also be as Fig.28B As shown, a CPU (Central Processing Unit, also called central processing device, processing device, computing device, microprocessor, microcomputer, processor or DSP (Digital Signal Processor)) 62 executes a program stored in a memory 63 .

[0310] When the processing circuit is dedicated hardware, the processing circuit 61 is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of each part of the position acquisition unit 41, the operation space determination unit 43, the pointer operation information output unit 44, the instruction determination unit 46, the instruction output unit 48, and the aerial image generation unit 50 can be implemented by the processing circuit 61 separately, or the functions of each part can be implemented by the processing circuit 61 in a centralized manner.

[0311] When the processing circuit is a CPU 62, the functions of the position acquisition unit 41, the operation space determination unit 43, the pointer operation information output unit 44, the instruction determination unit 46, the instruction output unit 48, and the aerial image generation unit 50 are implemented by software, firmware, or a combination of software and firmware. Software and firmware are expressed as programs and stored in the memory 63. The processing circuit reads and executes the program stored in the memory 63 to implement the functions of each part. That is, the device control device 12 has a memory for storing programs, and the program, when executed by the processing circuit, will eventually execute, for example, Figure 12 to Figure 15 and Figure 25-26 In addition, these programs can also be said to be steps and methods for causing the computer to execute the position acquisition unit 41, the operation space determination unit 43, the pointer operation information output unit 44, the instruction determination unit 46, the instruction output unit 48 and the aerial image generation unit 50. Here, as the memory 63, for example, it corresponds to a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically EPROM), magnetic disk, floppy disk, optical disk, compact disk, mini optical disk or DVD (Digital Versatile Disc), etc.

[0312] In addition, regarding each function of the position acquisition unit 41, the operation space determination unit 43, the pointer operation information output unit 44, the command determination unit 46, the command output unit 48, and the aerial image generation unit 50, part of it can be realized by dedicated hardware, and part of it can be realized by software or firmware. For example, regarding the position acquisition unit 41, its function can be realized by using a processing circuit as dedicated hardware, and regarding the operation space determination unit 43, the pointer operation information output unit 44, the command determination unit 46, the command output unit 48, and the aerial image generation unit 50, its function can be realized by causing the processing circuit to read and execute a program stored in the memory 63.

[0313] In this way, the processing circuit can implement the above functions through hardware, software, firmware or a combination thereof.

[0314] Furthermore, in the above description, an example is described in which the operation information output unit 51 uses at least the spatial determination result based on the operation space determination unit 43 to output operation information for executing a prescribed operation on the display device 1. However, the operation information output unit 51 is not limited to this, and may be configured to output operation information for executing a prescribed operation on an application displayed on the display device 1 using at least the spatial determination result based on the operation space determination unit 43. Here, the "application" includes an OS (Operating System) or various software that operates on the OS.

[0315] In addition, as operations on the application, in addition to the above-mentioned mouse operations, various operations performed with fingertips in a touch panel manner may also be included. In this case, each operation space may correspond to at least one of the various operations performed on the application using a mouse or a touch panel. Furthermore, continuous and different operations performed on the application may be associated with adjacent operation spaces in each operation space.

[0316] In addition, similar to the above-mentioned “continuous operation”, continuous and different operations performed on an application are operations that are assumed to be performed continuously in time, such as the user moving a pointer P on a displayed application and then executing a prescribed instruction.

[0317] In addition, the operation with continuity may be associated with all adjacent operation spaces in each operation space, or the operation with continuity may be associated with a portion of adjacent operation spaces. In other words, the operation without continuity may be associated with other adjacent operation spaces.

[0318] As described above, according to Embodiment 5, the interface system 100 comprises: a detection unit 21, which detects the three-dimensional position of a detection object in a virtual space K divided into a plurality of operation spaces; a position acquisition unit 41, which acquires the three-dimensional position of the detection object detected by the detection unit 21; a projection unit 20, which projects an aerial image S showing the boundary positions of each operation space in the virtual space K; an operation space determination unit 43, which determines the operation space containing the three-dimensional position of the detection object based on the three-dimensional position of the detection object acquired by the position acquisition unit 41 and the boundary positions of each operation space in the virtual space K; and an operation information output unit 51, which outputs operation information for executing a prescribed operation on an application displayed on the display device 1 using at least a determination result based on the operation space determination unit 43, wherein each operation space corresponds to at least any one of a plurality of operations performed on the application using a mouse or a touch panel, and a correspondence is established between continuous and different operations performed on the application and adjacent operation spaces in each operation space. Thus, in the interface system 100 according to the fifth embodiment, the boundary positions of the plurality of operation spaces constituting the virtual space K that is the operation target of the user can be visually confirmed.

[0319] Implementation method 6.

[0320] In the sixth embodiment, as another configuration example of the interface device 2 , an interface device 2 capable of controlling the spatial positional relationship of the aerial image with respect to the projection device 20 will be described.

[0321] Fig.29 2 is a perspective view showing an example of the configuration structure of the projection device 20 and the detection device 21 in the interface device 2 according to the sixth embodiment. Fig.30 1 is a top view showing an example of the configuration structure of the projection device 20 and the detection device 21 in the interface device 2 according to the sixth embodiment. Fig.31 This is a front view showing an example of the arrangement structure of the projection device 20 and the detection device 21 in the interface device 2 according to the sixth embodiment.

[0322] In the interface device 2 of the sixth embodiment, Figure 6 Similarly, in the interface device 2 of the embodiment 2 shown in FIG. 1 , the beam splitter 202 is divided into two beam splitters 202a and 202b, and the retroreflector 203 is divided into two retroreflectors 203a and 203b. Figure 6 The interface device 2 of the second embodiment shown is different in that the light source 201 is also divided into two light sources 201a and 201b.

[0323] In addition, the aerial image Sa is projected into the virtual space K ( Fig.29 The space in front of the paper surface) is projected into the virtual space K by using the second imaging optical system including the light source 201b, the beam splitter 202b and the retroreflector 203b. That is, the two divided light sources, the two beam splitters and the two retroreflectors are in a corresponding relationship, the light source 201a, the beam splitter 202a and the retroreflector 203a correspond to each other, and the light source 201b, the beam splitter 202b and the retroreflector 203b correspond to each other.

[0324] In addition, the projection (imaging) principle of the aerial image based on the first imaging optical system and the second imaging optical system is the same as that of embodiment 2. For example, the light (diffused light) emitted from the light source 201a is mirror-reflected on the surface of the beam splitter 202a, and the reflected light is incident on the retroreflective reflector 203a. The retroreflective reflector 203a retroreflects the incident light so that it is incident on the beam splitter 202a again. The light incident on the beam splitter 202a will pass through the beam splitter 202a and reach the user. And, by advancing along the above-mentioned optical path, the light emitted from the light source 201a will re-converge and re-diffuse at a position symmetrical to the light source 201a with the beam splitter 202a as the boundary. As a result, the user can perceive the aerial image Sa in the virtual space K.

[0325] Similarly, the light (diffused light) emitted from the light source 201b is specularly reflected on the surface of the beam splitter 202b, and the reflected light is incident on the retroreflector 203b. The retroreflector 203b retroreflects the incident light and makes it incident on the beam splitter 202b again. The light incident on the beam splitter 202b will pass through the beam splitter 202b and reach the user. And, by moving along the above-mentioned optical path, the light emitted from the light source 201b will re-converge and re-diffuse at a position symmetrical to the light source 201b with the beam splitter 202b as the boundary. As a result, the user can perceive the aerial image Sb in the virtual space K.

[0326] In the interface device 2 of the sixth embodiment, similarly to the interface device 2 of the second embodiment, the detection device 21 may be disposed inside the projection device 20 or outside the projection device 20. Fig.29 and Fig.30 An example is shown in which the detection device 21 is arranged inside the first imaging optical system and the second imaging optical system of the projection device 20, and in particular, an example is shown in which the detection device 21 is arranged in the area sandwiched between two light sources 201a, 201b and two beam splitters 202a, 202b.

[0327] In addition, at this time, similarly to the second embodiment, the viewing angle of the detection device 21 is set to a range that does not reflect the aerial images Sa and Sb projected by the projection device 20, and in particular, is set to make the viewing angle fall into the internal area U defined by the two aerial images Sa and Sb.

[0328] In addition, the light source 201a and the light source 201b are not arranged in parallel in space, and the aerial images Sa and Sb formed by the first imaging optical system and the second imaging optical system are formed in a parallel relationship in space.

[0329] More specifically, light source 201a and light source 201b are configured so that the spatial axes formed by the light sources are not parallel. For example, in the case of a rod-shaped light source, the spatial axis formed by the light source is an axis that passes through the center of the two end faces of the light source along the extension direction of the light source.

[0330] In addition, here, an example in which each light source is configured as a rod (bar) shape is described, but when each light source is not in a rod (bar) shape but is configured in a shape having a radiation surface for radiating light, each light source is arranged so that the planes (radiation surfaces) in the space formed by each light source are not parallel. In addition, in this case, the aerial images Sa and Sb are imaged in a parallel relationship with each other on the boundary surface which is an arbitrary surface on the virtual space K.

[0331] The reason why the light sources 201a, 201b and the aerial images Sa, Sb can be arranged in such a configuration is based on the following reason. That is, in the interface device 2, since the aerial images Sa, Sb are imaged at positions symmetrical to the light sources 201a, 201b with the beam splitters 202a, 202b as the spatial symmetry axes, while the imaging optical systems are separated, each imaging optical system images the light from the different light sources as an aerial image, thereby, although the optical components (light sources 201a, 201b) are arranged non-parallel, the aerial images Sa and Sb can be made parallel and imaged at a position closer to the user.

[0332] also, Fig.32 FIG. 2 is a diagram for supplementing the above-mentioned arrangement relationship between the light sources 201a, 201b and the aerial images Sa, Sb. Fig.32 In FIG. 1 , for convenience, the glass cover 204 is shown near the beam splitters 202a and 202b, but the glass cover 204 is omitted in other figures. Fig.32 The glass cover 204 is indicated by a dotted line.

[0333] Furthermore, in the interface device 2 of embodiment 6, by changing the configuration relationship and angle between the light source 201a and the beam splitter 202a and between the light source 201b and the beam splitter 202b, the spatial position relationship of the aerial images Sa and Sb relative to the projection device 20 can be controlled, and a boundary surface that allows the user to easily perform spatial operations can be formed.

[0334] For example, Fig.31 As shown, by arranging the two light sources 201a and 201b to form an eight-shaped shape when viewed from the front, the aerial images Sa and Sb are formed at an angle such that they appear in front of the user from the upper end side to the lower end side (see also Fig.29 ).

[0335] In addition, the two light sources 201a and 201b are configured to be able to change their postures when arranged, and the aerial images Sa and Sb are imaged in a manner such that the lower end side emerges at a position closer to the front side relative to the upper end side by increasing the opening between the two light sources when viewed from the front (making the two light sources close to the horizontal). That is, by increasing the opening between the two light sources when viewed from the front (making the two light sources close to the horizontal), the postures of the aerial images Sa and Sb are changed, and the angle formed by the boundary surface on which the aerial images Sa and Sb are projected relative to the horizontal plane is changed.

[0336] In addition, in the interface device 2, the configuration relationship and angle between the light source 201a and the beam splitter 202a and between the light source 201b and the beam splitter 202b can be changed manually or automatically by control. In addition, in this case, in the interface device 2, the configuration relationship and angle can be changed by moving the light sources 201a and 201b, the configuration relationship and angle can be changed by moving the beam splitters 202a and 202b, and the configuration relationship and angle can be changed by moving both the light sources 201a and 201b and the beam splitters 202a and 202b.

[0337] For example, the user manually adjusts the above-mentioned configuration relationship and angle, and controls the spatial position relationship between the boundary surface formed by the aerial images Sa and Sb and the projection device 20, so that the user can adjust the boundary surface that is easy to operate according to the environment in which the interface device 2 is actually installed. In addition, since this adjustment can also be performed after installing the interface device 2, the convenience for the user is also very excellent. For example, since the user can adjust the boundary surface that is easy to operate, the operability is improved, and various operations (pointer movement, pointer fixing, left click, right click, etc.) described in Embodiment 5 can be easily performed.

[0338] In addition, when the above-mentioned configuration relationship and angle are automatically adjusted, the interface device 2 obtains the position information of the user and the position information of the detection object (such as the user's hand) by using the detection device 21, and changes the above-mentioned configuration relationship and angle based on the obtained information, and controls the position of the boundary surface formed by the aerial images Sa and Sb. Thus, even in an environment where multiple unspecified users perform operations, it is possible to provide a boundary surface that is easy to operate for each different user. In addition, the user can also perform spatial operations based on the boundary surface that is easy to operate by himself, and can easily perform various operations (pointer movement, pointer fixing, left click, right click, etc.) as described in Embodiment 5.

[0339] Furthermore, in the interface device 2 of the sixth embodiment, the viewing angle of the detection device 21 is also set within a range that does not reflect the aerial images Sa and Sb projected by the projection device 20 , thereby suppressing a decrease in the resolution of the aerial images Sa and Sb.

[0340] In addition, in the above description, an example is described in which the imaging optical system is configured to include a beam splitter and a retroreflector, but the structure of the imaging optical system is not limited to this. For example, the imaging optical system may also be configured to include a dihedral corner reflector array element as described in Embodiment 2. In this case, in the interface device 2, as long as Fig.29 The retro-reflective elements 203a and 203b may be omitted and dihedral corner reflector array elements may be respectively arranged at the positions where the beam splitters 202a and 202b are arranged.

[0341] As described above, according to the sixth embodiment, the interface device 2 includes two or more light sources, each light source is arranged so that at least one of the axis and the plane in the space formed by each light source is not parallel, and the real image is formed as the aerial images Sa and Sb respectively by the paired beam splitter 202 and the retroreflector 203, and the aerial images Sa and Sb are formed parallel to each other on any surface on which the aerial images are projected in the virtual space K. Therefore, the interface device 2 of the sixth embodiment can control the spatial positional relationship of the aerial images Sa and Sb relative to the projection device 20 in addition to the effect of the second embodiment.

[0342] Furthermore, the posture of each light source is variable, and by changing the posture of each light source, the posture of each aerial image is changed, and the angle between the boundary surface on which each aerial image is projected and the horizontal plane is changed. Thus, the interface device 2 of the sixth embodiment can improve the operability of the user.

[0343] Implementation method 7.

[0344] In Embodiments 1 to 6, the interface device 2 is described as being configured separately from the display 10 in the display device 1. In Embodiment 7, the interface device 2 is described as being integrated with the display 10 in the display device 1.

[0345] Fig.33 1 is a perspective view showing a configuration example of the interface device 2 according to Embodiment 7, and is a perspective view showing an example of the configuration of the display 10 and the interface device 2 (projection device 20 and detection device 21). Fig.34 It is a side view showing a configuration example of the interface device 2 according to the seventh embodiment, and is a side view showing an example of the arrangement configuration of the display 10 and the interface device 2 (the projection device 20 and the detection device 21).

[0346] As in the first embodiment, the display 10 in the seventh embodiment is a device that displays digital image signals, such as a liquid crystal display and a plasma display. In the interface device 2 of the seventh embodiment, the display 10 is fixed to be integrated with the projection device 20 and the detection device 21. In addition, the display 10, the projection device 20 and the detection device 21 can be integrated by various methods, but as an example, the projection device 20 and the detection device 21 can be installed on the display 10 by applying a fixing fixture that is attached to the display 10 and complies with the VESA (Video Electronics Standards Association) standard, so as to integrate them.

[0347] For example Fig.33 As shown in FIG. 1 , the detection device 21 is arranged approximately near the center in the width direction (left-right direction) of the display 10. In addition, similarly to the second embodiment, the projection device 20 is configured to include a light source 201, two beam splitters 202a, 202b and two retro-reflective members 203a, 203b, such as Fig.33 and Fig.34 As shown, by arranging the projection device 20 from the front to the rear of the lower portion of the display 10 (from the front side to the rear side), the aerial images Sa and Sb are projected from the lower portion of the display 10 toward the front (front side).

[0348] In this case, for example Fig.33 As shown in FIG. 1 , the corresponding beam splitter 202a and the retroreflector 203a are arranged at the lower part of the display 10 and arranged at the left side of the detection device 21 in the width direction (left-right direction) of the display 10, and the corresponding beam splitter 202b and the retroreflector 203b are arranged at the lower part of the display 10 and arranged at the right side of the detection device 21 in the width direction (left-right direction) of the display 10. In addition, for example Fig.34As shown, the light source 201 is arranged in a position behind the beam splitters 202a, 202b and the retro-reflective members 203a, 203b in the frame of the projection device 20. As a result, the aerial image Sa is projected in a planar manner to the space on the left side of the detection device 21 in the width direction (left-right direction) of the display 10, and the aerial image Sb is projected in a planar manner to the space on the right side of the detection device 21 in the width direction (left-right direction) of the display 10. In this case, the two aerial images Sa and Sb are included in the same plane in space, and the plane including these aerial images Sa and Sb shows the boundary position (boundary plane) of each operation space in the virtual space K.

[0349] In this case, the larger the space between the light source 201 and the beam splitters 202a and 202b, the larger the imaging distance from the projection device 20 to the aerial images Sa and Sb becomes. Therefore, in the projection device 20, a convex lens may be arranged between the light source 201 and the beam splitters 202a and 202b to increase the imaging distance from the projection device 20 to the aerial images Sa and Sb. In addition, by arranging a mirror between the light source 201 and the beam splitters 202a and 202b to bend the straight optical path, the frame shape of the projection device 20 can be changed, and the versatility of the spatial setting of the projection device 20 can be improved.

[0350] The aerial images Sa and Sb projected by the projection device 20 are visually recognized by the user together with the image information displayed on the display 10. On the other hand, if the beam splitters 202a and 202b are not provided in the inner direction of the aerial images Sa and Sb on the light line that allows the aerial images Sa and Sb to be visually recognized from the user's viewpoint, the user cannot visually recognize the aerial images Sa and Sb. Therefore, in order for the user to visually recognize the aerial images Sa and Sb and the image information obtained from the display 10 within the same field of view, it is necessary to adjust the configuration of the projection device 20 and its internal structure.

[0351] For example, in the interface device 2, the angle ( Fig.34 The symbol α shown in the figure is thereby adjusted so that the beam splitters 202a, 202b are located in the inner direction of the aerial images Sa, Sb on the light line that enables the aerial images Sa, Sb to be visually confirmed from the user's viewpoint, and is adjusted so that the user can visually confirm the image information from the display 10 and the aerial images Sa, Sb within the same field of view.

[0352] In addition, in the interface device 2, by changing the distance between the light source 201 and the beam splitters 202a, 202b or the configuration angle of the beam splitters 202a, 202b, and changing the imaging position of the aerial images Sa, Sb, the beam splitters 202a, 202b are adjusted to be located in the inner direction of the aerial images Sa, Sb on the light line that can visually confirm the aerial images Sa, Sb from the user's viewpoint position, and the adjustment is made so that the user can visually confirm the image information from the display 10 and the aerial images Sa, Sb within the same field of view.

[0353] In addition, the function of adjusting the imaging position of the above-mentioned aerial images Sa and Sb can be achieved, for example, by manually adjusting the mechanical fixing position of the structural components (light source 201 and beam splitter 202, etc.) of the projection device 20, or by installing a control mechanism such as a stepping motor on the fixing fixture of the above-mentioned structural components and electronically controlling the fixing position of the structural components.

[0354] In addition, in the case where the fixed position of the above-mentioned structural component is electronically controlled as in the latter, the interface device 2 may also have a control unit (not shown), which obtains information representing the user's viewpoint position based on the detection results of the detection device 21 and prior parameter information, and automatically adjusts the fixed position of the above-mentioned structural component using the obtained information.

[0355] In addition, the control unit may appropriately adjust the fixed position of the structural member so as to change not only the imaging position of the aerial images Sa and Sb but also the angle at which the boundary surface shown by the aerial images Sa and Sb intersects with the display surface of the display 10 in space. For example, the control unit may appropriately adjust the fixed position of the structural member so as to make the boundary surface shown by the aerial images Sa and Sb close to horizontal and make the angle at which the boundary surface intersects with the display surface of the display 10 in space close to vertical (90 degrees).

[0356] In contrast, the control unit may appropriately adjust the fixed positions of the structural members so that the boundary surface shown by the aerial images Sa and Sb is close to vertical, and the angle at which the boundary surface intersects with the display surface of the display 10 in space is close to parallel (0 degrees). In this way, in the interface device 2, the spatial positional relationship of the aerial images Sa and Sb with respect to the display surface of the display 10 can be controlled, and a boundary surface that is easy for the user to operate can be provided.

[0357] Furthermore, in the interface device 2 of the seventh embodiment, the viewing angle of the detection device 21 is also set within a range where the aerial images Sa and Sb projected by the projection device 20 are not reflected, thereby suppressing a decrease in the resolution of the aerial images Sa and Sb.

[0358] In addition, in the above description, an example is described in which the imaging optical system is configured to include beam splitters 202a, 202b and retro-reflectors 203a, 203b, but the structure of the imaging optical system is not limited to this. For example, the imaging optical system may also be configured to include a dihedral corner reflector array element as described in Embodiment 2. In this case, in the interface device 2, as long as Fig.34 The retro-reflector 203a may be omitted and a dihedral corner reflector array element may be arranged at the position where the beam splitter 202a is arranged.

[0359] Thus, in the interface device 2 of embodiment 7, the projection device 20 and the detection device 21 are integrated with the display 10. Thus, the user can visually confirm the image information from the display 10 and the aerial images Sa and Sb projected by the projection device 20 within the same field of view. Such a configuration structure has the following advantages: even if the user only turns his consciousness to one of the visual feedback information relative to the spatial operation and the visual information displayed on the display 10 during the spatial operation of the interface device 2, the visual information of the other party can be visually confirmed. In addition, for users who are experiencing new spatial operations, the possibility of missing visual information can be reduced, the user's acceptability of spatial operations is improved, and the spatial operations can be understood intuitively and as early as possible.

[0360] In addition, in the description so far, an example in which the interface device 2 has the above-mentioned structure is described, but the interface system 100 described in Embodiment 5 may also have the above-mentioned structure. In this case, the user of the interface system 100 can not only visually confirm the image information from the display 10 and the aerial images Sa and Sb projected by the projection device 20 in the same visual field, but also control the spatial positional relationship of the aerial images Sa and Sb with respect to the display surface of the display 10, and can obtain a boundary surface that is easy for the user to operate.

[0361] As described above, according to Embodiment 7, the interface device 2 is integrally provided with the display 10 for displaying image information, and the aerial images Sa and Sb projected by the projection unit 20 can be visually confirmed by the user together with the image information displayed on the display 10. Thus, in addition to the effects of Embodiment 1, the interface device 2 of Embodiment 7 can reduce the possibility that the user misses the visual feedback information and image information with respect to the spatial operation.

[0362] In addition, the interface device 2 includes a control unit that changes the angle at which the boundary surface, which is a surface on which the aerial images Sa and Sb are projected in the virtual space K, intersects with the display surface of the display 10 in space. Thus, the interface device 2 of the seventh embodiment can control the spatial positional relationship of the aerial images Sa and Sb with respect to the display surface of the display 10, and can provide a boundary surface that is easy for the user to operate.

[0363] In addition, according to the seventh embodiment, the interface system 100 includes: a detection unit 21 that detects the three-dimensional position of the detection object in the virtual space K; a projection unit 20 that projects an aerial image into the virtual space K; and a display 10 that displays image information, wherein the virtual space K is divided into a plurality of operation spaces, wherein the plurality of operation spaces define operations that can be performed by the user when the three-dimensional position of the detection object detected by the detection unit 21 is included, and the boundary position of each operation space in the virtual space K is shown by the aerial image projected by the projection unit 20, and the aerial image projected by the projection unit 20 can be visually confirmed by the user together with the image information displayed on the display 10. Therefore, the interface system 100 of the seventh embodiment can reduce the possibility of the user missing the visual feedback information and image information with respect to the space operation in addition to the effect of the fifth embodiment.

[0364] In addition, the interface system 100 includes a control unit that changes the angle at which the boundary surface, which is a surface on which the aerial image is projected in the virtual space K, intersects with the display surface of the display 10 in space. Thus, the interface system 100 of Embodiment 7 can control the spatial positional relationship of the aerial images Sa and Sb relative to the display surface of the display 10, and can provide a boundary surface that is easy for the user to operate.

[0365] Implementation method 8.

[0366] In the description so far, the interface device 2 or the interface system 100 is described in which the boundary positions of each operation space in the virtual space K are displayed using the aerial image projected by the projection unit 20. In the eighth embodiment, the interface device 2 or the interface system 100 is described in which the boundary positions of each operation space can be displayed without using the aerial image.

[0367] For example, the interface device 2 according to the eighth embodiment is configured as follows.

[0368] An interface device 2, wherein the interface device 2 is capable of executing operations on an application displayed on a display, wherein the interface device 2 comprises:

[0369] a detection unit 21 for detecting a three-dimensional position of a detection object in a virtual space K divided into a plurality of operation spaces;

[0370] at least one boundary defining portion (not shown), the at least one boundary defining portion indicating a boundary of each operation space and being composed of a line or a surface; and

[0371] a boundary display unit (not shown), the boundary display unit being composed of points, lines or surfaces and being provided with at least one boundary of each operation space that can be visually confirmed,

[0372] When the three-dimensional position of the detection object detected by the detection unit 21 is included in the virtual space K, the detection object can be caused to execute a variety of operations on the application program that are associated with each operation space.

[0373] The boundary definition unit determines the boundaries of the virtual space K and each operation space. The virtual space K is an interface provided by the interface device 2 or the interface system 100 to enable the user to operate the application. By judging various user operations based on the definition of each boundary, software control that links user operations with application operations can be performed.

[0374] In other words, since the interface device 2 or the interface system 100 defines the boundaries of the virtual space K and each operation space, it is possible to establish a correspondence between the information of various user operations and the operations on the application that the user expects and to link them together. The information of various user operations is obtained by establishing a correspondence between the detection object existing in the virtual space K and the position or action of the detection object and each operation space and detecting them, or by detecting the action of the detection object that crosses each operation space or moves out of the virtual space K.

[0375] The boundary display unit is configured with components that enable the user to visually confirm the boundaries defined by the virtual space K and each operation space. The virtual space K and each operation space are provided as user interface components by the interface device 2 or the interface system 100 to the user who operates the application.

[0376] Specifically, for example Fig.35 As shown, the following methods can be cited: one or more components with marks indicating the boundary positions of each operation space are provided on the pillars indicating the upper and lower ranges of the virtual space K, or an aerial image indicating the boundaries of the virtual space K and each operation space is displayed in space. The marks indicating the boundary positions can be displayed by, for example, coloring, LEDs, or concave and convex configurations as dots or lines.

[0377] In addition, the display showing the boundary can be arranged in one or more positions relative to the same boundary, or the shape can be set to points or lines so that the user can recognize the boundaries of the virtual space K and each operation space.

[0378] That is, in the description so far, the interface device 2 or the interface system 100 is described as mainly using the aerial image projected by the projection unit 20 to show the boundary position of each operation space in the virtual space K. However, as long as the user can visually confirm the boundary position of each operation space, the interface device 2 or the interface system 100 does not necessarily need to project the aerial image. Therefore, in the eighth embodiment, the interface device 2 or the interface system 100 is not provided with an aerial image, but is provided with at least one visually confirmed boundary of each operation space composed of points, lines or surfaces. In this case, the user can also visually confirm the boundary positions of multiple operation spaces constituting the virtual space K as the operation object.

[0379] Furthermore, in the eighth embodiment, the boundary display unit may be composed of a projection unit 20 that projects an aerial image onto the virtual space K. In this case, the boundary positions of the operation spaces in the virtual space K may be indicated by the aerial image projected by the projection unit 20, and the aerial image projected by the projection unit 20 may be visually recognized by the user together with the image information displayed on the display 10. In this case, the structure is substantially the same as that of the interface device 2 in the seventh embodiment.

[0380] For example, compared with the case where an object other than an aerial image is displayed and the boundaries of each operation space are shown, in the case where an aerial image is displayed and the boundaries of each operation space are shown, there is no problem in arranging a display object near the operation space where the interface (gesture) is formed, and there is an advantage that the display object is unlikely to interfere with the user's actions. Therefore, in the case where it is desired to actively enjoy these advantages, it is preferable that the boundary display unit is composed of the projection unit 20 that projects the aerial image into the virtual space K, as described above.

[0381] As described above, according to the eighth embodiment, the interface device 2 can execute the operation of the application displayed on the display, wherein the interface device 2 includes: a detection unit 21, the detection unit 21 detects the three-dimensional position of the detection object in the virtual space K divided into a plurality of operation spaces; at least one boundary defining unit, the at least one boundary defining unit showing the boundary of each operation space and composed of a line or a surface; and a boundary display unit, the boundary display unit composed of a point, a line or a surface, and provided with at least one boundary of each operation space that can be visually confirmed, and when the three-dimensional position of the detection object detected by the detection unit 21 is included in the virtual space K, the detection object can be caused to execute a plurality of operations on the application that are respectively associated with each operation space. Thus, in the interface device 2 of the eighth embodiment, the boundary positions of the plurality of operation spaces constituting the virtual space that is the operation object of the user can be visually confirmed.

[0382] In addition, the boundary display unit is a projection unit 20 that projects an aerial image into the virtual space K. The boundary position of each operation space in the virtual space K is shown by the aerial image projected by the projection unit 20, and the aerial image projected by the projection unit 20 can be visually confirmed by the user together with the image information displayed on the display 10. Therefore, in the interface device 2 of the eighth embodiment, there is no problem in arranging a display object near the operation space where the interface (gesture) is formed, and the display object is unlikely to interfere with the user's actions.

[0383] In addition, if the correspondence between the boundary display unit and the boundary defining unit in the eighth embodiment and each functional unit described in other embodiments is supplemented, the boundary display unit in the eighth embodiment corresponds to, for example, the projection device (projection unit) 20 described in the first embodiment, etc. In addition, the boundary defining unit in the eighth embodiment corresponds to, for example, the position acquisition unit 41, the operation space determination unit 43, the pointer position control unit 45, the instruction generation unit 49, and the operation information output unit 51 described in the fifth embodiment.

[0384] Furthermore, in the present disclosure, each embodiment can be freely combined or arbitrary components of each embodiment can be modified, or arbitrary components can be omitted in each embodiment.

[0385] For example, in Embodiments 1 to 4, Embodiment 6, and Embodiment 7, examples are described in which the viewing angle of the detection unit 21 is set within a range that does not reflect the aerial images Sa and Sb showing the boundary positions of the operation space A and the operation space B in the virtual space K. However, as described in Embodiment 1, when an aerial image that does not show the boundary positions of the operation spaces in the virtual space K is projected into the virtual space K, it is not always necessary to prevent the aerial image from being reflected into the viewing angle of the detection unit 21.

[0386] For example, in the operation space B, an aerial image SC (see FIG. 1 ) showing the lower limit position of the detectable range of the detection unit 21 is shown. Figure 3 ) is sometimes projected by the projection unit 20. In addition, this aerial image SC is projected near the center position of the X-axis direction in the operation space B and shows the above-mentioned lower limit position, and sometimes becomes a reference for left and right instructions when the user moves the hand in the operation space B in accordance with the action corresponding to the instruction requiring left and right instructions such as left click and right click. Regarding such an aerial image SC, since it is not an aerial image showing the boundary position of each operation space in the virtual space K, it is not always necessary to prevent it from being reflected in the viewing angle of the detection device 21.

[0387] In addition, the projection device 20 may change the projection form of the aerial image projected into the virtual space K according to at least one of the operation space including the three-dimensional position of the detection object (e.g., the user's hand) detected by the detection device 21 and the movement of the detection object in the operation space including the three-dimensional position of the detection object. In addition, at this time, the projection device 20 may change the projection form of the aerial image projected into the virtual space K in units of pixels of the aerial image.

[0388] For example, the projection device 20 may also change the color or brightness of the aerial image projected to the virtual space K, depending on whether the operation space including the three-dimensional position of the detection object detected by the detection device 21 is the operation space A or the operation space B. In addition, at this time, the projection device 20 may change the color or brightness of the entire aerial image (all pixels of the aerial image) in the same manner, or may change the color or brightness of any part of the aerial image (pixels of any part of the aerial image). In addition, by changing the color or brightness of any part of the aerial image, the projection device 20 may increase the deformation of the projection form of the aerial image, such as adding any grayscale to the aerial image.

[0389] In addition, the projection device 20 may also flicker the aerial image projected to the virtual space K any number of times depending on whether the operation space containing the three-dimensional position of the detection object detected by the detection device 21 is the operation space A or the operation space B. In addition, at this time, the projection device 20 may flicker the entire aerial image (all pixels of the aerial image) in the same manner, or flicker any part of the aerial image (pixels of any part of the aerial image). According to the changes in the projection form as described above, the user can easily understand which operation space the operation space containing the three-dimensional position of the detection object is.

[0390] In addition, according to the action (gesture) of the detection object in the operation space B, the projection device 20 can, for example, change the color or brightness of the aerial image projected into the virtual space K, or can also flash the aerial image any number of times. In addition, in this case, the projection device 20 can also change or flash the color or brightness of the entire aerial image (all pixels of the aerial image) in the same way, or can change or flash the color or brightness of any part of the aerial image (pixels of any part of the aerial image). As a result, the user can easily grasp the action (gesture) of the detection object in the operation space B.

[0391] In addition, the "change in the projection form of the aerial image" mentioned here also includes the projection of the aerial image SC showing the lower limit position of the detectable range of the detection device 21. That is, when the operation space containing the three-dimensional position of the detection object detected by the detection device 21 is the operation space B, as an example of the change in the projection form of the aerial image, the projection device 20 can also project the aerial image SC showing the lower limit position of the detectable range of the detection device 21. In addition, as described above, the aerial image SC showing the lower limit position of the detectable range can also be projected within the viewing angle of the detection device 21. As a result, the user can easily understand to what extent the hand can be lowered in the operation space B, and can execute instructions that require left and right instructions.

[0392] According to the present disclosure, the operation information output unit 51 of the interface system 100 or the interface device 2 converts the information indicating the detection result of the three-dimensional position of the detection object in the virtual space K obtained by the position acquisition unit 41 (i.e., the information of the three-dimensional position of the detection object) into the information of the action of the detection object. Then, for example, in the operation space A, the operation information output unit 51 determines the action of the detection object in each operation space or across each operation space constituted in the virtual space K as the information of the pointer operation input, and in the operation space B, the operation information output unit 51 determines the action of the detection object in each operation space or across each operation space constituted in the virtual space K as the information of the instruction execution input. The content of the input operation such as the pointer operation and the instruction execution (or also referred to as "gesture" or "gesture operation", etc.) mentioned here is pre-set in multiple operation spaces in the virtual space K, and the operation information output unit 51 determines whether the action of the detection object in each operation space or across each operation space is equivalent to the prescribed input operation, and makes the prescribed operation of the application displayed on the display device 1 and the action of the detection object determined to be equivalent to the prescribed input operation linked. That is, the prescribed operation of the application can be executed in linkage with the action of the detection object in the virtual space K.

[0393] As described above, according to the technology disclosed in the present invention, the user can operate the application displayed on the display device 1 in a non-contact manner without using an operating device such as a mouse or a touch panel. This is related to reducing various constraints when the user operates the application. The various constraints are, for example, the space (width or height) of the platform where the operating device is set, the shape determined by the operating device itself, the function of the signal connection from the operating device to the display device 1, or the situation or state where it is difficult for the user to touch the operating device and operate it.

[0394] In this way, since the interface system 100 or the interface device 2 converts the user's actions in the virtual space K into information for operating the application, for example, even if no changes are made to the program or execution environment of the running (working) application executed on the existing display device 1, the user can operate the application in a non-contact manner via the virtual space K provided by the interface system 100 or the interface device 2.

[0395] Industrial Applicability

[0396] The present disclosure enables visual confirmation of the boundary positions of a plurality of operation spaces constituting a virtual space that is an operation target of a user, and is suitable for use in an interface device and an interface system.

[0397] Description of Reference Numerals

[0398] 1 display device, 2 interface device, 10 display, 11 display control device, 20 projection device (projection unit), 21 detection device (detection unit), 21a detection device, 21b detection device, 21c detection device, 31 aerial image projection unit, 32 position detection unit, 41 position acquisition unit (acquisition unit), 42 boundary position recording unit, 43 operation space determination unit (determination unit), 44 pointer operation information output unit, 45 pointer position control unit, 46 instruction determination unit, 47 instruction recording unit, 48 instruction output unit, 49 instruction generation unit, 50 aerial image generation unit, 51 operation information output unit, 100 interface Mouth system, 201 light source, 201a light source, 201b light source, 202 beam splitter, 202a beam splitter, 202b beam splitter, 203 retroreflector, 203a retroreflector, 203b retroreflector, 503 real image, 600 image display device, 604 display device, 605 light irradiator, 606 camera, 612 wavelength selection reflection component, 701 half mirror, 702 retroreflector, A operating space, B operating space, K virtual space, P pointer, R, operating screen, S aerial image, Sa aerial image, Sb aerial image, SC aerial image, U internal area.

Claims

1. An interface device, characterized in that: The interface device comprises: a detection unit configured to detect a three-dimensional position of a detection object in a virtual space; and a projection unit that projects an aerial image into the virtual space, The virtual space is divided into a plurality of operation spaces, and the plurality of operation spaces define operations that can be performed by a user when the three-dimensional position of the detection object detected by the detection unit is included. The boundary positions of the respective operation spaces in the virtual space are indicated by the aerial image projected by the projection unit.

2. The interface device according to claim 1, characterized in that: The projection unit forms an image of the aerial image in the virtual space such that the aerial image includes the viewing angle of the detection unit.

3. The interface device according to claim 1 or 2, characterized in that: The projection unit includes an imaging optical system having a light bending surface, wherein the light bending surface constitutes a plane that bends the optical path of light emitted from a light source, and the imaging optical system images a real image generated by the light source arranged on one side of the light bending surface as the aerial image on the opposite side of the light bending surface.

4. The interface device according to claim 3, characterized in that: The imaging optical system is composed of: A beam splitter having the light bending surface and separating the light radiated from the light source into transmitted light and reflected light; and A retro-reflector that, when the reflected light from the beam splitter is incident on the retro-reflector, reflects the reflected light along an incident direction.

5. The interface device according to claim 4, characterized in that: The beam splitter and the retroreflector are divided into n pieces (n is an integer greater than 2), respectively. The n beam splitters correspond one to one to the n retro-reflective elements, The n retro-reflective elements respectively reflect the reflected light from the corresponding beam splitter along the incident direction.

6. The interface device according to claim 3, characterized in that: The interface device comprises two or more light sources. The interface device comprises one or more imaging optical systems. Each of the light sources forms an image as the aerial image using one or more of the imaging optical systems.

7. The interface device according to claim 3, characterized in that: The imaging optical system is configured to include a dihedral corner reflector array element having the light bending surface.

8. The interface device according to claim 3, characterized in that: The detection unit is arranged in an internal region of the imaging optical system and is arranged on one side of the light bending surface of the imaging optical system.

9. The interface device according to claim 4, characterized in that: The detection unit is arranged at a position and viewing angle such that a detection path when detecting the three-dimensional position of the detection object is substantially the same as an optical path of light from the light source in the imaging optical system through the beam splitter and the retroreflector to reach the aerial image.

10. The interface device according to claim 1, characterized in that: The detection unit is composed of three or more line sensors, and the three or more line sensors set at least the area inside the boundary surface which is the surface projecting the aerial image in the virtual space and the area inside the surface sandwiching the boundary surface in the virtual space as a detectable range.

11. The interface device according to claim 1, characterized in that: The aerial image projected into the virtual space is formed at a position where a decrease in the detection accuracy of the three-dimensional position of the detection object by the detection unit is suppressed.

12. The interface device according to claim 1, characterized in that: The viewing angle of the detection unit is set to a range where the detection unit is not reflected in the aerial image projected by the projection unit.

13. The interface device according to claim 1, characterized in that: The projection unit changes the projection form of the aerial image projected into the virtual space based on at least one of an operation space including the three-dimensional position of the detection object detected by the detection unit and an action of the detection object within the operation space including the three-dimensional position of the detection object.

14. The interface device according to claim 1, characterized in that: One or more of the aerial images are projected in the virtual space, and one or more of the aerial images show the outer frame or outer surface of the virtual space to the user.

15. The interface device according to claim 12, characterized in that: At least any one of the plurality of projected aerial images is projected within the field of view of the detection unit.

16. The interface device according to claim 5, characterized in that: The interface device comprises two or more light sources. The light sources are arranged so that at least one of the axis and the plane in the space formed by the light sources is not parallel, and the real images are formed as the aerial images by using a pair of beam splitters and retroreflectors. The aerial images are formed in parallel with each other on any plane on which the aerial images are projected in the virtual space.

17. The interface device according to claim 16, characterized in that: The posture of each light source is changeable. By changing the posture of each light source, the posture of each aerial image is changed, and the angle formed by the boundary surface on which each aerial image is projected with respect to the horizontal plane is changed.

18. The interface device according to claim 1, characterized in that: The interface device is integrally provided with a display for displaying image information. The aerial image projected by the projection unit can be visually recognized by a user together with the video information displayed on the display.

19. The interface device according to claim 18, characterized in that: The interface device includes a control unit configured to change an angle at which a boundary surface, which is a surface on which the aerial image is projected in the virtual space, intersects with a display surface of the display in space.

20. An interface device capable of executing an operation of an application displayed on a display, characterized in that: The interface device comprises: a detection unit configured to detect a three-dimensional position of a detection object in a virtual space divided into a plurality of operation spaces; at least one boundary defining portion, the at least one boundary defining portion indicating a boundary of each of the operation spaces and being composed of a line or a surface; and a boundary display unit, the boundary display unit being composed of points, lines or surfaces and being provided with at least one boundary of each of the operation spaces that can be visually confirmed, When the three-dimensional position of the detection object detected by the detection unit is included in the virtual space, the detection object can be caused to execute a plurality of operations on the application program that are respectively associated with the operation spaces.

21. The interface device according to claim 20, characterized in that: The boundary display unit is a projection unit that projects an aerial image into the virtual space. using the aerial image projected by the projection unit to indicate the boundary positions of the operation spaces in the virtual space, The aerial image projected by the projection unit can be visually recognized by a user together with the video information displayed on the display.

22. An interface system, characterized in that: The interface system comprises: A detection unit, the detection unit detects a three-dimensional position of a detection object in a virtual space; a projection unit, the projection unit projects an aerial image into the virtual space; as well as a display, wherein the display displays image information, The virtual space is divided into a plurality of operation spaces, and the plurality of operation spaces define operations that can be performed by a user when the three-dimensional position of the detection object detected by the detection unit is included. using the aerial image projected by the projection unit to indicate the boundary positions of the operation spaces in the virtual space, The aerial image projected by the projection unit can be visually recognized by a user together with the video information displayed on the display.

23. The interface system according to claim 22, characterized in that: The interface system includes a control unit configured to change an angle at which a boundary surface, which is a surface on which the aerial image is projected in the virtual space, intersects with a display surface of the display in space.

24. An interface system, characterized in that: The interface system comprises: a detection unit configured to detect a three-dimensional position of a detection object in a virtual space divided into a plurality of operation spaces; an acquisition unit that acquires a three-dimensional position of the detection object detected by the detection unit; a projection unit configured to project an aerial image showing boundary positions of each of the operation spaces in the virtual space; a determination unit that determines the operation space including the three-dimensional position of the detection object based on the three-dimensional position of the detection object acquired by the acquisition unit and the boundary position of each of the operation spaces in the virtual space; as well as an operation information output unit that outputs operation information for executing a predetermined operation on an application displayed on a display device, using at least a determination result based on the determination unit, Each of the operation spaces corresponds to at least one of a plurality of operations performed on the application using a mouse or a touch panel, The continuous and different operations performed on the application are corresponding to adjacent operation spaces in each of the operation spaces.

25. An interface system, characterized in that: The interface system comprises: a detection unit configured to detect a three-dimensional position of a detection object in a virtual space divided into a plurality of operation spaces; an acquisition unit that acquires a three-dimensional position of the detection object detected by the detection unit; a projection unit configured to project an aerial image showing boundary positions of each of the operation spaces in the virtual space; a determination unit that determines the operation space including the three-dimensional position of the detection object based on the three-dimensional position of the detection object acquired by the acquisition unit and the boundary position of each of the operation spaces in the virtual space; as well as an operation information output unit that outputs operation information for executing a predetermined operation on an application displayed on a display device, using at least a determination result based on the determination unit, The operation information output unit determines the movement of the detection object based on the three-dimensional position of the detection object. A correspondence is established between the movement of the detection object within or across each of the operation spaces and at least one of a plurality of operations performed on the application using a mouse or a touch panel, and a prescribed operation on the application is linked with the movement of the detection object.

Images