Interface device and interface system

JPWO2024079971A5Active Publication Date: 2024-11-28MITSUBISHI ELECTRIC CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024551244
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-08-09
Filing Date
2023-08-09
Publication Date
2024-11-28
Estimated Expiration
2043-08-09

AI Technical Summary

Technical Problem

Conventional non-contact operation input technologies for electronic devices struggle to visually distinguish the boundary positions between multiple operation spaces in a virtual space, making it difficult for users to recognize and switch between different modes of interaction.

Method used

An interface device and system that includes a detection unit for tracking three-dimensional positions within a virtual space and a projection unit for displaying aerial images indicating the boundaries of operation spaces, allowing users to visually recognize and interact with multiple operation spaces.

Benefits of technology

Enables users to accurately recognize and switch between different operation modes by projecting boundary indicators within the virtual space, enhancing the usability and accuracy of non-contact operation inputs.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

An interface device (2) comprises: a detection unit (21) for detecting the three-dimensional position of a detection object in a virtual space (K); and a projection unit (20) for projecting an aerial image (S) in the virtual space. The virtual space is divided into a plurality of operation spaces in which operations executable by a user, when the three-dimensional position of the detection object detected by the detection unit is included, are set. The boundary position of each of the operation spaces in the virtual space is indicated by the aerial image projected by the projection unit.
Need to check novelty before this filing date? Find Prior Art

Description

Interface device and interface system

[0001] The present disclosure relates to an interface device and an interface system.

[0002] Conventionally, as an operation input technology for electronic devices, etc., a technology has been proposed in which a user operates a virtual space set in a real world to enable contactless operation input. In relation to such a technology, Patent Literature 1 discloses a display device having a function of controlling operation input by a user remotely operating a display screen.

[0003] This display device is equipped with two cameras that capture an area including a user viewing a display screen, and detects from the images captured by the cameras a second point that represents a user reference position relative to a first point that represents a camera reference position, and a third point that represents the position of the user's fingers, and sets a virtual surface space at a position a predetermined length from the second point in a first direction within the space, and determines and detects a predetermined operation by the user based on the degree to which the user's fingers have entered the virtual surface space.The display device then generates operation input information based on the results of the determination and detection, and controls the operation of the display device based on the generated information.

[0004] Here, the virtual surface space has no physical entity and is set as a three-dimensional space with position coordinates calculated by a processor or the like of the display device. This virtual surface space is configured as a roughly rectangular parallelepiped or flat space sandwiched between two virtual surfaces. The two virtual surfaces are a first virtual surface located in the front, closer to the user, and a second virtual surface located behind it.

[0005] For example, when a point representing a finger position reaches the first virtual surface from a first space in front of the first virtual surface and then enters a second space behind the first virtual surface, the display device automatically transitions to a state in which a predetermined operation is accepted and displays a cursor on the display screen. Furthermore, when a point representing a finger position reaches the second virtual surface through the second space and then enters a third space behind the second virtual surface, the display device determines and detects a predetermined operation (e.g., a touch, tap, swipe, or pinch on the second virtual surface). Upon detecting the predetermined operation, the display device controls the operation of the display device, including display control of a GUI on the display screen, based on the position coordinates of the detected point representing the finger position and operation information representing the predetermined operation.

[0006] Japanese Patent Application Laid-Open No. 2021-15637

[0007] The display device described in Patent Document 1 (hereinafter also referred to as the "conventional device") switches between a mode for accepting a predetermined operation and a mode for determining and detecting a predetermined operation depending on the position of the user's fingers in a virtual surface space. However, with the conventional device, it is difficult for the user to visually recognize at which position in the virtual surface space the above-mentioned modes are switched, in other words, the boundary positions of each space constituting the virtual surface space (the boundary position between the first space and the second space, and the boundary position between the second space and the third space).

[0008] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a technology that enables a user to visually recognize the boundary positions of multiple operation spaces that make up a virtual space that is the target of operation by the user.

[0009] An interface device according to the present disclosure includes a detection unit that detects the three-dimensional position of a detection target in a virtual space and a projection unit that projects an aerial image into the virtual space, the virtual space being divided into a plurality of operation spaces, each of which defines operations that a user can perform when the three-dimensional position of the detection target detected by the detection unit is contained within the virtual space, and the aerial image projected by the projection unit indicates the boundary position of each operation space in the virtual space. The interface device according to the present disclosure is also an interface device that enables the user to perform operations on an application displayed on a display, the interface device including: a detection unit that detects the three-dimensional position of the detection target in the virtual space divided into a plurality of operation spaces; at least one boundary definition unit consisting of a line or a surface that indicates the boundary of each operation space; and a boundary display unit that sets at least one visible boundary of each operation space consisting of a point, a line, or a surface, and is characterized in that when the three-dimensional position of the detection target detected by the detection unit is contained in the virtual space, the interface device enables the detection target to perform a plurality of types of operations on the application associated with each operation space. In addition, the interface system according to the present disclosure comprises a detection unit that detects the three-dimensional position of a detection target in a virtual space, a projection unit that projects an aerial image into the virtual space, and a display that displays video information, wherein the virtual space is divided into a plurality of operation spaces, each of which defines operations that the user can perform when the three-dimensional position of the detection target detected by the detection unit is contained within the operation space, and the aerial image projected by the projection unit indicates the boundary position of each operation space in the virtual space, and the aerial image projected by the projection unit can be viewed by the user together with the video information displayed on the display.The interface system according to the present disclosure also includes a detection unit that detects the three-dimensional position of a detection target in a virtual space divided into a plurality of operation spaces; an acquisition unit that acquires the three-dimensional position of the detection target detected by the detection unit; a projection unit that projects an aerial image indicating the boundary positions of each operation space in the virtual space; a determination unit that determines the operation space that contains the three-dimensional position of the detection target based on the three-dimensional position of the detection target acquired by the acquisition unit and the boundary positions of each operation space in the virtual space; and an operation information output unit that uses at least the determination result by the determination unit to output operation information for executing a predetermined operation on an application displayed on a display device, wherein each operation space corresponds to at least one of a plurality of types of operations on the application using a mouse or a touch panel, and adjacent operation spaces among the operation spaces are associated with consecutive different operations on the application. The interface system according to the present disclosure also includes a detection unit that detects the three-dimensional position of a detection target in a virtual space divided into a plurality of operation spaces; an acquisition unit that acquires the three-dimensional position of the detection target detected by the detection unit; a projection unit that projects an aerial image indicating the boundary positions of each operation space in the virtual space; a determination unit that determines the operation space that contains the three-dimensional position of the detection target based on the three-dimensional position of the detection target acquired by the acquisition unit and the boundary positions of each operation space in the virtual space; and an operation information output unit that uses at least the determination result by the determination unit to output operation information for executing a predetermined operation on an application displayed on a display device, wherein the operation information output unit identifies the movement of the detection target based on the three-dimensional position of the detection target, and associates the movement of the detection target within or across each operation space with at least one of a plurality of types of operations on the application using a mouse or a touch panel, thereby linking the predetermined operation on the application to the movement of the detection target.

[0010] According to the present disclosure, with the above configuration, it becomes possible to visually recognize the boundary positions of a plurality of operation spaces that make up the virtual space that is the target of operation by the user.

[0011] 1A is a perspective view showing an example of the configuration of an interface system according to embodiment 1, and FIG. 1B is a side view showing an example of the configuration of the interface system according to embodiment 1. FIG. 2A is a perspective view showing an example of the configuration of a projection device according to embodiment 1, and FIG. 2B is a side view showing an example of the configuration of the projection device according to embodiment 1. FIG. 3 is a diagram showing an example of a basic operation of the interface system according to embodiment 1. FIG. 4 is a perspective view showing an example of the arrangement of a projection device and a detection device in an interface device according to embodiment 1. FIG. 5 is a top view showing an example of the arrangement of a projection device and a detection device in an interface device according to embodiment 1. FIG. 6 is a perspective view showing an example of the arrangement of a projection device and a detection device in an interface device according to embodiment 2. FIG. 7 is a top view showing an example of the arrangement of a projection device and a detection device in an interface device according to embodiment 2. FIG. 8 is a side view showing an example of the arrangement of a projection device and a detection device in an interface device according to embodiment 3. FIG. 9 is a side view showing an example of the arrangement of a projection device and a detection device in an interface device according to embodiment 4. FIG. 10 is a diagram showing an example of the configuration of a conventional aerial image display system. FIG. 11 is a diagram showing an example of functional blocks of an interface system according to embodiment 5. FIG. 12 is a flowchart showing an example of the operation of the interface system according to embodiment 5 in "A. Aerial image projection phase". 22A to 22D are diagrams illustrating a continuous pointer movement operation in embodiment 5.

[0046] FIG. 22B is a flowchart illustrating an example of an operation in "B. Control execution phase" of the interface system according to embodiment 5.

[0047] FIG. 22C is a flowchart illustrating an example of an operation in "spatial processing A" of the interface system according to embodiment 5.

[0048] FIG. 22D is a diagram illustrating an operation in "spatial processing B" of the interface system according to embodiment 5.

[0049] FIG. 22D is a diagram illustrating a continuous pointer movement operation in embodiment 5.

[0050] FIG. 22C is a diagram illustrating a continuous pointer movement operation in embodiment 5.

[0051] FIG. 22D is a diagram illustrating a continuous pointer movement operation in embodiment 5.FIG. 23A is a diagram illustrating a continuous pointer movement operation in a conventional device, and FIG. 23B is a diagram illustrating a continuous pointer movement operation in embodiment 5. FIGS. 24A and 24B are diagrams illustrating a scroll operation in embodiment 5. FIG. 24A is a flowchart illustrating another example of operation in "B. Control Execution Phase" of the interface system according to embodiment 5. FIG. 24B is a flowchart illustrating an example of operation in "Spatial Processing AB" of the interface system according to embodiment 5. FIG. 27A is a diagram illustrating a left drag operation in embodiment 5, and FIG. 27B is a diagram illustrating a right drag operation in embodiment 5. FIGS. 28A and 28B are diagrams illustrating an example of a hardware configuration of a device control device according to embodiment 5. FIG. 28A is a perspective view illustrating an example of an arrangement of a projection device and a detection device in an interface device according to embodiment 6. FIG. 28B is a top view illustrating an example of an arrangement of a projection device and a detection device in an interface device according to embodiment 6. FIG. 28B is a front view illustrating an example of an arrangement of a projection device and a detection device in an interface device according to embodiment 6. FIG. 28B is a diagram for supplementing the explanation of the arrangement relationship between a light source and an aerial image in embodiment 6. FIG. 28A is a perspective view illustrating an example of a configuration of an interface device according to embodiment 7. FIG. 28B is a side view illustrating an example of a configuration of an interface device according to embodiment 7. FIG. 20 is a perspective view showing a configuration example of a boundary display unit in embodiment 8.

[0012] Hereinafter, embodiments will be described in detail with reference to the drawings. Embodiment 1. FIGS. 1A and 1B are diagrams showing an example of the configuration of an interface system 100 according to embodiment 1. As shown in FIGS. 1A and 1B, the interface system 100 includes a display device 1 and an interface device 2. FIG. 1A is a perspective view showing the example of the configuration of the interface system 100, and FIG. 1B is a side view showing the example of the configuration of the interface device 2.

[0013] <Display Device 1> As shown in FIG. 1A, the display device 1 includes a display 10 and a display control device 11, for example.

[0014] The display 10 displays various screens, including a predetermined operation screen R on which a pointer P operable by the user is displayed, under the control of, for example, a display control device 11. The display 10 is configured, for example, by a liquid crystal display, a plasma display, or the like.

[0015] The display control device 11 performs control for displaying various screens on the display 10. The display control device 11 is configured by, for example, a PC (Personal Computer), a server, and the like.

[0016] In the first embodiment, a user uses an interface device 2 (described later) to perform various operations on the display device 1. For example, the user uses the interface device 2 (described later) to operate a pointer P on an operation screen displayed on the display 10, or to execute various commands on the display device 1.

[0017] <Interface Device 2> The interface device 2 is a non-contact device that allows a user to input operations to the display device 1 without directly touching it. As shown in, for example, FIGS. 1A and 1B , the interface device 2 includes a projection device 20 and a detection device 21 disposed inside the projection device 20.

[0018] <Projection Device 20> The projection device 20 uses, for example, an imaging optical system to project one or more aerial images S into the virtual space K. The imaging optical system is, for example, an optical system having a ray bending surface that forms a plane along which the optical path of light emitted from a light source is bent.

[0019] 1B, the virtual space K is a space without a physical entity that is set within the range detectable by the detection device 21, and is a space that is divided into multiple operation spaces. Note that, although FIG. 1B shows an example in which the virtual space K is set in a position that is aligned with the detection direction of the detection device 21, the virtual space K is not limited to this and may be set in any position.

[0020] In the following description, for ease of understanding, an example will be described in which virtual space K is divided into two operation spaces (operation space A and operation space B). In this case, in the first embodiment, the aerial image S projected by the projection device 20 indicates the boundary position between operation space A and operation space B that constitute virtual space K, as shown in FIG. 1B , for example.

[0021] Next, a specific configuration example of the projection device 20 will be described with reference to Figures 2A and 2B. Figures 2A and 2B show an example in which the imaging optical system mounted in the projection device 20 includes a beam splitter 202 and a retroreflector 203. Reference numeral 201 denotes a light source. Figure 2A is a perspective view showing the configuration example of the projection device 20, and Figure 2B is a side view showing the configuration example of the projection device 20. Note that the detection device 21 is not shown in Figure 2B.

[0022] The light source 201 is configured by a display device that emits incoherent diffused light. The light source 201 is configured by, for example, a display device equipped with a liquid crystal element and a backlight, such as a liquid crystal display, a display device of a self-luminous device using organic EL elements and LED elements, or a projection device using a projector and a screen.

[0023] The beam splitter 202 is an optical element that separates incident light into transmitted light and reflected light, and its element surface functions as the above-mentioned light bending surface. The beam splitter 202 is made of, for example, an acrylic plate and a glass plate. When the beam splitter 202 is made of an acrylic plate and a glass plate, the intensity of transmitted light is generally higher than the intensity of reflected light. Therefore, the beam splitter 202 may be made of a half mirror in which a metal is added to the acrylic plate and the glass plate to improve the reflection intensity.

[0024] The beam splitter 202 may also be configured using a reflective polarizing plate whose reflection behavior and transmission behavior change depending on the polarization state of incident light due to liquid crystal elements and thin film elements.The beam splitter 202 may also be configured using a reflective polarizing plate whose transmittance and reflectance ratio change depending on the polarization state of incident light due to liquid crystal elements and thin film elements.

[0025] The retroreflective material 203 is a sheet-like optical element that has the retroreflective property of reflecting incident light directly in the direction it was incident. Optical elements that achieve retroreflection include bead-type optical elements in which small glass beads are spread over a mirror-like surface, micro triangular pyramids with convex mirror surfaces on each side, and microprism-type optical elements in which a triangular pyramid with its center cut out is spread over a surface.

[0026] In the projection device 20 equipped with the imaging optical system configured as described above, for example, light (diffused light) emitted from the light source 201 is specularly reflected on the surface of the beam splitter 202, and the reflected light is incident on the retroreflector 203. The retroreflector 203 retroreflects the incident light and causes it to be incident on the beam splitter 202 again. The light that has entered the beam splitter 202 passes through the beam splitter 202 and reaches the user. Then, by following the above-described optical path, the light emitted from the light source 201 reconverges and rediffuses at a position that is plane-symmetrical to the light source 201 with the beam splitter 202 as the boundary. This allows the user to perceive an aerial image S in the virtual space K.

[0027] Although an example in which the aerial image S is projected in a star shape is shown in FIGS. 2A and 2B, the shape of the aerial image S is not limited to this, and may be any shape.

[0028] In addition, in the above description, an example was described in which the imaging optical system provided in the projection device 20 is configured to include a beam splitter 202 and a retroreflective material 203, but the configuration of the imaging optical system is not limited to the above example.

[0029] For example, the imaging optical system may include a dihedral corner reflector array element, which is an element configured by arranging a plurality of two orthogonal mirror elements (mirrors) on a flat plate (substrate).

[0030] The dihedral corner reflector array element has the function of reflecting light incident from a light source 201 arranged on one side of the plate by one of two mirror elements, and then reflecting the reflected light by the other mirror element and passing it to the other side of the plate. When this light path is viewed from the side, the entrance path and exit path of the light are plane-symmetric with respect to the plate. In other words, the element surface of the dihedral corner reflector array element functions as the above-mentioned ray bending surface, and forms an aerial image S at a plane-symmetric position on the other side of the plate from a real image formed by the light source 201 on one side of the plate.

[0031] When the imaging optical system is configured with a dihedral corner reflector array element, this dihedral corner reflector array element is disposed at the position where the beam splitter 202 is disposed in the configuration using the above-described retroreflector 203. In this case, the retroreflector 203 is omitted.

[0032] The imaging optical system may also include, for example, a lens array element. The lens array element is an element configured by arranging a plurality of lenses on, for example, a flat plate (substrate). In this case, the element surface of the lens array element functions as the above-mentioned ray bending surface, and forms a real image formed by the light source 201 arranged on one side of the plate as an aerial image S at a plane-symmetric position on the other side. In this case, the distance from the light source 201 to the element surface is roughly proportional to the distance from the element surface to the aerial image S.

[0033] The imaging optical system may also include, for example, a holographic element. In this case, the element surface of the holographic element functions as the ray bending surface described above. By projecting light from light source 201, which is reference light, onto the holographic element, the holographic element outputs the light so as to reproduce the phase information of the light stored in the element. As a result, the holographic element forms a real image formed by light source 201, which is arranged on one side of the element, as an aerial image S at a plane-symmetric position on the other side.

[0034] <Detection Device 21> The detection device 21 detects the three-dimensional position of a detection target (for example, a user's hand) present in the virtual space K, for example.

[0035] A method for detecting a detection target by the detection device 21 includes, for example, irradiating the detection target with infrared light and detecting the Time of Flight (ToF) and an infrared pattern to calculate the depth position of the detection target present within the imaging angle of view of the detection device 21. In the first embodiment, the detection device 21 is configured, for example, with a three-dimensional camera sensor or a two-dimensional camera sensor that can also detect infrared wavelengths. In this case, the detection device 21 can calculate the depth position of the detection target present within the imaging angle of view and detect the three-dimensional position of the detection target.

[0036] Alternatively, the detection device 21 may be configured with a device that detects a one-dimensional position in the depth direction, such as a line sensor. When the detection device 21 is configured with a line sensor, it is possible to detect the three-dimensional position of the detection target by arranging multiple line sensors according to the detection range. An example in which the detection device 21 is configured with the line sensor will be described in detail in embodiment 4.

[0037] Furthermore, for example, the detection device 21 may be configured as a stereo camera device configured with multiple cameras. In this case, the detection device 21 performs triangulation from feature points detected within the imaging angle of view to detect the three-dimensional position of the detection target.

[0038] <Virtual Space K> Next, a specific configuration example of the virtual space K will be described with reference to FIG.

[0039] As described above, the virtual space K is a space without a physical entity that is set within the detectable range of the detection device 21, and is a space that is divided into the operational space A and the operational space B. For example, as shown in Fig. 3, the virtual space K is set to have a rectangular parallelepiped shape as a whole, and is a space that is divided into two operational spaces (the operational space A and the operational space B). In the following description, the operational space A will also be referred to as the "first operational space" and the operational space B will also be referred to as the "second operational space."

[0040] In this case, the aerial image S projected into the virtual space K by the projection device 20 indicates the boundary position between the two operational spaces, operational space A and operational space B. In FIG. 3 , two aerial images S are projected. These aerial images S are projected onto a closed plane (hereinafter, this plane will also be referred to as a "boundary surface") that separates operational space A and operational space B. Note that while FIG. 3 shows an example in which two aerial images S are projected, the number of aerial images S is not limited to this and may be one, three, or more. For ease of explanation, the shorter direction of the boundary surface is defined as the X-axis direction, the longer direction is defined as the Y-axis direction, and the direction perpendicular to the X-axis and Y-axis directions is defined as the Z-axis direction, as shown in FIG. 3 .

[0041] Furthermore, the operation space A and the operation space B are associated with operations that the user can perform when the three-dimensional position of the detection target detected by the detection device 21 is included in each operation space. In the following description, for ease of understanding, a case will be described where the detection target detected by the detection device 21 is the user's hand. In this case, the detection device 21 detects the three-dimensional position of the user's hand in the virtual space K, in particular, the three-dimensional positions of the five fingers of the user's hand in the virtual space K.

[0042] For example, the operation of a pointer P is associated with the operational space A as an operation that can be performed by the user. Specifically, for example, when the user places his / her hand in the operational space A, that is, when the three-dimensional positions of the five fingers of the user's hand detected by the detection device 21 are all contained within the operational space A, the user can move the pointer P displayed on the operation screen R of the display 10 in conjunction with the hand movement by moving the hand in the operational space A (left side of Figure 3). Note that although the left side of Figure 3 conceptually depicts the pointer P in the operational space A, in reality, the pointer P displayed on the operation screen R of the display 10 moves.

[0043] In the following description, "the three-dimensional position of the user's hand is contained within the operational space A" means "the three-dimensional positions of all five fingers of the user's hand are contained within the operational space A." Furthermore, in the following description, "the user operates the operational space A" means "the user moves his / her hand with the three-dimensional position of the user's hand contained within the operational space A."

[0044] Furthermore, when the user moves his / her hand from the operational space A across the boundary position (boundary surface) into the operational space B, that is, when the three-dimensional positions of the five fingers of the user's hand detected by the detection device 21 are all contained within the operational space B, the movement of the pointer P displayed on the operation screen R on the display 10 is fixed (right side of FIG. 3 ). Note that on the right side of FIG. 3 , the fact that the movement of the pointer P is fixed is indicated by brackets displayed at the four corners of the pointer P.

[0045] At this time, the pointer P does not move even if the user moves his / her hand in the operational space B. On the other hand, if the user moves his / her hand in a predetermined pattern in the operational space B, he / she can execute a command (left click, right click, etc.) corresponding to this movement (gesture). In other words, the operational space B is associated with, for example, the input (execution) of a command as an operation that can be executed by the user.

[0046] In the following description, "the three-dimensional position of the user's hand is contained within the operational space B" means "the three-dimensional positions of all five fingers of the user's hand are contained within the operational space B." Furthermore, in the following description, "the user operates the operational space B" means "the user moves his / her hand in a state where the three-dimensional position of the user's hand is contained within the operational space B."

[0047] In this way, the user can operate the operation space A to move the pointer P displayed on the operation screen R of the display 10, and then operate the operation space B to execute a command corresponding to the hand movement. In other words, adjacent operation spaces A and B are associated with operations performed by the user, particularly operations having continuity. Here, "operations having continuity" refers to operations that are normally expected to be performed consecutively in time, such as when a user moves the pointer P displayed on the operation screen R of the display 10 and then executes a predetermined command. Note that, among the operation spaces, all adjacent ones may be associated with operations having continuity, or some adjacent operation spaces may be associated with operations having continuity. In other words, it is also possible to associate non-continuous operations with other adjacent operation spaces.

[0048] 3 are projected onto a closed plane (boundary surface) that separates adjacent operational spaces A and B. In other words, these aerial images S indicate the adjacent boundary between the two adjacent operational spaces.

[0049] The range of operational space A is, for example, in the Z-axis direction in Fig. 3 , the range from the position of the boundary surface onto which the aerial image S is projected to the upper limit position of the range detectable by the detection device 21. The range of operational space B is, for example, in the Z-axis direction in Fig. 3 , the range from the position of the boundary surface onto which the aerial image S is projected to the lower limit position of the range detectable by the detection device 21.

[0050] Note that, on the right side of FIG. 3 , the aerial image SC is an aerial image projected by the projection device 20 when the user moves their hand from the operational space A across the boundary position (boundary surface) into the operational space B. The aerial image SC indicates the lower limit of the detectable range of the detection device 21 and also indicates a reference position for dividing the operational space B into left and right spaces as viewed from the user's side. The aerial image SC is projected by the projection device 20 near the lower limit of the detectable range of the detection device 21 and approximately near the center of the operational space B in the X-axis direction. While the aerial image S exists on a plane (boundary surface) where the coordinate position in the Z-axis direction is 0, the aerial image SC exists in a region where the coordinate position in the Z-axis direction is negative. This allows the user to easily grasp how far they can lower their hand in the operational space B and to execute commands that require left and right designation, such as left clicks and right clicks. How to input commands such as left clicks and right clicks will be described later.

[0051] Next, an example of the arrangement of the projection device 20 and the detection device 21 in the interface device 2 will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a perspective view showing an example of the arrangement of the projection device 20 and the detection device 21 in the interface device 2, and Fig. 5 is a top view showing an example of the arrangement of the projection device 20 and the detection device 21 in the interface device 2.

[0052] In the following explanation, for ease of understanding, an example will be given in which the imaging optical system of the projection device 20 is configured to include the beam splitter 202 and the retroreflective material 203 shown in Figures 2A and 2B.

[0053] In addition, in the following explanation, an example will be given in which the projection device 20 is configured to include two bar-shaped light sources 201a, 201b, and the light emitted from these two light sources 201a, 201b is re-converged and re-diffused at positions that are plane-symmetrical to each light source 201a, 201b across the beam splitter 202, thereby projecting two aerial images Sa, Sb composed of line-shaped figures into the virtual space K.

[0054] In addition, in the following explanation, an example will be given in which the detection device 21 is configured as a camera device that can detect the three-dimensional position of the user's hand by emitting infrared light as detection light and receiving infrared light reflected from the user's hand, which is the detection target.

[0055] 4 and 5, the detection device 21 is disposed inside the projection device 20. More specifically, the detection device 21 is disposed inside the imaging optical system provided in the projection device 20, particularly inside the beam splitter 202 that constitutes the imaging optical system.

[0056] In addition, the imaging angle of view (hereinafter also simply referred to as the "angle of view") of the detection device 21 is set within a range in which the aerial images Sa, Sb projected by the projection device 20 are not captured. Note that in FIGS. 4 and 5 , the angle of view of the detection device 21 is set within a range in which the aerial images Sa, Sb projected by the projection device 20 are not captured, and within an internal region U defined by these two aerial images Sa, Sb. In other words, the projection device 20 forms the aerial images Sa, Sb in the virtual space K so that the aerial images Sa, Sb are contained within the angle of view of the detection device 21. When this point is viewed from the aerial images Sa, Sb, the aerial images Sa, Sb are formed at positions that suppress a decrease in the accuracy with which the detection device 21 detects the three-dimensional position of the user's hand (detection target).

[0057] Here, the "internal area defined by the two aerial images Sa, Sb" refers to a rectangular area drawn on the boundary surface onto which the two aerial images Sa, Sb are projected, by connecting one end of the opposing aerial images Sa, Sb and connecting the other end of the opposing aerial images Sa, Sb together using these connecting lines and the two aerial images Sa, Sb.

[0058] While the above description is based on an example in which two aerial images are projected, the same applies to the case in which three or more aerial images composed of linear (straight) shapes are projected. For example, the "internal area defined by the three aerial images Sa, Sb, and Sc" refers to the area drawn on the boundary surface on which the three aerial images Sa, Sb, and Sc are projected by connecting the ends of adjacent aerial images Sa, Sb, and Sc and the three aerial images Sa, Sb, and Sc. Furthermore, the projection device 20 forms the three aerial images in the virtual space K so that the three aerial images encompass the angle of view of the detection device 21. When viewed from the aerial image, the three aerial images are each formed at a position that minimizes a decrease in the detection accuracy of the detection device 21 for the three-dimensional position of the user's hand (detection target).

[0059] Furthermore, when the aerial image S is configured not as a line (straight line) figure but as a figure having a closed area, such as a frame-shaped figure or a circular figure, the "internal area defined by the aerial image S" refers to the closed area, such as the area surrounded by the frame line of the frame-shaped figure or the area surrounded by the circumference of the circular figure. Furthermore, the projection device 20 forms the aerial image in the virtual space K so that the closed area of ​​the aerial image formed from the figure having a closed area is contained within the angle of view of the detection device 21. When viewed from the aerial image, the aerial image is formed at a position that suppresses a decrease in the accuracy of detection of the three-dimensional position of the user's hand (detection target) by the detection device 21.

[0060] In this way, by arranging the detection device 21 inside the imaging optical system provided in the projection device 20, particularly inside the beam splitter 202 that constitutes the imaging optical system, it is possible to reduce the size of the projection device 20, including the structure of the imaging optical system, while ensuring the specified detection distance for the detection device 21, which requires a specified detection distance from the user's hand, which is the detection target.

[0061] Furthermore, since the detection device 21 is disposed particularly inside the beam splitter 202 that constitutes the imaging optical system, this also contributes to stabilizing the accuracy with which the detection device 21 detects the user's hand.

[0062] For example, if the detection device 21 is exposed to the outside of the projection device 20, it is conceivable that the detection accuracy of the three-dimensional position of the user's hand may decrease due to external factors such as dust, dirt, and water. Furthermore, if the detection device 21 is exposed to the outside of the projection device 20, it is conceivable that external light such as sunlight or illumination light may enter the sensor unit of the detection device 21, causing noise when detecting the three-dimensional position of the user's hand.

[0063] In this regard, in the first embodiment, the detection device 21 is disposed inside the beam splitter 202 that constitutes the imaging optical system, and therefore it is possible to prevent a decrease in the detection accuracy of the three-dimensional position of the user's hand due to external factors such as dust, dirt, water, etc. Furthermore, by adding an optical material, such as a phase polarizer, that absorbs light other than the infrared light emitted by the detection device 21 and the light emitted from the light sources 201 a and 201 b to the surface of the beam splitter 202 (the surface facing the user), it is possible to prevent a decrease in detection accuracy due to external light such as sunlight or illumination light.

[0064] Furthermore, as described above, when a phase polarizer is added to the surface of the beam splitter 202 (the surface facing the user), in the interface device 2, this phase polarizer makes it difficult for the detection device 21 itself to be seen from outside the projection device 20. Therefore, in the interface device 2, the user does not get the impression that they are being photographed by a camera, and effects in terms of design can also be expected.

[0065] Furthermore, in the interface device 2, the angle of view of the detection device 21 is set within a range in which the aerial images Sa, Sb projected by the projection device 20 are not captured. Note that, as described above, in Figures 4 and 5, the angle of view of the detection device 21 is set within a range in which the aerial images Sa, Sb projected by the projection device 20 are not captured and within the internal region U defined by these two aerial images Sa, Sb. This prevents a decrease in the resolution of the aerial images Sa, Sb in the interface device 2. This will be described in detail below.

[0066] For example, International Publication No. 2018-78777 discloses an aerial image display system (hereinafter also referred to as the "conventional system") having a configuration similar to that of the interface device 2 according to the first embodiment.

[0067] This aerial image display system comprises an image display device that displays an image on a screen, an image forming element that forms an image light containing the displayed image into a real image in the air, a wavelength-selective reflecting element that is arranged on the incident surface side of the image forming element on which the image light is incident and that has the property of transmitting visible light and reflecting invisible light, and an imager that receives the invisible light reflected by a detectable object that performs an input operation on the real image and captures an image of the detectable object consisting of an invisible light image.

[0068] The image display device also includes an input operation determination unit that acquires an image of the detectable object from the image capture device and analyzes the image of the detectable object to analyze the input operation content of the detectable object, a main control unit that outputs an operation control signal based on the input operation content analyzed by the input operation determination unit, and an image generation unit that generates an image signal that reflects the input operation content in accordance with the operation control signal and outputs it to the image display device, and the wavelength-selective reflecting member is positioned at a position where the real image falls within the field of view of the image capture device.

[0069] An example configuration of an aerial image display system configured as described above is shown in Figure 10. In Figure 10, reference numeral 600 denotes an image display device, reference numeral 604 denotes an image display device, reference numeral 605 denotes a light irradiator, and reference numeral 606 denotes an image capture device. Reference numeral 610 denotes a wavelength-selective imaging device, reference numeral 611 denotes an imaging member, and reference numeral 612 denotes a wavelength-selective reflecting member. Reference numeral 701 denotes a half mirror, and reference numeral 702 denotes a retroreflective sheet. Reference numeral 503 denotes a real image.

[0070] 10 , the image display device 600 includes a display device 604 that emits image light to form a real image 503 that the user visually recognizes, a light irradiator 605 that irradiates infrared light to detect the three-dimensional position of the user's fingers, and an imager 606 consisting of a visible light camera. Also, in the conventional system shown in FIG. 10 , a wavelength-selective reflecting member 612 that reflects infrared light is added to the surface of the retroreflective sheet 702, so that the infrared light irradiated from the light irradiator 605 is reflected by the wavelength-selective reflecting member 612 and irradiated to the position of the user's hand, and a portion of the infrared light diffused by the user's fingers or the like is reflected by the wavelength-selective reflecting member 612 and incident on the imager 606, thereby enabling the user's position to be detected, etc.

[0071] However, in the conventional system configured as described above, the user operates the real image 503 by touching it. In other words, the position of the user's hand, whose position is to be detected, matches the position of the real image (aerial image) 503. Therefore, the wavelength-selective reflecting member 612 that reflects infrared light needs to be placed in the optical path of the image light originating from the display device 604 that emits the image light for forming the real image 503. In other words, in the conventional system described above, it is necessary to replace a portion of the image light emitted from the display device 604 with infrared light, which may result in a reduction in the resolution of the real image 503. Furthermore, the wavelength-selective reflecting member 612 added to the surface of the retroreflective sheet 702 also affects the optical path for forming the real image 503, which may cause a reduction in the brightness and resolution of the real image 503.

[0072] In contrast, in the interface device 2 according to embodiment 1, the aerial image S is used as a guide, so to speak, to indicate the boundary position between the operational space A and the operational space B that constitute the virtual space K, and therefore the user does not necessarily need to touch the aerial image S, and the detection device 21 does not need to detect the three-dimensional position of the user's hand that is touching the aerial image S.

[0073] Therefore, in the interface device 2 according to the first embodiment, the angle of view of the detection device 21 is set within a range in which the aerial images Sa, Sb projected by the projection device 20 are not captured, for example, so as to fall within an internal region U defined by the two aerial images Sa, Sb, and it is sufficient that the three-dimensional position of the user's hand can be detected within the internal region U. In this way, in the interface device 2 according to the first embodiment, the angle of view of the detection device 21 is set within a range in which the aerial images Sa, Sb projected by the projection device 20 are not captured. Therefore, unlike conventional systems, the optical path for forming the aerial image S is not obstructed by the optical path of the infrared light irradiated from the detection device 21. As a result, in the interface device 2 according to the first embodiment, a decrease in the resolution of the aerial image S is suppressed.

[0074] Furthermore, in the interface device 2 according to the first embodiment, the angle of view of the detection device 21 only needs to be set within a range that does not capture the aerial images Sa and Sb projected by the projection device 20, and therefore, unlike conventional systems, it is not necessary to take into consideration the positional relationship with other components that constitute the imaging optical system when arranging the detection device 21. As a result, in the interface device 2 according to the first embodiment, the detection device 21 can be arranged in a position close to the other components that constitute the imaging optical system, and as a result, the interface device 2 as a whole can be made smaller.

[0075] Furthermore, in the interface device 2, the projection device 20 forms the aerial images Sa and Sb in the virtual space K so that the aerial images Sa and Sb are within the angle of view of the detection device 21. That is, the aerial images Sa and Sb are formed at positions that suppress a decrease in the accuracy with which the detection device 21 detects the three-dimensional position of the user's hand (detection target). More specifically, for example, the aerial images Sa and Sb are formed at least outside the angle of view of the detection device 21. As a result, in the interface device 2, the aerial images Sa and Sb projected into the virtual space K do not interfere with the detection of the three-dimensional position of the user's hand by the detection device 21. Therefore, in the interface device 2, a decrease in the accuracy with which the detection device 21 detects the three-dimensional position of the user's hand, which is caused by the aerial images Sa and Sb being captured in the angle of view of the detection device 21, is suppressed.

[0076] In the above description, an example has been described in which the detection device 21 is disposed inside the projection device 20 (inside the beam splitter 202), but the detection device 21 does not necessarily have to be disposed inside the projection device 20 as long as the angle of view is set within a range in which the aerial images Sa, Sb projected by the projection device 20 are not captured. In that case, however, there is a risk that the overall size of the interface device 2, which includes the projection device 20 and the detection device 21, will increase. Therefore, it is desirable that the detection device 21 be disposed inside the projection device 20 as described above, and that the angle of view is set within a range in which the aerial images Sa, Sb projected by the projection device 20 are not captured.

[0077] In the above description, the imaging optical system of the projection device 20 is configured to include the beam splitter 202 and the retroreflector 203, and the detection device 21 is disposed inside the beam splitter 202 that constitutes the imaging optical system. However, the imaging optical system may have a configuration other than the above. In that case, the detection device 21 only needs to be disposed inside the above-mentioned ray bending surface included in the imaging optical system. "Inside the ray bending surface" refers to one side of the ray bending surface, which is the side on which the light source is disposed relative to the ray bending surface.

[0078] For example, if the imaging optical system is configured to include a dihedral corner reflector array element, the element surface of the dihedral corner reflector array element functions as the above-mentioned ray bending surface, and therefore the detection device 21 may be positioned inside the element surface of the dihedral corner reflector array element.

[0079] Furthermore, for example, if the imaging optical system is configured to include a lens array element, the element surface of the lens array element functions as the above-mentioned light bending surface, and therefore the detection device 21 may be positioned inside the element surface of the lens array element.

[0080] In the above explanation, an example was given in which the angle of view of the detection unit 21 is set to a range in which the aerial images Sa and Sb indicating the boundary positions between operation spaces A and B in the virtual space K are not reflected. However, when an aerial image that does not indicate the boundary positions of each operation space in the virtual space K is projected into the virtual space K, it is not necessarily required to prevent this aerial image from being reflected into the angle of view of the detection unit 21.

[0081] For example, in the operational space B, the projection unit 20 may project an aerial image SC indicating the lower limit of the range detectable by the detection unit 21 (see FIG. 3 ). Note that this aerial image SC is projected near the center position in the X-axis direction of the operational space B and indicates the lower limit position. It may also serve as a reference for specifying left and right when the user moves their hand in the operational space B in a motion corresponding to a command that requires specifying left and right, such as a left click or a right click. Such an aerial image SC does not indicate the boundary positions of the operational spaces in the virtual space K, and therefore does not necessarily need to be prevented from being captured within the angle of view of the detection device 21. In other words, aerial images other than those indicating the boundary positions of the operational spaces in the virtual space K may be projected within the angle of view of the detection device 21.

[0082] Furthermore, in the interface device 2, as described above, one or more aerial images are projected by the projection device 20, and in this case, the one or more aerial images may show the outer frame or outer surface of the virtual space K to the user.

[0083] For example, in the interface device 2, the projection device 20 can project an aerial image that indicates the boundary positions of each operation space in the virtual space K, and an aerial image that does not indicate the boundary positions. Of these, the former aerial image, i.e., the aerial image that indicates the boundary positions of each operation space in the virtual space K, can be an aerial image that indicates the boundary positions of each operation space in the virtual space K and also indicates the outer frame or outer surface of the virtual space K, by setting the projection position to, for example, a position along the outer edge of the virtual space K. In this case, by visually recognizing the aerial image, the user can easily grasp not only the boundary positions of each operation space in the virtual space K but also the outer edge of the virtual space K.

[0084] As described above, according to the first embodiment, the interface device 2 includes the detection unit 21 that detects the three-dimensional position of the detection target in the virtual space K and the projection unit 20 that projects the aerial image S into the virtual space K. The virtual space K is divided into a plurality of operation spaces, each of which defines an operation that the user can perform when the three-dimensional position of the detection target detected by the detection unit 21 is contained within the virtual space K. The aerial image S projected by the projection unit 20 indicates the boundary position of each operation space in the virtual space K. This makes it possible for the interface device 2 according to the first embodiment to visually recognize the boundary positions of the plurality of operation spaces that constitute the virtual space that is the target of operation by the user.

[0085] Furthermore, the projection unit 20 forms the aerial images Sa and Sb in the virtual space K so that the aerial images Sa and Sb are contained within the angle of view of the detection unit 21. As a result, in the interface device 2 according to the first embodiment, a decrease in the detection accuracy of the three-dimensional position of the detection target by the detection unit 21 is suppressed.

[0086] The projection unit 20 is also an imaging optical system having a ray bending surface that forms a plane along which the optical path of light emitted from the light source bends, and is equipped with an imaging optical system that forms a real image by a light source arranged on one side of the ray bending surface as aerial images Sa, Sb on the opposite side of the ray bending surface. This makes it possible for the interface device 2 according to the first embodiment to project the aerial images Sa, Sb using the imaging optical system.

[0087] The imaging optical system includes a beam splitter 202 having a ray bending surface that separates light emitted from the light source 201 into transmitted light and reflected light, and a retroreflector 203 that reflects the reflected light from the beam splitter 202 back in the incident direction when the reflected light is incident on the beam splitter 202. This enables the interface device 2 according to the first embodiment to project aerial images Sa and Sb using the retroreflection of light.

[0088] The imaging optical system includes a dihedral corner reflector array element having a ray-refracting surface, which enables the interface device 2 according to the first embodiment to project aerial images Sa and Sb using specular reflection of light.

[0089] The detector 21 is located in an internal region of the imaging optical system, on one side of a ray bending surface of the imaging optical system. This allows the interface device 2 according to the first embodiment to be miniaturized as a whole. It also prevents a decrease in the detection accuracy of the three-dimensional position of the detection target due to external factors such as dust, dirt, and water.

[0090] Furthermore, the aerial images Sa and Sb projected into the virtual space K are formed at positions that suppress a decrease in the detection accuracy of the three-dimensional position of the detection target by the detection unit 21. As a result, in the interface device 2 according to embodiment 1, a decrease in the detection accuracy of the three-dimensional position of the detection target by the detection unit 21 is suppressed.

[0091] The angle of view of the detector 21 is set within a range that does not capture the aerial images Sa and Sb projected by the projection unit 20. This prevents a decrease in the resolution of the aerial images Sa and Sb in the interface device 2 according to the first embodiment.

[0092] Furthermore, one or more aerial images are projected into the virtual space K, and the one or more aerial images show the user the outer frame or outer surface of the virtual space K. This allows the user to easily grasp the outer edge of the virtual space K in the interface device 2 according to the first embodiment.

[0093] Furthermore, at least one of the multiple projected aerial images is projected within the angle of view of the detection unit 21. This improves the degree of freedom in the projection position of the aerial image that indicates, for example, the lower limit position of the range detectable by the detection unit 21 in the interface device 2 according to the first embodiment.

[0094] Embodiment 2. In the first embodiment, an interface device 2 that can suppress a decrease in the resolution of the aerial images Sa, Sb and can reduce the size of the entire device has been described. In the second embodiment, an interface device 2 that can suppress a decrease in the resolution of the aerial images Sa, Sb and can further reduce the size of the entire device will be described.

[0095] Fig. 6 is a perspective view showing an example of the arrangement of the projection device 20 and the detection device 21 in the interface device 2 according to embodiment 2. Fig. 7 is a top view showing an example of the arrangement of the projection device 20 and the detection device 21 in the interface device 2 according to embodiment 2.

[0096] The interface device 2 according to the second embodiment differs from the interface device 2 according to the first embodiment shown in Figures 4 and 5 in that the beam splitter 202 is divided into two beam splitters 202a and 202b, and the retroreflector 203 is divided into two retroreflectors 203a and 203b.

[0097] Furthermore, a first imaging optical system including beam splitter 202a and retroreflector 203a projects an aerial image Sa into virtual space K (the space on the near side of the paper in FIG. 6 ), and a second imaging optical system including beam splitter 202b and retroreflector 203b projects an aerial image Sb into virtual space K. In other words, the two split beam splitters and the two retroreflectors correspond to each other, with beam splitter 202a corresponding to retroreflector 203a and beam splitter 202b corresponding to retroreflector 203b.

[0098] The principle of projection (imaging) of an aerial image by the first imaging optical system and the second imaging optical system is the same as that in embodiment 1. For example, retroreflector 203a reflects light reflected from corresponding beam splitter 202a in the incident direction, and retroreflector 203b reflects light reflected from corresponding beam splitter 202b in the incident direction.

[0099] Also in the interface device 2 according to the second embodiment, similarly to the interface device 2 according to the first embodiment, the detection device 21 is arranged inside the projection device 20. More specifically, the detection device 21 is arranged inside the first imaging optical system and the second imaging optical system included in the projection device 20, particularly in a region sandwiched between the light source 201 and the two beam splitters 202 a and 202 b.

[0100] Also, at this time, the angle of view of the detection device 21 is set to a range in which the aerial images Sa, Sb projected by the projection device 20 are not captured, as in embodiment 1, and in particular, the angle of view is set to fall within the internal area U defined by the two aerial images Sa, Sb.

[0101] In this way, in the interface device 2 according to the second embodiment, by using two imaging optical systems each including divided beam splitters 202 a, 202 b and retroreflectors 203 a, 203 b, it is possible to project aerial images Sa, Sb visible to the user onto the virtual space K, while making the overall size of the interface device 2 even smaller than that of the first embodiment. In this case, furthermore, by arranging the detection device 21 inside these two imaging optical systems, the overall size of the interface device 2 can be further reduced.

[0102] Furthermore, in the interface device 2 according to the second embodiment, the angle of view of the detection device 21 is set within a range in which the aerial images Sa and Sb projected by the projection device 20 are not captured, and therefore, as in the interface device 2 according to the first embodiment, a decrease in the resolution of the aerial images Sa and Sb is suppressed.

[0103] Although the above description has been given of an example in which there is one light source 201 and the beam splitter 202 and retroreflector 203 are each divided into two, the interface device 2 is not limited to this, and the number of light sources 201 may be increased to two, and separate light sources may be used for the first imaging optical system and the second imaging optical system. Furthermore, the number of additional light sources 201 and the number of divisions of the beam splitter 202 and retroreflector 203 are not limited to the above, and may be n (n is an integer of 2 or more).

[0104] In the above description, an example has been described in which the imaging optical system includes a beam splitter and a retroreflector, but the imaging optical system is not limited to this and may include, for example, a dihedral corner reflector array element as described in embodiment 1. In this case, in the interface device 2, the retroreflectors 203a and 203b in Fig. 6 are omitted, and the dihedral corner reflector array elements are disposed at the positions where the beam splitters 202a and 202b are disposed, respectively.

[0105] Furthermore, in the above description, an example has been described in which the beam splitter 202 and the retroreflector 203 are each divided into two in one imaging optical system, but the interface device 2 is not limited to this. For example, the interface device 2 may be provided with one or more imaging optical systems and two or more light sources 201. In this case, the number of imaging optical systems and the number of light sources 201 do not necessarily have to be the same, and the imaging optical systems and the light sources do not necessarily have to correspond to each other. In this case, each of the two or more light sources 201 may form a real image as an aerial image using one or more imaging optical systems.

[0106] For example, if one imaging optical system and two light sources 201 (first and second light sources) are provided, the first light source may form a real image as an aerial image using the imaging optical system, and the second light source may also form a real image as an aerial image using the imaging optical system. This configuration corresponds to the configuration shown in Figures 4 and 5.

[0107] Furthermore, for example, if three imaging optical systems are provided (first to third imaging optical systems) and four light sources 201 are provided (first to fourth light sources), the first light source may form a real image as an aerial image using only one of the imaging optical systems (for example, the first imaging optical system), or may form a real image as an aerial image using any two of the imaging optical systems (for example, the first imaging optical system and the second imaging optical system), or may form a real image as an aerial image using all of the imaging optical systems (first to third imaging optical systems).

[0108] Similarly, the second light source may form a real image as an aerial image S using only one imaging optical system (e.g., the second imaging optical system), may form a real image as an aerial image S using any two imaging optical systems (e.g., the second imaging optical system and the third imaging optical system), or may form a real image as an aerial image S using all imaging optical systems (the first to third imaging optical systems). The same applies to the third light source and the fourth light source below. This makes it easy to adjust the brightness of the aerial image S, the imaging position of the aerial image S, etc. in the interface device 2.

[0109] As described above, according to the second embodiment, the beam splitter 202 and the retroreflector 203 are each divided into n pieces (n is an integer of 2 or more), the n beam splitters and the n retroreflectors correspond one-to-one to each other, and each of the n retroreflectors reflects the reflected light from the corresponding beam splitter back in the incident direction. As a result, the interface device 2 according to the second embodiment has the same effects as the first embodiment, and can further reduce the overall size of the interface device 2 compared to the first embodiment.

[0110] Furthermore, the interface device 2 includes two or more light sources 201 and one or more imaging optical systems, and each light source forms a real image as an aerial image through one or more imaging optical systems. As a result, the interface device 2 according to the second embodiment not only achieves the effects of the first embodiment, but also makes it easy to adjust the brightness and imaging position of the aerial image.

[0111] Embodiment 3. In the first embodiment, an interface device 2 that can suppress a decrease in the resolution of the aerial images Sa, Sb and can reduce the size of the entire device has been described. In the third embodiment, an interface device 2 that can not only suppress a decrease in the resolution of the aerial images Sa, Sb and reduce the size of the entire device, but also extend the detection path from the detection device 21 to the detection target will be described.

[0112] 8 is a side view showing an example of the arrangement of the projection device 20 and the detection device 21 in an interface device 2 according to embodiment 3. In the interface device 2 according to embodiment 3, the arrangement of the detection device 21 is changed to a position near the light sources 201a and 201b compared to the interface device 2 according to embodiment 1 shown in FIGS. 4 and 5 . More specifically, the arrangement of the detection device 21 is changed to a position sandwiched between the light sources 201a and 201b in top view and slightly forward of the light sources 201a and 201b (closer to the beam splitter 202) in side view. Note that FIG. 8 shows the interface device 2 according to embodiment 3 as viewed from the side of the light source 201b and the aerial image Sb.

[0113] The angle of view of the detecting device 21 is set to face in substantially the same direction as the direction of light emitted from the light sources 201 a and 201 b in the imaging optical system, and is set within a range in which the aerial images Sa and Sb projected by the projection device 20 are not captured, as in the first embodiment.

[0114] In this way, by placing the detection device 21 near the light sources 201a and 201b and by setting the angle of view of the detection device 21 in approximately the same direction as the emission direction of the light emitted from the light sources 201a and 201b, the infrared light emitted by the detection device 21 when detecting the three-dimensional position of the user's hand is reflected by the beam splitter 202, retroreflected by the retroreflective material 203, passes through the beam splitter 202, and follows a path that leads to the user's hand at the end of the transmission.

[0115] That is, the infrared light emitted from detection device 21 follows substantially the same path as the light emitted from light sources 201 a and 201 b when the imaging optical system forms aerial images Sa and Sb. As a result, in interface device 2 according to embodiment 3, it is possible to suppress a decrease in the resolution of aerial image S and reduce the size of the entire device, while also extending the distance (detection distance) from detection device 21 to the user's hand, which is the detection target, compared to interface device 2 according to embodiment 1 in which the paths of the two lights are different.

[0116] In particular, when the detection device 21 is configured with a camera device capable of detecting the three-dimensional position of the user's hand, the camera device has a minimum distance (shortest detectable distance) that must be maintained between the camera device and the detection target in order to perform appropriate detection. The detection device 21 must ensure this shortest detectable distance in order to perform appropriate detection. On the other hand, there is also a demand for miniaturization of the overall size of the interface device 2.

[0117] In this regard, in the interface device 2 according to embodiment 3, by configuring the arrangement of the detection device 21 as described above, it is possible to reduce the overall size of the interface device 2 while extending the detection distance of the detection device 21 to ensure the shortest detectable distance and suppress a decrease in detection accuracy.

[0118] As described above, according to the third embodiment, the detector 21 is disposed at a position and angle of view such that the detection path when detecting the three-dimensional position of the detection target is substantially the same as the optical path of light in the imaging optical system that travels from the light sources 201 a and 201 b to the aerial images Sa and Sb via the beam splitter 202 and the retroreflector 203. As a result, in the interface device 2 according to the third embodiment, in addition to the effects of the first embodiment, the interface device 2 can achieve a reduction in the overall size of the interface device 2, while ensuring the shortest detectable distance of the detector 21 and suppressing a decrease in detection accuracy.

[0119] Embodiment 4. In the first embodiment, an example has been described in which the detection device 21 is configured with a camera device that can detect the three-dimensional position of the user's hand by irradiating it with detection light (infrared light). In the fourth embodiment, an example will be described in which the detection device 21 is configured with a device that detects the position in the depth direction in one dimension.

[0120] 9 is a side view showing an example of the arrangement of the projection device 20 and the detection device 21 in the interface device 2 according to embodiment 4. In the interface device 2 according to embodiment 4, the detection device 21 is changed to detection devices 21a, 21b, and 21c in comparison with the interface device 2 according to embodiment 1 shown in FIGS.

[0121] The detection devices 21a, 21b, and 21c are configured, for example, by line sensors that detect the one-dimensional depth position of the user's hand by emitting detection light (infrared light) to the user's hand, which is the detection target. Note that Fig. 9 shows the interface device 2 according to the fourth embodiment as viewed from the side of the light source 201b and the aerial image Sb.

[0122] In addition, the angle of view of the detection device 21b is set so that it faces the direction in which the aerial images Sa and Sb are projected, and so that the plane (scanning plane) formed by the detection light (infrared light) substantially overlaps with the boundary surface onto which the aerial images Sa and Sb are projected. In other words, the detection device 21b detects the position of the user's hand in the area near the boundary surface onto which the aerial images Sa and Sb are projected. However, the angle of view of the detection device 21b is set within a range that does not capture the aerial images Sa and Sb, similar to the interface device 2 according to the first embodiment.

[0123] Furthermore, detection device 21a is installed higher than detection device 21b, and its angle of view is set so that it faces the direction in which the aerial images Sa and Sb are projected, and the plane (scanning plane) formed by the detection light is set so that it is approximately parallel to the boundary plane. In other words, detection device 21a has as its detectable range the area inside the scanning plane in the space (operation space A) above the boundary plane, and detects the position of the user's hand in this area.

[0124] Furthermore, detection device 21c is installed below detection device 21b, and its angle of view is set so that it faces the direction in which the aerial images Sa and Sb are projected, and so that the plane (scanning plane) formed by the detection light is approximately parallel to the boundary plane. In other words, detection device 21c has as its detectable range the area inside the scanning plane in the space (operation space B) below the boundary plane, and detects the position of the user's hand in this area. Note that the angles of view of detection devices 21a and 21c are also set to a range in which the aerial images Sa and Sb are not captured, similar to the interface device 2 according to embodiment 1.

[0125] As described above, the interface device 2 according to the fourth embodiment uses the detection devices 21a, 21b, and 21c configured as line sensors as the detection device 21, and the angle of view of each detection device is set so that the planes (scanning planes) formed by the detection light from each detection device are parallel to each other and so that the planes are positioned in the vertical (front-back) space centered on the boundary plane. This makes it possible for the interface device 2 according to the fourth embodiment to detect the three-dimensional position of the user's hand in the virtual space K using the line sensors.

[0126] Furthermore, since line sensors are smaller and cheaper than camera devices capable of detecting the three-dimensional position of a user's hand, as described in embodiment 1, by using a line sensor as detection device 21, the overall size of the device can be made smaller than that of interface device 2 according to embodiment 1, and costs can also be reduced.

[0127] Although the above description has been given of an example in which three detection devices each made up of a line sensor are used, the number is not limited to this. However, as described above, it is desirable to install at least three or more detection devices each made up of a line sensor so that the position of the user's hand can be detected in a space including planes in the up-down direction (front-back direction) centered on the boundary surface.

[0128] Thus, according to the fourth embodiment, the detection unit 21 is configured with three or more line sensors whose detectable ranges include at least the area inside the boundary surface, which is the surface onto which the aerial images Sa and Sb are projected in the virtual space K, and the areas inside the surfaces sandwiching the boundary surface in the virtual space K. As a result, in addition to the effects of the first embodiment, the interface device 2 according to the fourth embodiment can be made smaller in size as a whole than the interface device 2 according to the first embodiment, and costs can also be reduced.

[0129] Fifth Embodiment In the first to fourth embodiments, a configuration example of the interface device 2 included in the interface system 100 has been mainly described. In the fifth embodiment, a functional block example of the interface system 100 will be described. Fig. 11 shows an example of a functional block diagram of the interface system 100 in the fifth embodiment.

[0130] As shown in FIG. 11 , the interface system 100 includes an aerial image projection unit 31, a position detection unit 32, a position acquisition unit 41, a boundary position recording unit 42, an operation space determination unit 43, a pointer operation information output unit 44, a pointer position control unit 45, a command identification unit 46, a command recording unit 47, a command output unit 48, a command generation unit 49, and an aerial image generation unit 50.

[0131] The aerial image projection unit 31 acquires data indicating the aerial image S generated by the aerial image generation unit 50, and projects the aerial image S based on the acquired data into the virtual space K. The aerial image projection unit 31 is configured, for example, by the above-mentioned projection device 20. Note that the aerial image projection unit 31 may also acquire data indicating the above-mentioned aerial image SC generated by the aerial image generation unit 50, and project the aerial image SC based on the acquired data into the virtual space K.

[0132] The position detection unit 32 detects the three-dimensional position of the detection target (here, the user's hand) in the virtual space K. The position detection unit 32 is configured, for example, by the above-mentioned detection device 21. The position detection unit 32 outputs the detection result of the three-dimensional position of the detection target (hereinafter also referred to as the "position detection result") to the position acquisition unit 41.

[0133] In addition, the position detection unit 32 may detect the three-dimensional position of the aerial image S projected into the virtual space K, and record data indicating the detected three-dimensional position of the aerial image S in the boundary position recording unit 42.

[0134] In addition, when the aerial image projection unit 31 is configured by the above-mentioned projection device 20 and the position detection unit 32 is configured by the above-mentioned detection device 21, the functions of the aerial image projection unit 31 and the position detection unit 32 are realized by the above-mentioned interface device 2.

[0135] The position acquisition unit 41 acquires the position detection result output from the position detection unit 32. The position acquisition unit 41 outputs the acquired position detection result to the operational space determination unit 43.

[0136] The boundary position recording unit 42 records data indicating the boundary position between the operational space A and the operational space B that constitute the virtual space K, i.e., the three-dimensional position of the aerial image S. The boundary position recording unit 42 is configured by, for example, an HDD (Hard Disc Drive), an SSD (Solid State Drive), etc.

[0137] For example, if the aerial image S is configured as a line (straight line) as shown in Fig. 3, the boundary position recording unit 42 records data indicating the three-dimensional position of at least one of the points (pixels) of the aerial image S that make up the line. For example, the boundary position recording unit 42 may record data indicating the three-dimensional positions of any three of the points of the aerial image S that make up the line, or may record data indicating the three-dimensional positions of all of the points of the aerial image S that make up the line. Note that, because the aerial image S is projected onto the boundary surface shown in Fig. 3, the coordinate positions in the Z-axis direction of each point recorded in the boundary position recording unit 42 will all be the same.

[0138] The operation space determination unit 43 acquires the position detection result output from the position acquisition unit 41. Furthermore, the operation space determination unit 43 determines the operation space in which the user's hand is present based on the acquired position detection result and the boundary positions of each operation space in the virtual space K. The operation space determination unit 43 outputs the determination result (hereinafter also referred to as the "space determination result") to the aerial image generation unit 50. Furthermore, the operation space determination unit 43 outputs the space determination result, together with the position detection result acquired from the position acquisition unit 41, to the operation information output unit 51.

[0139] The operation information output unit 51 uses at least the space determination result by the operation space determination unit 43 to output operation information for executing a predetermined operation on the display device 1. The operation information output unit 51 is configured to include a pointer operation information output unit 44, a command identification unit 46, and a command output unit 48.

[0140] The pointer operation information output unit 44 acquires the space determination result and the position detection result output from the operational space determination unit 43. When the acquired space determination result indicates that the user's hand is present in the operational space A, the pointer operation information output unit 44 generates information (hereinafter also referred to as "movement control information") for moving the pointer P displayed on the operation screen R of the display 10 in accordance with the movement of the user's hand in the operational space A. Note that the "movement of the user's hand" includes information related to the movement, such as the amount of movement of the user's hand. For example, the pointer operation information output unit 44 calculates the amount of movement of the user's hand based on the position detection result output from the operational space determination unit 43. The amount of movement of the user's hand includes information related to the direction in which the user's hand moved and the distance the user's hand moved in that direction.

[0141] Then, based on the calculated amount of movement, the pointer operation information output unit 44 generates information (movement control information) for moving the pointer P displayed on the operation screen R of the display 10 in accordance with the movement of the user's hand in the operation space A. The pointer operation information output unit 44 outputs the operation information including the generated movement control information to the pointer position control unit 45.

[0142] Furthermore, when the acquired space determination result indicates that the user's hand is present in the operation space B, the pointer operation information output unit 44 generates information (hereinafter also referred to as "fixation control information") to fix the pointer P displayed on the operation screen R of the display 10. The pointer operation information output unit 44 outputs the operation information including the generated fixation control information to the pointer position control unit 45.

[0143] In addition, the pointer operation information output unit 44 may output information including in the operation information that the amount of movement or the speed of movement of the pointer P displayed on the screen of the display device 1 is variable depending on the distance between the three-dimensional position of the user's hand contained in the operation space A and the boundary surface of the virtual space K shown by the aerial image S, in a direction perpendicular to the boundary surface (the Z-axis direction in Figure 3).

[0144] The pointer position control unit 45 acquires operation information output from the pointer operation information output unit 44. When movement control information is included in the operation information acquired from the pointer operation information output unit 44, the pointer position control unit 45 moves the pointer P on the operation screen R displayed on the display 10 in accordance with the movement of the user's hand based on the movement control information. For example, the pointer position control unit 45 moves the pointer P by an amount equivalent to the amount of movement of the user's hand, in other words, in a direction included in the amount of movement and by a distance included in the amount of movement.

[0145] In addition, if the operation information obtained from the pointer operation information output unit 44 includes fixation control information, the pointer position control unit 45 fixes the pointer P on the operation screen R displayed on the display 10 based on the fixation control information.

[0146] The command identification unit 46 acquires the space determination result and the position detection result output from the operation space determination unit 43. When the acquired space determination result indicates that the user's hand is present in operation space B, the command identification unit 46 identifies the user's hand movement (gesture) based on the position detection result output from the operation space determination unit 43.

[0147] Command information is pre-recorded in the command recording unit 47. The command information is information in which a user's hand movements (gestures) are associated with commands that the user can execute. The command recording unit 47 is configured by, for example, a hard disc drive (HDD) or a solid state drive (SSD).

[0148] Command identification unit 46 identifies a command corresponding to the identified hand movement (gesture) of the user, based on the command information recorded in command recording unit 47. Command identification unit 46 outputs the identified command to command output unit 48 and aerial image generation unit 50.

[0149] The command output unit 48 acquires the command output from the command identification unit 46. The command output unit 48 outputs the operation information including information indicating the acquired command to the command generation unit 49.

[0150] The command generating unit 49 receives the operation information output from the command output unit 48 and generates a command included in the received operation information, thereby causing the interface system 100 to execute a command corresponding to the user's hand movement (gesture).

[0151] The aerial image generation unit 50 generates data representing the aerial image S to be projected by the aerial image projection unit 31 into the virtual space K. The aerial image generation unit 50 outputs the data representing the generated aerial image S to the aerial image projection unit 31.

[0152] The aerial image generation unit 50 may also acquire the space determination result output from the operation space determination unit 43, and regenerate data representing the aerial image S to be projected in a manner corresponding to the acquired space determination result. The aerial image generation unit 50 may also output data representing the regenerated aerial image S to the aerial image projection unit 31.

[0153] For example, when the space determination result indicates that the user's hand is present in operation space A, the aerial image generation unit 50 may regenerate data representing the aerial image S to be projected in blue. Furthermore, when the space determination result indicates that the user's hand is present in operation space B, the aerial image generation unit 50 may regenerate data representing the aerial image S to be projected in red. Furthermore, when the space determination result indicates that the user's hand is present in operation space B, the aerial image generation unit 50 may generate data representing the above-mentioned aerial image SC and output the generated data representing the aerial image SC to the aerial image projection unit 31.

[0154] The aerial image generation unit 50 may also acquire a command output from the command identification unit 46, and regenerate data representing the aerial image S to be projected in a manner corresponding to the acquired command. The aerial image generation unit 50 may also output data representing the regenerated aerial image S to the aerial image projection unit 31.

[0155] For example, if the command acquired from the command identification unit 46 is a left click, the aerial image generation unit 50 may regenerate data representing the aerial image S that blinks once. Furthermore, if the command acquired from the command identification unit 46 is a left double click, the aerial image generation unit 50 may regenerate data representing the aerial image S that blinks twice consecutively.

[0156] The above-described operation information output unit 51 may include a sound information output unit (not shown) that generates information indicating that a sound corresponding to the fixation of the pointer P (a sound notifying that the pointer P has been fixed) is to be output when operation information including fixation control information is output from the pointer operation information output unit 44 to the pointer position control unit 45, and outputs the generated information together with the operation information. In this case, when the pointer position control unit 45 fixes the pointer P based on the fixation control information, a sound corresponding to the fixation of the pointer P is output. Therefore, the user can easily know that the pointer P has been fixed by hearing this sound.

[0157] Furthermore, the sound information output unit may generate information indicating that a sound corresponding to the command identified by the command identification unit 46 will be output, and output the generated information together with the operation information. In this case, when the command generation unit 49 generates a command, a sound corresponding to the command is output. Therefore, by hearing this sound, the user can easily recognize that the command has been generated.

[0158] Furthermore, the sound information output unit may generate information indicating that a sound corresponding to the three-dimensional position of the user's hand in the operational space A or a sound corresponding to the movement of the user's hand in the operational space A will be output, and the generated information may be included in and output by the operation information. For example, the sound information output unit may generate information indicating that a sound corresponding to the three-dimensional position of the user's hand in the operational space A is output based on the three-dimensional position of the user's hand detected by the position detection unit 32, and the generated information may be included in and output by the operation information. In this case, for example, when the user brings their hand closer to a boundary surface in the operational space A, a sound whose volume increases as the user's hand approaches the boundary surface is output. By hearing this sound, the user can easily know that their hand is approaching the boundary surface.

[0159] Furthermore, for example, the sound information output unit may generate information indicating that a sound corresponding to the amount of movement of the user's hand calculated by the pointer operation information output unit 44 is to be output, and output the generated information as part of the operation information. In this case, for example, the greater the user's hand movement in the operational space A (the greater the amount of hand movement), the louder the sound that is output. By hearing this sound, the user can easily understand that their hand has moved significantly. In this way, by listening to the sound, the user can easily understand the three-dimensional position or movement of their hand in the operational space A.

[0160] In the fifth embodiment, the above-described position acquisition unit 41, boundary position recording unit 42, operational space determination unit 43, pointer operation information output unit 44, pointer position control unit 45, command identification unit 46, command recording unit 47, command output unit 48, command generation unit 49, and aerial image generation unit 50 are mounted on, for example, the above-described display control device 11. In this case, the device control device 12 is configured to include the position acquisition unit 41, boundary position recording unit 42, operational space determination unit 43, pointer operation information output unit 44, command identification unit 46, command recording unit 47, command output unit 48, and aerial image generation unit 50. The device control device 12 controls the interface device 2.

[0161] In the above explanation, an example was described in which the boundary position recording unit 42 and the command recording unit 47 are mounted on the device control device 12, but the boundary position recording unit 42 and the command recording unit 47 are not limited to this and may also be provided outside the device control device 12.

[0162] Next, an operation example of interface system 100 according to embodiment 5 will be described with reference to the flowcharts shown in Figures 12 to 15. Here, for ease of understanding, the operation example of interface system 100 will be described divided into "A. Aerial image projection phase" and "B. Control execution phase."

[0163] <A. Aerial Image Projection Phase> First, the aerial image projection phase will be described with reference to the flowchart shown in Fig. 12. In the aerial image projection phase, an aerial image S is projected into a virtual space K. Note that the aerial image projection phase is executed at least once when the interface system 100 is started up.

[0164] First, the aerial image generation unit 50 generates data representing the aerial image S to be projected by the aerial image projection unit 31 into the virtual space K (step A001). The aerial image generation unit 50 outputs the data representing the generated aerial image S to the aerial image projection unit 31.

[0165] Next, the aerial image projection unit 31 acquires data representing the aerial image S generated by the aerial image generation unit 50, and projects the aerial image S based on the acquired data into the virtual space K (step A002).

[0166] Next, the position detection unit 32 detects the three-dimensional position of the aerial image S projected into the virtual space K, and records data indicating the detected three-dimensional position of the aerial image S in the boundary position recording unit 42 (step A003).

[0167] In the above description, an example has been described in which the aerial image projection unit 31 first projects the aerial image S, then the position detection unit 32 detects the three-dimensional position of the aerial image S, and then data indicating the detected three-dimensional position of the aerial image S is recorded in the boundary position recording unit 42. However, step A003 is not an essential process and may be omitted. For example, in the interface system 100, the user may first record data indicating the three-dimensional position of the aerial image S in the boundary position recording unit 42, and the aerial image projection unit 31 may project the aerial image S at the three-dimensional position indicated by this data. In this case, step A003 may be omitted.

[0168] <B. Control Execution Phase> Next, the control execution phase will be described with reference to the flowchart shown in Fig. 13. In the control execution phase, the user uses the interface device 2, and control is executed by the display control device 11 and the device control device 12. Note that the control execution phase is repeatedly executed at predetermined intervals after the above-described aerial image projection phase is completed.

[0169] First, when the user places his / her hand in the virtual space K, the position detection unit 32 detects the three-dimensional position of the user's hand in the virtual space K (step B001). The position detection unit 32 outputs the detection result of the three-dimensional position of the user's hand (position detection result) to the position acquisition unit 41.

[0170] Next, the position acquisition unit 41 acquires the position detection result output from the position detection unit 32 (step B002). The position acquisition unit 41 outputs the acquired position detection result to the operational space determination unit 43.

[0171] Next, the operation space determination unit 43 acquires the detection results output from the position acquisition unit 41, and determines the operation space in which the user's hands are located based on the acquired position detection results and the boundary positions of each operation space in the virtual space K.

[0172] For example, the operational space determination unit 43 compares the position coordinates of the five fingers of the user's hand in the Z-axis direction shown in Fig. 3 with the position coordinates of the boundary position between operational spaces A and B in the Z-axis direction. Then, if the former and the latter are equal or if the former is higher than the latter (in the +Z direction), the operational space determination unit 43 determines that the user's hand is in operational space A. On the other hand, if the former is lower than the latter (in the -Z direction), the operational space determination unit 43 determines that the user's hand is in operational space B.

[0173] Next, the operational space determination unit 43 checks whether it has determined that the user's hand is present in the operational space A (step B003). If it has determined that the user's hand is present in the operational space A (step B003; YES), the operational space determination unit 43 outputs the determination result (space determination result) to the aerial image generation unit 50 (step B004). The operational space determination unit 43 also outputs the space determination result, together with the position detection result acquired from the position acquisition unit 41, to the pointer operation information output unit 44 (step B004). Thereafter, the process proceeds to step B005 (spatial processing A).

[0174] On the other hand, if it is determined in step B003 that the user's hand is not present in operation space A (step B003; NO), the operation space determination unit 43 checks whether it has determined that the user's hand is present in operation space B (step B006). If it is determined that the user's hand is present in operation space B (step B006; YES), the operation space determination unit 43 outputs the determination result (space determination result) to the aerial image generator 50 (step B007). Furthermore, the operation space determination unit 43 outputs the space determination result, together with the position detection result acquired from the position acquisition unit 41, to the pointer operation information output unit 44 and the command identification unit 46 (step B007). Thereafter, the process proceeds to step B008 (spatial processing B).

[0175] On the other hand, if it is determined in step B006 that the user's hand is not present in the operational space B (step B006; NO), the interface system 100 ends the process.

[0176] <Spatial Processing A> Next, spatial processing A in step B005 will be described with reference to the flowchart shown in FIG.

[0177] First, the aerial image generation unit 50 acquires the space determination result output from the operation space determination unit 43, indicating that the user's hand is present in the operation space A, and regenerates data indicating the aerial image S to be projected in a manner corresponding to the acquired space determination result (step C001). For example, the aerial image generation unit 50 regenerates data indicating the aerial image S to be projected in blue as the aerial image S indicating that the user's hand is present in the operation space A. The aerial image generation unit 50 outputs the data indicating the regenerated aerial image S to the aerial image projection unit 31.

[0178] Next, the aerial image projection unit 31 acquires data indicating the aerial image S regenerated by the aerial image generation unit 50, and reprojects the aerial image S based on the acquired data into the virtual space K (step C002). That is, the aerial image projection unit 31 updates the aerial image S projected into the virtual space K. As a result, for example, the color of the aerial image S changes to blue, allowing the user to easily understand that their hand has entered the operation space A (that the pointer operation mode has been entered). Note that steps C001 and C002 are not essential processes and may be omitted.

[0179] Next, the pointer operation information output unit 44 determines whether or not the user's hand has moved based on the position detection result output from the operational space determination unit 43 (step C003). As a result, if it is determined that the user's hand has not moved (step C003; NO), the process returns. On the other hand, if it is determined that the user's hand has moved (step C003; YES), the process proceeds to step C004.

[0180] In step C004, the pointer operation information output unit 44 identifies the movement of the user's hand based on the position detection result output from the operational space determination unit 43. Then, the pointer operation information output unit 44 generates information (movement control information) for moving the pointer P displayed on the operation screen R of the display 10 in accordance with the movement of the user's hand in the operational space A (step C004). Furthermore, the pointer operation information output unit 44 outputs operation information including the generated movement control information to the pointer position control unit 45 (step C005).

[0181] Next, the pointer position control unit 45 controls the pointer P based on the movement control information included in the operation information output from the pointer operation information output unit 44 (step C006). Specifically, the pointer position control unit 45 moves the pointer P on the operation screen R displayed on the display 10 in accordance with the movement of the user's hand based on the movement control information. More specifically, the pointer position control unit 45 moves the pointer P on the operation screen R displayed on the display 10 by an amount equivalent to the amount of movement of the user's hand, in other words, in a direction included in the amount of movement and by a distance included in the amount of movement. As a result, the pointer P moves in conjunction with the movement of the user's hand. Then, the process returns.

[0182] <Spatial Processing B> Next, spatial processing B in step B008 will be described with reference to the flowchart shown in FIG.

[0183] First, the aerial image generation unit 50 acquires the space determination result output from the operation space determination unit 43, indicating that the user's hand is present in the operation space B, and regenerates data indicating the aerial image S to be projected in a manner corresponding to the acquired space determination result (step D001). For example, the aerial image generation unit 50 regenerates data indicating the aerial image S to be projected in red as the aerial image S indicating that the user's hand is present in the operation space B. The aerial image generation unit 50 outputs the data indicating the regenerated aerial image S to the aerial image projection unit 31.

[0184] Next, the aerial image projection unit 31 acquires data indicating the aerial image S regenerated by the aerial image generation unit 50, and reprojects the aerial image S based on the acquired data into the virtual space K (step D002). That is, the aerial image projection unit 31 updates the aerial image S projected into the virtual space K. As a result, for example, the color of the aerial image S changes to red, allowing the user to easily understand that their hand has entered the operation space B (the command execution mode has been entered). Note that steps D001 and D002 are not essential processes and may be omitted.

[0185] Next, the pointer operation information output unit 44 generates control information (fixation control information) for fixing the pointer P displayed on the operation screen R of the display 10 (step D003).The pointer operation information output unit 44 also outputs operation information including the generated fixation control information to the pointer position control unit 45 (step D004).

[0186] Next, the pointer position control unit 45 fixes the pointer P on the operation screen R displayed on the display 10 based on the fixation control information included in the operation information output from the pointer operation information output unit 44 (step D005).

[0187] Next, the command identification unit 46 determines whether or not the user's hand has moved based on the position detection result output from the operational space determination unit 43 (step D006). As a result, if it is determined that the user's hand has not moved (step D006; NO), the process returns. On the other hand, if it is determined that the user's hand has moved (step D006; YES), the process proceeds to step D007.

[0188] In step D007, the command specifying unit 46 specifies the user's hand movement (gesture) based on the position detection result output from the operational space determining unit 43 (step D007).

[0189] Next, the command identification unit 46 refers to the command information recorded in the command recording unit 47 and determines whether the command information contains a movement corresponding to the identified hand movement (step D008). As a result, if it is determined that the command information does not contain a movement corresponding to the identified hand movement (step D008; NO), the process returns. On the other hand, if it is determined that the command information contains a movement corresponding to the identified hand movement (step D008; YES), the command identification unit 46 identifies the command associated with the movement in the command information (step D009). The command identification unit 46 outputs the identified command to the command output unit 48.

[0190] Next, the command output unit 48 outputs operation information including information indicating the command acquired from the command identification unit 46 to the command generation unit 49 (step D010).

[0191] Next, the command generating unit 49 receives the operation information output from the command output unit 48 and generates a command included in the received operation information (step D011). As a result, the interface system 100 executes a command corresponding to the user's hand movement (gesture).

[0192] Although not shown in the above flowchart, in step D009, the command identification unit 46 may output the identified command to the aerial image generation unit 50. Then, the aerial image generation unit 50 may acquire the command output from the command identification unit 46 and regenerate data representing the aerial image S to be projected in a mode corresponding to the acquired command. Furthermore, the aerial image generation unit 50 may output data representing the regenerated aerial image S to the aerial image projection unit 31.

[0193] Furthermore, the aerial image projection unit 31 may acquire data indicating the aerial image S regenerated by the aerial image generation unit 50, and reproject the aerial image S based on the acquired data into the virtual space K. That is, the aerial image projection unit 31 may update the aerial image S projected into the virtual space K. This causes the aerial image S to flash once, for example, allowing the user to easily understand that a left-click command has been executed.

[0194] Next, examples of control by the interface system 100 according to the fifth embodiment will be described with reference to Fig. 16 to Fig. 24. By operating as described above, the interface system 100 according to the fifth embodiment can perform the following control, for example.

[0195] (1) Pointer Movement When the user's hand is in the operational space A, the pointer P moves on the operation screen R of the display 10 according to the amount of movement of the user's hand in the virtual space K (XYZ coordinate system) (see FIG. 16). Note that while FIG. 16 conceptually depicts the pointer P on the operational space A, in reality, the pointer P displayed on the operation screen R of the display 10 moves.

[0196] In the above case, the pointer operation information output unit 44 may generate movement control information such that the movement amount or movement speed of the pointer P changes depending on how far the three-dimensional position of the user's hand is from the boundary surface (XY plane) of the virtual space represented by the aerial image S in a direction perpendicular to the boundary surface (i.e., the Z-axis direction), even if the amount of movement of the user's hand is the same.

[0197] For example, as shown in Fig. 17 , if the three-dimensional position of the user's hand is far from the boundary surface (XY plane) in the Z-axis direction, the pointer operation information output unit 44 may generate movement control information to move the pointer P by approximately the same distance as the distance moved by the user's hand or at approximately the same speed as the speed at which the user's hand moved (symbol W1 in Fig. 17 ). On the other hand, if the three-dimensional position of the user's hand is close to the boundary surface (XY plane) in the Z-axis direction, even if the amount of movement of the user's hand is the same as above, the pointer operation information output unit 44 may generate movement control information to move the pointer P by approximately half the distance moved by the user's hand or at approximately half the speed at which the user's hand moved (symbol W2 in Fig. 17 ).

[0198] That is, the pointer operation information output unit 44 may generate movement control information by multiplying the movement amount or movement speed of the user's hand projected onto the boundary surface (XY plane) onto which the aerial image S is projected by a coefficient corresponding to the distance in the Z-axis direction between the three-dimensional position of the user's hand and the boundary surface (XY plane).

[0199] In this case, if the user moves their hand in a position far from the boundary surface (XY plane) on which the aerial image S is projected in the Z axis direction, they can move the pointer P by an amount equivalent to the amount of hand movement or at the same speed as their hand movement. On the other hand, if the user moves their hand in a position close to the boundary surface (XY plane) on which the aerial image S is projected in the Z axis direction, they can move the pointer P finely (smaller) or slowly. In particular, when switching from the pointer movement mode to the command execution mode, it is expected that the user will move their hand near the boundary surface on which the aerial image S is projected. In this case, because the user can move the pointer P finely or slowly, they can precisely specify the position of the pointer P when executing a command, improving convenience.

[0200] Here, an example has been described in which, if the three-dimensional position of the user's hand is far away from the boundary surface (XY plane) in the Z-axis direction, the pointer operation information output unit 44 generates movement control information to move the pointer P a distance approximately equal to the distance moved by the user's hand or at a speed approximately equal to the speed at which the user's hand moved, and, if the three-dimensional position of the user's hand is close to the boundary surface (XY plane) in the Z-axis direction, the pointer operation information output unit 44 generates movement control information to move the pointer P a distance approximately half the distance moved by the user's hand or at a speed approximately half the speed at which the user's hand moved. However, the pointer operation information output unit 44 may, conversely to the above, generate movement control information to move the pointer P a distance approximately half the distance moved by the user's hand or at a speed approximately half the speed at which the user's hand moved if the three-dimensional position of the user's hand is far away from the boundary surface (XY plane) in the Z-axis direction, and may generate movement control information to move the pointer P a distance approximately equal to the distance moved by the user's hand or at a speed approximately equal to the speed at which the user's hand moved if the three-dimensional position of the user's hand is close to the boundary surface (XY plane) in the Z-axis direction.

[0201] (2) Pointer Fixation When the user's hand crosses the position of the aerial image (boundary position) from the operational space A and enters the operational space B, the pointer P is fixed on the operation screen R of the display 10 (see FIG. 18 ). Even if the user's hand moves in the operational space B after that, the pointer P remains fixed on the operation screen R of the display 10. At this time, the aerial image S may be updated, for example, the color of the aerial image S may be changed from blue to red. This allows the user to easily understand that their hand has entered the operational space B (the mode has been changed to command execution mode). At this time, the projection device 20 may project the aerial image SC at a position near the lower limit of the range detectable by the detection device 21 and approximately near the center of the virtual space K in the X-axis direction.

[0202] (3) Left Click For example, in operational space B, when the user moves his / her hand in the −Y direction and the hand reaches a predetermined left click occurrence area, the movement (gesture) of the hand is identified by the command identification unit 46. The left click occurrence area is, for example, a predetermined area to the left (−X direction side) of the aerial image SC in operational space B and on the far side (−Y direction side) as seen from the user.

[0203] This movement (gesture) is associated with a "left click" command in the command information. Therefore, the command identification unit 46 identifies the "left click" command, and the left click is executed (see FIG. 19 ). At this time, the aerial image generation unit 50 may regenerate data representing the aerial image S, which flashes once, for example, and the aerial image projection unit 31 may project the aerial image S based on the regenerated data. In this way, the aerial image S flashes once in the interface system 100, allowing the user to easily recognize that a left click has been executed. At this time, the interface system 100 may output a sound corresponding to the left click, such as a "click." By hearing this sound, the user can more easily recognize that a left click has been executed.

[0204] (4) Right-Click For example, in operational space B, when the user moves his / her hand in the −Y direction and the hand reaches a preset right-click occurrence area, the hand movement (gesture) is identified by command identification unit 46. The right-click occurrence area is, for example, a predetermined area to the right (+X direction side) of the aerial image SC in operational space B and on the far side (−Y direction side) as seen from the user.

[0205] This movement (gesture) is associated with a "right-click" command in the command information. Therefore, the command identification unit 46 identifies the "right-click" command, and a right-click is executed (see FIG. 20 ). At this time, the aerial image generation unit 50 may regenerate data representing the aerial image S that flashes once, for example, and the aerial image projection unit 31 may project the aerial image S based on the regenerated data. In this way, in the interface system 100, the aerial image S flashes once, allowing the user to easily understand that a right-click has been executed.

[0206] (5) Left Double-Click: For example, in operational space B, if a user moves their hand in the −Y direction and reaches a predetermined left-click occurrence area, and then moves their hand consecutively in the +Y and −Y directions, the command identification unit 46 identifies the hand movement (gesture). This movement (gesture) is associated with a “left double-click” command in the command information. Therefore, the command identification unit 46 identifies the “left double-click” command, and a left double-click is executed (see FIG. 21 ). At this time, the aerial image generator 50 may regenerate data representing the aerial image S, for example, flashing twice consecutively, and the aerial image projection unit 31 may project the aerial image S based on the regenerated data. In this way, the aerial image S flashes twice consecutively in the interface system 100, allowing the user to easily recognize that a left double-click has been executed. At this time, the interface system 100 may output a series of sounds, such as “click, click,” corresponding to the left double-click. By hearing this sound, the user can more easily recognize that a left double-click has been executed.

[0207] (6) Continuous pointer movement operation When the user moves their hand in the +Y direction in operational space A, the pointer P also moves in the +Y direction in conjunction with that movement (see FIG. 22A). Here, the user moves their hand once into operational space B to fix the pointer P (see FIG. 22B). In this state, if the user moves their hand in the -Y direction, the pointer P remains fixed (see FIG. 22C).

[0208] Then, when the user moves his / her hand from operational space B across the boundary position (boundary surface) to operational space A, the pointer P again moves in conjunction with the movement of the user's hand (see FIG. 22D ). By repeating the above operations, the user can continuously move the pointer P simply by moving his / her hand within the limited space of operational space A and operational space B.

[0209] In this regard, in the above-described conventional device, for example, as shown in Fig. 23A , continuous operations such as long-distance movement of the pointer P and scrolling require a large amount of movement of the user's hand, and a wide space is required to allow such large movements. In contrast, in embodiment 5, for example, as shown in Fig. 23B , the correlation between the pointer P and the user's hand can be reset by moving the user's hand back and forth across the boundary position (boundary surface). Therefore, by repeating short-distance hand movements, the user can achieve continuous operations such as long-distance movement of the pointer P and scrolling even in the limited space of the operational space A and the operational space B.

[0210] (7) Scrolling Operation When a user initiates a movement (gesture), such as rotating their hand in the operation space B without reaching the left-click or right-click occurrence area, the command identification unit 46 identifies the hand movement (gesture). This movement (gesture) is associated with a “scrolling operation” command in the command information. Therefore, in the interface system 100, the command identification unit 46 identifies the “scrolling operation” command, and the scroll operation is executed (see FIG. 24A ). At this time, the aerial image generator 50 may regenerate data representing an aerial image SE obtained by adding a predetermined graphic to the current aerial image S, and the aerial image projection unit 31 may project the aerial images S and SE based on the regenerated data (see FIG. 24B ). This allows the aerial images S and SE to which the predetermined graphic has been added to be projected, allowing the user to easily understand that the scroll operation can be executed.

[0211] Next, an example of an applied operation in the control execution phase of the interface system 100 according to the fifth embodiment will be described with reference to the flowchart shown in Fig. 25. In this example of applied operation, a case will be described in which the user operates both the operational space A and the operational space B using the left and right hands.

[0212] First, when the user places his / her hand in the virtual space K, the position detection unit 32 detects the three-dimensional position of the user's hand in the virtual space K (step E001). The position detection unit 32 outputs the detection result of the three-dimensional position of the user's hand (position detection result) to the position acquisition unit 41.

[0213] Next, the position acquisition unit 41 acquires the position detection result output from the position detection unit 32 (step E002). The position acquisition unit 41 outputs the acquired position detection result to the operational space determination unit 43.

[0214] Next, the operation space determination unit 43 acquires the detection results output from the position acquisition unit 41, and determines the operation space in which the user's hands are located based on the acquired position detection results and the boundary positions of each operation space in the virtual space K.

[0215] Next, the operational space determination unit 43 checks whether it has determined that the user's hands are present in both the operational space A and the operational space B (step E003). If it has determined that the user's hands are not present in both the operational space A and the operational space B (step E003; NO), the process proceeds to step B003 in the flowchart of FIG. 13 described above.

[0216] On the other hand, if it is determined that the user's hands are present in both operation space A and operation space B (step E003; YES), operation space determination unit 43 outputs the determination result (space determination result) to aerial image generation unit 50. Furthermore, operation space determination unit 43 outputs the space determination result, together with the position detection result acquired from position acquisition unit 41, to pointer operation information output unit 44 and command identification unit 46 (step E004). Thereafter, the process proceeds to step E005 (spatial processing AB).

[0217] <Spatial Processing AB> Next, the spatial processing AB of step E005 will be described with reference to the flowchart shown in FIG.

[0218] First, the aerial image generation unit 50 acquires the space determination result output from the operation space determination unit 43, indicating that the user's hands are present in both operation space A and operation space B, and regenerates data indicating the aerial image S to be projected in a manner corresponding to the acquired space determination result (step F001). For example, the aerial image generation unit 50 regenerates data indicating the aerial image S to be projected in green as the aerial image S indicating that the user's hands are present in both operation space A and operation space B. The aerial image generation unit 50 outputs the data indicating the regenerated aerial image S to the aerial image projection unit 31.

[0219] Next, the aerial image projection unit 31 acquires data indicating the aerial image S regenerated by the aerial image generation unit 50, and reprojects the aerial image S based on the acquired data into the virtual space K (step F002). That is, the aerial image projection unit 31 updates the aerial image S projected into the virtual space K. As a result, for example, the color of the aerial image S changes to green, allowing the user to easily understand that his or her hand has entered both the operation space A and the operation space B. Note that steps F001 and F002 are not essential processes and may be omitted.

[0220] Next, the pointer operation information output unit 44 determines whether or not the user's hand has moved based on the position detection result output from the operational space determination unit 43 (step F003). As a result, if it is determined that the user's hand has not moved (step F003; NO), the process returns. On the other hand, if it is determined that the user's hand has moved (step F003; YES), the process proceeds to step F004.

[0221] In step F004, the command identification unit 46 identifies the user's hand movement (gesture) based on the position detection result output from the operational space determination unit 43. In this case, the user's hand movement (gesture) is a combination of the hand movement in operational space A and the hand movement in operational space B.

[0222] Next, the command identification unit 46 refers to the command information recorded in the command recording unit 47 and determines whether or not the command information contains a movement corresponding to the identified hand movement (step F005). As a result, if it is determined that the command information does not contain a movement corresponding to the identified hand movement (step F005; NO), the process returns.

[0223] On the other hand, if it is determined that the command information contains a movement corresponding to the identified hand movement (step F005; YES), the command identification unit 46 identifies the command associated with the movement in the command information (step F006). The command identification unit 46 outputs the identified command to the command output unit 48.

[0224] Next, the command output unit 48 outputs the operation information including the information indicating the command acquired from the command identification unit 46 to the command generation unit 49 (step F007).

[0225] Next, the command generating unit 49 receives the operation information output from the command output unit 48 and generates a command included in the received operation information (step F008). As a result, the interface system 100 executes a command corresponding to the user's hand movement (gesture).

[0226] The interface system 100 according to the fifth embodiment operates as described above, and can thereby perform the following control, for example.

[0227] (8) Left Drag Operation The user brings his / her left hand to the left click occurrence area in operational space B and moves his / her right hand in operational space A. Then, in the interface system 100, the command identification unit 46 identifies the movement (gesture) of the left and right hands. This movement (gesture) is associated with the command "left drag operation" in the command information. Therefore, in the interface system 100, the command identification unit 46 identifies the command "left drag operation," and a left drag operation linked to the movement of the user's right hand is executed (see FIG. 27A ).

[0228] (9) Right-drag operation The user brings his / her right hand to a right-click occurrence area in operational space B and moves his / her left hand in operational space A. Then, in the interface system 100, the command identification unit 46 identifies the movement (gesture) of the left and right hands. This movement (gesture) is associated with the command "right-drag operation" in the command information. Therefore, in the interface system 100, the command identification unit 46 identifies the command "right-drag operation," and a right-drag operation linked to the movement of the user's left hand is executed (see FIG. 27B ).

[0229] In the above description, an example has been given in which the user performs a left drag operation and a right drag operation by moving their left and right hands, but this is merely an example, and commands executed by combinations of the user's left and right hand movements are not limited to the above examples. In this way, by associating combinations of the user's left and right hand movements with commands, the interface system 100 can increase the variety of commands that the user can execute.

[0230] In the above description, for ease of understanding, the operation example in spatial processing AB and the operation example in spatial processing B have been described separately, but these processes may be executed consecutively. For example, in the interface system 100, first, in spatial processing B, the pointer position control unit 45 may fix the pointer P on the operation screen R based on the fixation control information generated by the pointer operation information output unit 44, and then the above-mentioned spatial processing AB may be executed. That is, the user may, for example, place one of his or her left or right hands in operation space B to fix the pointer P on the operation screen R, and then, while maintaining this state, perform the above-mentioned left drag operation and right drag operation by moving his or her left or right hands in operation space A and operation space B. In this case, the interface system 100 executes spatial processing B and spatial processing AB consecutively. This allows the interface system 100 to achieve both accurate pointing operations by the user and an expanded variety of commands that the user can execute.

[0231] As described above, in interface system 100 according to embodiment 5, aerial image S indicating the boundary position between operational spaces A and B constituting virtual space K is projected into virtual space K. This allows the user to visually recognize the boundary position between operational spaces A and B in virtual space K, and to easily grasp the position at which the operational space (mode) switches.

[0232] In this regard, with the conventional devices described above, it is difficult for the user to visually determine at which position in the virtual plane the mode switches, in other words, the boundary positions of the spaces that make up the virtual plane (the boundary positions between the first space and the second space, and the boundary positions between the second space and the third space), and the user has to move their hand to some extent to grasp these positions. Furthermore, as a result, the user cannot grasp the correlation between the pointer and their hand unless they move their hand to some extent, and it can take a long time before they can start operating.

[0233] On the other hand, in the fifth embodiment, as described above, the user can visually recognize the boundary position between the operational space A and the operational space B in the virtual space K, and can easily grasp the boundary position at which the operational space (mode) switches. This also eliminates the need for the user to move their hand to grasp the boundary position at which the operational space switches, and allows the user to start operation more quickly than with conventional devices.

[0234] Furthermore, with conventional non-contact pointing systems, including conventional devices, it can be difficult for users to understand the location in virtual space that corresponds to pressing a button on the operation screen displayed on the display, so it may be necessary to add an auxiliary display to the operation screen. Alternatively, it may be necessary to make changes, such as increasing the size of the button on the operation screen, to ensure that the button on the operation screen is pressed reliably in response to a touch operation in virtual space. In other words, with conventional non-contact pointing systems, it may be necessary to reconfigure the software used to display the existing operation screen.

[0235] Furthermore, with conventional contactless pointing systems, even if a user holds their hand still in the air and performs a gesture such as pressing, it can be difficult to specify an accurate position on the operation screen because the pointer position may shift when the user presses. Furthermore, with conventional contactless pointing systems, operations involving continuous movement of the pointer over long distances and scrolling can require a large amount of movement of the user's hand, which can require a wide area of ​​space.

[0236] In this regard, in the fifth embodiment, as described above, the virtual space K is divided into the operational space A and the operational space B, and the pointer P is movable in the operational space A in conjunction with the user's hand movement, while the pointer P is fixed in the operational space B, and the user's hand movement (gesture) to issue a command is recognized with the pointer P fixed. This prevents the position of the pointer P from shifting while the hand movement (gesture) to issue a command is being executed. Therefore, not only can the user perform accurate pointing operations when executing a command, but the user can also operate an operation screen with small buttons designed for operating a PC mouse, for example, without any modification to the software for displaying the operation screen.

[0237] Furthermore, in the fifth embodiment, the user can operate the display device, including the pointer P, without contact, so that the user can operate the device without contact even in a work environment where hygiene is important, for example, when the user's hands are dirty or where the user does not want to get their hands dirty.

[0238] In addition, in the fifth embodiment, the user can execute commands by moving their hands regardless of the shape of their fingers, so there is no need to memorize specific finger gestures. In addition, in the fifth embodiment, the detection target of the detection device 21 is not limited to the user's hand, so if the detection target is an object other than the user's hand, the user can perform operations even when, for example, they are holding an object in their hand.

[0239] In addition, with regard to the means described in this disclosure of providing a user with an interface that gives the user the sensation of operating a mouse (a sensation similar to operating a mouse) by using an aerial image as a guide, as long as the aerial image guide can show the user the area for operation, other structures may be used as the imaging optical system that forms the aerial image, rather than relying on an imaging optical system that combines a beam splitter 202 and a retroreflective material 203.

[0240] Next, an example hardware configuration of device control device 12 included in interface system 100 according to embodiment 5 will be described with reference to Fig. 28 . The functions of position acquisition unit 41, operational space determination unit 43, pointer operation information output unit 44, command identification unit 46, command output unit 48, and aerial image generation unit 50 in device control device 12 are realized by a processing circuit. The processing circuit may be dedicated hardware as shown in Fig. 28A , or may be a CPU (also referred to as a central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 62 that executes a program stored in memory 63 as shown in Fig. 28B .

[0241] When the processing circuit is dedicated hardware, processing circuit 61 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of each of position acquisition unit 41, operational space determination unit 43, pointer operation information output unit 44, command identification unit 46, command output unit 48, and aerial image generation unit 50 may be realized individually by processing circuit 61, or the functions of each unit may be realized collectively by processing circuit 61.

[0242] When the processing circuit is a CPU 62, the functions of the position acquisition unit 41, operational space determination unit 43, pointer operation information output unit 44, command identification unit 46, command output unit 48, and aerial image generation unit 50 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in memory 63. The processing circuit realizes the functions of each unit by reading and executing the programs stored in memory 63. That is, the device control device 12 includes a memory for storing programs that, when executed by the processing circuit, result in the execution of each step shown in, for example, Figures 12 to 15 and Figures 25 to 26. Furthermore, these programs can also be said to cause a computer to execute the procedures and methods of the position acquisition unit 41, operational space determination unit 43, pointer operation information output unit 44, command identification unit 46, command output unit 48, and aerial image generation unit 50. Here, examples of the memory 63 include non-volatile or volatile semiconductor memories such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), and EEPROM (Electrically EPROM), magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs (Digital Versatile Discs).

[0243] It is also possible to realize some of the functions of the position acquisition unit 41, the operational space determination unit 43, the pointer operation information output unit 44, the command identification unit 46, the command output unit 48, and the aerial image generation unit 50 with dedicated hardware and some with software or firmware. For example, the function of the position acquisition unit 41 can be realized by a processing circuit as dedicated hardware, and the functions of the operational space determination unit 43, the pointer operation information output unit 44, the command identification unit 46, the command output unit 48, and the aerial image generation unit 50 can be realized by the processing circuit reading and executing programs stored in the memory 63.

[0244] Thus, the processing circuitry can implement each of the above-described functions by hardware, software, firmware, or a combination thereof.

[0245] In the above description, an example has been described in which the operation information output unit 51 uses at least the space determination result by the operation space determination unit 43 to output operation information for executing a predetermined operation on the display device 1. However, the operation information output unit 51 is not limited to this, and may be configured to use at least the space determination result by the operation space determination unit 43 to output operation information for executing a predetermined operation on an application displayed on the display device 1. Here, the term "application" includes an OS (Operating System) or various software programs that run on the OS.

[0246] Furthermore, operations on an application may include various touch-panel fingertip operations in addition to the mouse operations described above. In this case, each operation space may correspond to at least one of a plurality of types of operations on an application using a mouse or a touch panel. Furthermore, adjacent operation spaces among the operation spaces may be associated with different consecutive operations on the application. Note that, similar to the above-described "continuous operations," different consecutive operations on an application refer to operations that are typically expected to be performed consecutively in time, such as when a user moves a pointer P on a displayed application and then executes a predetermined command. Note that, among the operation spaces, all adjacent operation spaces may be associated with continuous operations, or continuous operations may be associated with some adjacent operation spaces. In other words, it is also possible to associate non-continuous operations with other adjacent operation spaces.

[0247] As described above, according to the fifth embodiment, the interface system 100 includes the detection unit 21 that detects the three-dimensional position of a detection target in a virtual space K divided into a plurality of operation spaces, the position acquisition unit 41 that acquires the three-dimensional position of the detection target detected by the detection unit 21, the projection unit 20 that projects an aerial image S that indicates boundary positions of each operation space in the virtual space K, the operation space determination unit 43 that determines an operation space that contains the three-dimensional position of the detection target based on the three-dimensional position of the detection target acquired by the position acquisition unit 41 and the boundary positions of each operation space in the virtual space K, and the operation information output unit 51 that outputs operation information for executing a predetermined operation on an application displayed on the display device 1, using at least the determination result by the operation space determination unit 43. Each operation space corresponds to at least one of a plurality of types of operation on the application using a mouse or a touch panel, and adjacent operation spaces among the operation spaces are associated with consecutive different operations on the application. As a result, the interface system 100 according to the fifth embodiment enables the user to visually recognize the boundary positions of the plurality of operation spaces that constitute the virtual space K that is the target of operation by the user.

[0248] Sixth Embodiment In a sixth embodiment, as another configuration example of the interface device 2, an interface device 2 that can control the spatial positional relationship of the aerial image with respect to the projection device 20 will be described.

[0249] Fig. 29 is a perspective view showing an example of the arrangement of the projection device 20 and the detection device 21 in the interface device 2 according to embodiment 6. Fig. 30 is a top view showing an example of the arrangement of the projection device 20 and the detection device 21 in the interface device 2 according to embodiment 6. Fig. 31 is a front view showing an example of the arrangement of the projection device 20 and the detection device 21 in the interface device 2 according to embodiment 6.

[0250] In the interface device 2 according to the sixth embodiment, the beam splitter 202 is divided into two beam splitters 202a and 202b, and the retroreflector 203 is divided into two retroreflectors 203a and 203b, similar to the interface device 2 according to the second embodiment shown in Fig. 6. Furthermore, in the interface device 2 according to the sixth embodiment, unlike the interface device 2 according to the second embodiment shown in Fig. 6, the light source 201 is also divided into two light sources 201a and 201b.

[0251] Furthermore, a first imaging optical system including light source 201a, beam splitter 202a, and retroreflector 203a projects an aerial image Sa into virtual space K (the space on the near side of the paper in FIG. 29 ), and a second imaging optical system including light source 201b, beam splitter 202b, and retroreflector 203b projects an aerial image Sb into virtual space K. In other words, there is a correspondence between each of the two split light sources, beam splitters, and retroreflectors, with light source 201a, beam splitter 202a, and retroreflector 203a corresponding to each other, and light source 201b, beam splitter 202b, and retroreflector 203b corresponding to each other.

[0252] The principles of projection (imaging) of the aerial image by the first imaging optical system and the second imaging optical system are the same as those in the second embodiment. For example, light (diffused light) emitted from the light source 201a is specularly reflected on the surface of the beam splitter 202a, and the reflected light enters the retroreflector 203a. The retroreflector 203a retroreflects the incident light and causes it to re-enter the beam splitter 202a. The light that enters the beam splitter 202a passes through the beam splitter 202a and reaches the user. Then, by following the above-described optical path, the light emitted from the light source 201a reconverges and rediffuses at a position that is plane-symmetrical to the light source 201a across the beam splitter 202a. This allows the user to perceive the aerial image Sa in the virtual space K.

[0253] Similarly, light (diffused light) emitted from light source 201b is specularly reflected on the surface of beam splitter 202b, and the reflected light enters retroreflector 203b. Retroreflector 203b retroreflects the incident light and causes it to enter beam splitter 202b again. The light that enters beam splitter 202b passes through beam splitter 202b and reaches the user. Then, by following the above optical path, the light emitted from light source 201b reconverges and rediffuses at a position that is plane-symmetrical to light source 201b with respect to beam splitter 202b. This allows the user to perceive aerial image Sb in virtual space K.

[0254] Also, in the interface device 2 according to the sixth embodiment, similarly to the interface device 2 according to the second embodiment, the detection device 21 may be disposed inside or outside the projection device 20. Note that Figures 29 and 30 show an example in which the detection device 21 is disposed inside the first imaging optical system and the second imaging optical system included in the projection device 20, and in particular show an example in which the detection device 21 is disposed in a region sandwiched between two light sources 201 a and 201 b and two beam splitters 202 a and 202 b.

[0255] Also, at this time, the angle of view of the detection device 21 is set to a range in which the aerial images Sa, Sb projected by the projection device 20 are not captured, as in embodiment 2, and in particular, the angle of view is set to fall within the internal area U defined by the two aerial images Sa, Sb.

[0256] In addition, light source 201a and light source 201b are arranged spatially non-parallel, and the aerial images Sa and Sb formed by the first imaging optical system and the second imaging optical system are formed so as to be in a spatially parallel relationship.

[0257] More specifically, light source 201a and light source 201b are arranged such that the spatial axes formed by the light sources are non-parallel. In the case of a bar-shaped light source, for example, the spatial axis formed by the light source is an axis passing through the centers of both end faces of the light source along the extension direction of the light source.

[0258] Although the example in which the light sources are configured in a bar shape has been described above, if the light sources are not configured in a bar shape but in a shape having a radiation surface from which light is emitted, the light sources are arranged so that the planes (radiation surfaces) in space formed by the light sources are non-parallel. In this case, the aerial images Sa and Sb are formed so as to be parallel to each other on a boundary surface, which is an arbitrary surface in the virtual space K.

[0259] The reason why light sources 201a, 201b and aerial images Sa, Sb can be arranged in this manner is as follows: In interface device 2, aerial images Sa, Sb are formed at positions that are plane-symmetrical to light sources 201a, 201b with beam splitters 202a, 202b as the spatial symmetry axis, and therefore, while separating the imaging optical systems, each imaging optical system forms an aerial image using light from a separate light source.This makes it possible to form aerial images Sa, Sb parallel to each other and at positions closer to the user, even though the optical members (light sources 201a, 201b) are arranged non-parallel.

[0260] Fig. 32 is a diagram for supplementing the positional relationship between the light sources 201a and 201b and the aerial images Sa and Sb described above. For convenience, Fig. 32 shows the cover glass 204 near the beam splitters 202a and 202b, but the cover glass 204 is omitted from other drawings. Therefore, Fig. 32 shows the cover glass 204 with a dashed line.

[0261] In the interface device 2 according to the sixth embodiment, the spatial positional relationship of the aerial images Sa and Sb relative to the projection device 20 can be controlled by changing the relative positional relationship and angle between the light source 201a and the beam splitter 202a, and between the light source 201b and the beam splitter 202b, thereby forming a boundary surface that makes it easy for the user to manipulate the space.

[0262] For example, as shown in Figure 31, by arranging two light sources 201a and 201b so that they form a V-shape when viewed from the front, the aerial images Sa and Sb are formed at an angle that makes them appear to emerge from the top to the bottom (see also Figure 29).

[0263] Furthermore, the two light sources 201 a, 201 b are configured so that their orientations can be changed when they are arranged, and by increasing the distance between the two light sources when viewed from the front (bringing the two light sources closer to horizontal), the aerial images Sa, Sb are formed so that the lower ends thereof appear to stand out more in front than the upper ends thereof. In other words, by increasing the distance between the two light sources when viewed from the front (bringing the two light sources closer to horizontal), the orientations of the aerial images Sa, Sb change, and the angle that the boundary plane onto which the aerial images Sa, Sb are projected makes with the horizontal plane also changes.

[0264] In the interface device 2, the relative positional relationship and angle between the light source 201a and the beam splitter 202a, and between the light source 201b and the beam splitter 202b, may be changed manually or automatically by control. In this case, in the interface device 2, the relative positional relationship and angle may be changed by moving the light sources 201a and 201b, or the beam splitters 202a and 202b, or the relative positional relationship and angle may be changed by moving both the light sources 201a and 201b and the beam splitters 202a and 202b.

[0265] For example, the user can manually adjust the above-described positional relationship and angle to control the spatial positional relationship between the boundary surface formed by the aerial images Sa and Sb and the projection device 20, thereby adjusting the boundary surface to one that is easy for the user to operate depending on the environment in which the interface device 2 is actually installed. Furthermore, this adjustment can be made even after the interface device 2 has been installed, which is extremely convenient for the user. For example, by allowing the user to adjust the boundary surface to one that is easy for the user to operate, operability is improved, making it easier to perform various operations (pointer movement, pointer fixation, left clicking, right clicking, etc.) as described in the fifth embodiment.

[0266] Furthermore, when the above-mentioned positional relationship and angle are automatically adjusted, the interface device 2 acquires, for example, positional information of the user and the detection target (e.g., the user's hand) using the detection device 21, and changes the above-mentioned positional relationship and angle based on the acquired information to control the position of the boundary surface formed by the aerial images Sa and Sb, thereby providing a boundary surface that is easy for each user to operate, even in an environment where an unspecified number of users operate. Furthermore, users can perform spatial operations using a boundary surface that is easy for them to operate, making it easier to perform various operations (pointer movement, pointer fixation, left click, right click, etc.) as described in embodiment 5.

[0267] In the interface device 2 according to embodiment 6, the angle of view of the detection device 21 is set within a range in which the aerial images Sa and Sb projected by the projection device 20 are not captured, thereby preventing a decrease in the resolution of the aerial images Sa and Sb.

[0268] In the above description, an example has been described in which the imaging optical system includes a beam splitter and a retroreflector, but the configuration of the imaging optical system is not limited to this. For example, the imaging optical system may include a dihedral corner reflector array element, as described in the second embodiment. In this case, in the interface device 2, the retroreflectors 203a and 203b in FIG. 29 are omitted, and the dihedral corner reflector array elements are disposed at the positions where the beam splitters 202a and 202b are disposed, respectively.

[0269] As described above, according to the sixth embodiment, the interface device 2 includes two or more light sources, and the light sources are arranged such that at least one of the axes or planes in space formed by the light sources is non-parallel, and the pair of beam splitter 202 and retroreflector 203 form real images as aerial images Sa and Sb, respectively, and the aerial images Sa and Sb are formed parallel to each other on any plane onto which the aerial images are projected in the virtual space K. Thus, the interface device 2 according to the sixth embodiment can control the spatial positional relationship of the aerial images Sa and Sb with respect to the projection device 20, in addition to the effect of the second embodiment.

[0270] Furthermore, the attitude of each light source is variable, and changing the attitude of each light source changes the attitude of each aerial image and the angle that the boundary surface onto which each aerial image is projected makes with respect to the horizontal plane. This improves the operability of the interface device 2 according to the sixth embodiment.

[0271] Seventh Embodiment In the first to sixth embodiments, the interface device 2 configured separately from the display 10 of the display device 1 has been described. In the seventh embodiment, an interface device 2 integrated with the display 10 of the display device 1 will be described.

[0272] Fig. 33 is a perspective view showing an example of the configuration of the interface device 2 according to embodiment 7, and is a perspective view showing an example of the arrangement of the display 10 and the interface device 2 (projection device 20 and detection device 21). Fig. 34 is a side view showing an example of the configuration of the interface device 2 according to embodiment 7, and is a side view showing an example of the arrangement of the display 10 and the interface device 2 (projection device 20 and detection device 21).

[0273] The display 10 in the seventh embodiment is a device for displaying digital video signals, such as a liquid crystal display or a plasma display, as in the first embodiment. In the interface device 2 according to the seventh embodiment, the display 10, the projection device 20, and the detection device 21 are fixed together so as to be integrated. The display 10, the projection device 20, and the detection device 21 can be integrated together in various ways. As an example, the projection device 20 and the detection device 21 may be integrated by mounting them on the display 10 using a fixing jig that complies with the VESA (Video Electronics Standards Association) standard and that is attached to the display 10.

[0274] 33, for example, detection device 21 is disposed near the approximate center in the width direction (left-right direction) of display 10. Similarly to the second embodiment, projection device 20 includes light source 201, two beam splitters 202a and 202b, and two retroreflectors 203a and 203b, and is disposed from the front to the rear (from the front side to the rear side) of the lower part of display 10, as shown in FIGS.

[0275] In this case, the corresponding beam splitter 202a and retroreflector 203a are arranged at the bottom of the display 10 to the left of the detection device 21 in the width direction (left-right direction) of the display 10, as shown in Fig. 33, for example, and the corresponding beam splitter 202b and retroreflector 203b are arranged at the bottom of the display 10 to the right of the detection device 21 in the width direction (left-right direction) of the display 10. Furthermore, the light source 201 is arranged more rearward than the beam splitters 202a and 202b and the retroreflectors 203a and 203b within the housing of the projection device 20, as shown in Fig. 34, for example. As a result, the aerial image Sa is projected in a planar manner into the space to the left of the detection device 21 in the width direction (left-right direction) of the display 10, and the aerial image Sb is projected in a planar manner into the space to the right of the detection device 21 in the width direction (left-right direction) of the display 10. In this case, the two aerial images Sa and Sb are contained within the same plane in space, and the plane containing these aerial images Sa and Sb indicates the boundary position (boundary plane) of each operation space in the virtual space K.

[0276] Furthermore, in this case, the larger the space between the light source 201 and the beam splitters 202a and 202b, the longer the imaging distance from the projection device 20 to the aerial images Sa and Sb. Therefore, in the projection device 20, a convex lens may be disposed between the light source 201 and the beam splitters 202a and 202b to increase the imaging distance from the projection device 20 to the aerial images Sa and Sb. Furthermore, by disposing a mirror surface between the light source 201 and the beam splitters 202a and 202b, the linear optical path can be bent, making it possible to change the shape of the housing of the projection device 20 and improving the versatility of spatial installation of the projection device 20.

[0277] The aerial images Sa, Sb projected by the projection device 20 are viewed by the user together with the video information displayed on the display 10. On the other hand, unless the beam splitters 202a, 202b are installed in the direction behind the aerial images Sa, Sb on the light beam that allows the aerial images Sa, Sb to be viewed from the user's viewpoint, the user will not be able to view the aerial images Sa, Sb. Therefore, in order for the user to view the aerial images Sa, Sb and the video information obtained from the display 10 within the same field of view, it is necessary to adjust the arrangement of the projection device 20 and its internal structure.

[0278] For example, in the interface device 2, by changing the angle (symbol α shown in Figure 34) between the entire projection device 20 and the display 10 when the interface device 2 is viewed from the side, adjustment can be made so that the beam splitters 202a and 202b are positioned behind the aerial images Sa and Sb on the light ray that makes the aerial images Sa and Sb visible from the user's viewpoint, thereby allowing the user to view the video information from the display 10 and the aerial images Sa and Sb within the same field of view.

[0279] Furthermore, in the interface device 2, by changing the distance between the light source 201 and the beam splitters 202a, 202b or the arrangement angle of the beam splitters 202a, 202b and changing the imaging positions of the aerial images Sa, Sb, the beam splitters 202a, 202b can be adjusted so that they are positioned behind the aerial images Sa, Sb on the light beam that allows the aerial images Sa, Sb to be seen from the user's viewpoint, thereby enabling the user to see the video information from the display 10 and the aerial images Sa, Sb within the same field of view.

[0280] The function of adjusting the imaging positions of the above-mentioned aerial images Sa and Sb may be realized, for example, by manually adjusting the mechanical fixed positions of the components of the projection device 20 (such as the light source 201 and the beam splitter 202), or by implementing a control mechanism such as a stepping motor in the fixing jig of the above-mentioned components and electronically controlling the fixed positions of the components.

[0281] Furthermore, in the latter case where the fixed positions of the components are electronically controlled, the interface device 2 may be provided with a control unit (not shown) that acquires information indicating the user's viewpoint position from the detection results by the detection device 21 and prior parameter information, and automatically adjusts the fixed positions of the components using the acquired information.

[0282] Furthermore, the control unit may appropriately adjust the fixed positions of the components to change not only the image formation positions of the aerial images Sa, Sb but also the angle at which the boundary plane indicated by the aerial images Sa, Sb spatially intersects with the display surface of display 10. For example, the control unit may appropriately adjust the fixed positions of the components to bring the boundary plane indicated by the aerial images Sa, Sb closer to horizontal, and the angle at which the boundary plane spatially intersects with the display surface of display 10 closer to vertical (90 degrees).

[0283] Conversely, the control unit may adjust the fixed positions of the above-mentioned components appropriately to bring the boundary plane indicated by the aerial images Sa, Sb closer to perpendicularity and bring the spatial intersection angle between the boundary plane and the display surface of display 10 closer to parallel (0 degrees). This enables the interface device 2 to control the spatial positional relationship of the aerial images Sa, Sb with respect to the display surface of display 10, and to provide a boundary plane that is easy for the user to operate.

[0284] In the interface device 2 according to embodiment 7, the angle of view of the detection device 21 is set within a range in which the aerial images Sa and Sb projected by the projection device 20 are not captured, thereby preventing a decrease in the resolution of the aerial images Sa and Sb.

[0285] In the above description, an example has been described in which the imaging optical system includes beam splitters 202a and 202b and retroreflectors 203a and 203b, but the configuration of the imaging optical system is not limited to this. For example, the imaging optical system may include a dihedral corner reflector array element, as described in the second embodiment. In this case, in the interface device 2, the retroreflector 203a in FIG. 34 is omitted, and a dihedral corner reflector array element is placed in the position where the beam splitter 202a is placed.

[0286] As described above, in the interface device 2 according to the seventh embodiment, the projection device 20, the detection device 21, and the display 10 are integrated into one unit. This allows the user to view the video information from the display 10 and the aerial images Sa and Sb projected by the projection device 20 within the same field of view. This arrangement has the advantage that, when the user performs spatial operations using the interface device 2, even if the user focuses on only one of the visual feedback information for the spatial operations or the visual information displayed on the display 10, the other visual information can be viewed. Furthermore, for a user experiencing new spatial operations, the possibility of overlooking visual information can be reduced, improving the user's acceptance of the spatial operations and enabling the user to intuitively and quickly understand the spatial operations.

[0287] In the above description, an example has been described in which interface device 2 is provided with the above configuration, but the above configuration may also be provided in interface system 100 described in embodiment 5. In this case, the user of interface system 100 can also view the video information from display 10 and the aerial images Sa, Sb projected by projection device 20 within the same field of view, and can control the spatial positional relationship of the aerial images Sa, Sb with respect to the display surface of display 10, thereby obtaining a boundary surface that is easy for the user to operate.

[0288] As described above, according to the seventh embodiment, the interface device 2 is integrally provided with the display 10 that displays video information, and the aerial images Sa, Sb projected by the projection unit 20 can be viewed by the user together with the video information displayed on the display 10. Thus, in addition to the effects of the first embodiment, the interface device 2 according to the seventh embodiment can reduce the possibility that the user will overlook the visual feedback information and video information in response to a spatial operation.

[0289] The interface device 2 also includes a control unit that changes the angle at which a boundary surface, which is a surface onto which the aerial images Sa and Sb are projected in the virtual space K, intersects with the display surface of the display 10. This enables the interface device 2 according to the seventh embodiment to control the spatial positional relationship of the aerial images Sa and Sb with respect to the display surface of the display 10, and can provide a boundary surface that is easy for the user to operate.

[0290] Furthermore, according to the seventh embodiment, the interface system 100 includes a detection unit 21 that detects the three-dimensional position of a detection target in a virtual space K, a projection unit 20 that projects an aerial image into the virtual space K, and a display 10 that displays video information, the virtual space K being divided into a plurality of operation spaces, each of which defines an operation that the user can perform when the three-dimensional position of the detection target detected by the detection unit 21 is contained therein, the aerial image projected by the projection unit 20 indicating the boundary position of each operation space in the virtual space K, and the aerial image projected by the projection unit 20 being visible to the user together with the video information displayed on the display 10. As a result, the interface system 100 according to the seventh embodiment can reduce the possibility that the user will overlook visual feedback information and video information in response to a spatial operation, in addition to the effects of the fifth embodiment.

[0291] Furthermore, interface system 100 includes a control unit that changes the angle at which a boundary surface, which is a surface onto which the aerial image is projected in virtual space K, intersects with the display surface of display 10. This enables interface system 100 according to the seventh embodiment to control the spatial positional relationship of aerial images Sa, Sb with respect to the display surface of display 10, thereby providing a boundary surface that is easy for the user to operate.

[0292] Eighth Embodiment The above description has been given of the interface device 2 or the interface system 100 that indicates the boundary positions of each operation space in the virtual space K by an aerial image projected by the projection unit 20. In the eighth embodiment, an interface device 2 or an interface system 100 that can indicate the boundary positions of each operation space by something other than an aerial image will be described.

[0293] For example, an interface device 2 according to an eighth embodiment is configured as follows: The interface device 2 enables the execution of operations of an application displayed on a display, and includes: a detection unit 21 that detects the three-dimensional position of a detection target in a virtual space K divided into a plurality of operation spaces; at least one boundary definition unit (not shown) consisting of a line or a surface that indicates the boundary of each operation space; and a boundary display unit (not shown) that sets at least one visible boundary of each operation space consisting of a point, a line, or a surface, wherein, when the three-dimensional position of the detection target detected by the detection unit 21 is included in the virtual space K, the interface device 2 enables the detection target to execute a plurality of types of operations on the applications associated with each operation space.

[0294] The boundary definition unit defines the boundaries of the virtual space K and each operation space, which are interfaces provided by the interface device 2 or the interface system 100 to allow the user to operate an application. By defining each boundary and determining various user operations after defining each boundary, software control is enabled to link user operations with application operations. In other words, since the interface device 2 or the interface system 100 defines the boundaries of the virtual space K and each operation space, it is possible to detect a detection target present in the virtual space K and the position or movement of the detection target in association with each operation space, or detect the movement of a detection target across each operation space or out of the virtual space K, and associate and link various user operation information acquired by the detection target with the application operation desired by the user. The boundary display unit arranges a display that allows a user operating an application to visually recognize the boundaries defined in the virtual space K and each operation space, which the interface device 2 or the interface system 100 provides as an interface means to the user. Specifically, as shown in Fig. 35, for example, one or more marks indicating the boundary positions of each operation space may be placed on a support that indicates the upper and lower ranges of the virtual space K, or an aerial image indicating each boundary between the virtual space K and each operation space may be displayed in space. The mark indicating the boundary position may be, for example, a colored mark, an LED, or an uneven mark arranged as a dot or a line. Furthermore, one or more marks indicating the boundary may be arranged on the same boundary, or may be in the form of a dot or a line, so that the user can recognize each boundary between the virtual space K and each operation space.

[0295] That is, in the explanation up to this point, the interface device 2 or the interface system 100 has been described, which indicates the boundary positions of each operation space in the virtual space K mainly by an aerial image projected by the projection unit 20. However, as long as the user can visually recognize the boundary positions of each operation space, the interface device 2 or the interface system 100 does not necessarily have to project an aerial image. Therefore, in the eighth embodiment, the interface device 2 or the interface system 100 provides at least one visible boundary of each operation space consisting of a point, a line, or a surface, rather than an aerial image. Even in this case, the user can visually recognize the boundary positions of the multiple operation spaces that make up the virtual space K that is the operation target.

[0296] In the eighth embodiment, the boundary display unit may be configured by a projection unit 20 that projects an aerial image into the virtual space K. In this case, the aerial image projected by the projection unit 20 indicates the boundary positions of the operation spaces in the virtual space K, and the aerial image projected by the projection unit 20 may be visible to the user together with the video information displayed on the display 10. In this case, the configuration is substantially the same as that of the interface device 2 according to the seventh embodiment described above.

[0297] For example, displaying an aerial image to indicate the boundary of each operational space rather than displaying an object other than the aerial image has the advantage that there is no problem of placing a displayed object near the operational space that forms the field of the interface (gesture), and the displayed object is less likely to obstruct the user's actions, etc. Therefore, if one wishes to actively enjoy these advantages, it is desirable to configure the boundary display unit with the projection unit 20 that projects an aerial image into the virtual space K, as described above.

[0298] As described above, according to the eighth embodiment, the interface device 2 is an interface device 2 that enables the user to operate an application displayed on a display, and includes a detection unit 21 that detects the three-dimensional position of a detection target in a virtual space K divided into a plurality of operation spaces, at least one boundary definition unit consisting of a line or a surface that indicates the boundary of each operation space, and a boundary display unit that sets at least one visible boundary of each operation space consisting of a point, a line, or a surface, and when the three-dimensional position of the detection target detected by the detection unit 21 is contained in the virtual space K, the interface device 2 enables the user to perform a plurality of types of operations on the application associated with each operation space. This makes it possible for the interface device 2 according to the eighth embodiment to visually recognize the boundary positions of the plurality of operation spaces that constitute the virtual space that is the target of operation by the user.

[0299] The boundary display unit is a projection unit 20 that projects an aerial image into the virtual space K, and the aerial image projected by the projection unit 20 indicates the boundary positions of each operation space in the virtual space K. The aerial image projected by the projection unit 20 can be viewed by the user together with the video information displayed on the display 10. As a result, in the interface device 2 according to the eighth embodiment, there is no problem in arranging a display object near the operation space that forms the field of the interface (gesture), and the display object is less likely to obstruct the user's actions.

[0300] Regarding the correspondence between the boundary display unit and boundary definition unit in the eighth embodiment and the respective functional units described in the other embodiments, the boundary display unit in the eighth embodiment corresponds to, for example, the projection device (projection unit) 20 described in the first embodiment, etc. The boundary definition unit in the eighth embodiment corresponds to, for example, the position acquisition unit 41, the operation space determination unit 43, the pointer position control unit 45, the command generation unit 49, and the operation information output unit 51 described in the fifth embodiment.

[0301] In addition, the present disclosure allows for free combination of the embodiments, modification of any of the components of the embodiments, or omission of any of the components of the embodiments.

[0302] For example, in embodiments 1 to 4, 6, and 7, examples have been described in which the angle of view of the detection unit 21 is set to a range in which the aerial images Sa and Sb indicating the boundary positions between operation spaces A and B in virtual space K are not captured. However, as also described in embodiment 1, when an aerial image that does not indicate the boundary positions of each operation space in virtual space K is projected into virtual space K, it is not necessarily necessary to prevent this aerial image from being captured into the angle of view of the detection unit 21.

[0303] For example, in the operational space B, an aerial image SC (see FIG. 3 ) indicating the lower limit position of the range detectable by the detection unit 21 may be projected by the projection unit 20. Note that this aerial image SC is projected near the center position in the X-axis direction of the operational space B, and indicates the lower limit position, and may also serve as a reference for specifying left and right when the user moves their hand in the operational space B in a motion corresponding to a command that requires specification of left and right, such as a left click and a right click. Such an aerial image SC does not indicate the boundary position of each operational space in the virtual space K, and therefore does not necessarily need to be prevented from being captured by the angle of view of the detection device 21.

[0304] Furthermore, the projection device 20 may change the projection mode of the aerial image projected into the virtual space K in accordance with at least one of the operation space containing the three-dimensional position of the detection target (e.g., the user's hand) detected by the detection device 21 and the movement of the detection target in the operation space containing the three-dimensional position of the detection target. In this case, the projection device 20 may change the projection mode of the aerial image projected into the virtual space K in pixel units of the aerial image.

[0305] For example, the projection device 20 may change the color or brightness of the aerial image projected into the virtual space K depending on whether the operation space containing the three-dimensional position of the detection target detected by the detection device 21 is operation space A or operation space B. In addition, at this time, the projection device 20 may change the color or brightness of the entire aerial image (all pixels of the aerial image) in the same manner, or may change the color or brightness of any part of the aerial image (any part of the pixels of the aerial image). Note that by changing the color or brightness of any part of the aerial image, the projection device 20 can increase the variety of projection modes of the aerial image, for example, by adding any gradation to the aerial image.

[0306] Furthermore, projection device 20 may blink the aerial image projected into virtual space K an arbitrary number of times depending on whether the operation space containing the three-dimensional position of the detection target detected by detection device 21 is operation space A or operation space B. In addition, at this time, projection device 20 may blink the entire aerial image (all pixels of the aerial image) in the same manner, or may blink an arbitrary part of the aerial image (an arbitrary part of pixels of the aerial image). By changing the projection mode as described above, the user can easily understand which operation space contains the three-dimensional position of the detection target.

[0307] Furthermore, for example, projection device 20 may change the color or brightness of the aerial image projected into virtual space K or may flash the aerial image any number of times in accordance with the movement (gesture) of the detection target in operational space B. Also in this case, projection device 20 may change or flash the color or brightness of the entire aerial image (all pixels of the aerial image) in the same manner, or may change or flash the color or brightness of any part of the aerial image (any part of the pixels of the aerial image). This allows the user to easily grasp the movement (gesture) of the detection target in operational space B.

[0308] Furthermore, the "change in the projection mode of the aerial image" here also includes the projection of the aerial image SC indicating the lower limit position of the detectable range by the detection device 21, as described above. That is, when the operation space B is the operation space that contains the three-dimensional position of the detection target detected by the detection device 21, the projection device 20 may project the aerial image SC indicating the lower limit position of the detectable range by the detection device 21, as an example of the change in the projection mode of the aerial image. Furthermore, as described above, the aerial image SC indicating the lower limit position of the detectable range may be projected within the angle of view of the detection device 21. This allows the user to easily grasp how far they can lower their hand in the operation space B and to execute commands that require designation of left or right.

[0309] According to the present disclosure, the operation information output unit 51 of the interface system 100 or the interface device 2 converts information indicating the detection result of the three-dimensional position of the detection target in the virtual space K acquired by the position acquisition unit 41 (i.e., information on the three-dimensional position of the detection target) into information on the movement of the detection target. The operation information output unit 51 then identifies the movement of the detection target within or across each operation space configured in the virtual space K as, for example, pointer operation input information in operation space A and command execution input information in operation space B. The contents of input operations such as pointer operation and command execution (also referred to as "gestures" or "gesture operations") are predetermined for multiple operation spaces within the virtual space K. The operation information output unit 51 determines whether the movement of the detection target within or across each operation space corresponds to a predetermined input operation, and links a predetermined operation of an application displayed on the display device 1 to the movement of the detection target determined to correspond to the predetermined input operation. In other words, a predetermined operation of the application can be executed in linkage with the movement of the detection target in the virtual space K.

[0310] As described above, according to the technology of the present disclosure, a user can operate an application displayed on the display device 1 in a non-contact manner, without using an operation device such as a mouse or a touch panel. This reduces various constraints that a user may have when operating an application. These constraints may include, for example, the space (width or height) of the stand on which the operation device is installed, the required shape of the operation device itself, the function of connecting the operation device to the display device 1, and a situation or condition that makes it difficult for the user to touch the operation device and operate it.

[0311] In this way, the interface system 100 or the interface device 2 converts the user's movements in the virtual space K into information for operating an application, so that, for example, the user can operate the application contactlessly via the virtual space K provided by the interface system 100 or the interface device 2 without having to change the program or execution environment of an application currently in operation (running) on ​​an existing display device 1.

[0312] The present disclosure makes it possible to visually recognize the boundary positions of multiple operation spaces that make up a virtual space that is the target of operation by a user, and is suitable for use in interface devices and interface systems.

[0313] 1 Display device, 2 Interface device, 10 Display, 11 Display control device, 20 Projection device (projection unit), 21 Detection device (detection unit), 21a Detection device, 21b Detection device, 21c Detection device, 31 Aerial image projection unit, 32 Position detection unit, 41 Position acquisition unit (acquisition unit), 42 Boundary position recording unit, 43 Operation space determination unit (determination unit), 44 Pointer operation information output unit, 45 Pointer position control unit, 46 Command identification unit, 47 Command recording unit, 48 Command output unit, 49 Command generation unit, 50 Aerial image generation unit, 51 Operation information output unit, 100 Interface system, 201 Light source, 201a Light source, 201b Light source, 202 Beam splitter, 202a Beam splitter, 202b Beam splitter, 203 Retroreflector, 203a Retroreflector, 203b Retroreflective material, 503 real image, 600 image display device, 604 display device, 605 light irradiator, 606 image capture device, 612 wavelength-selective reflecting member, 701 half mirror, 702 retroreflective sheet, A operation space, B operation space, K virtual space, P pointer, R operation screen, S aerial image, Sa aerial image, Sb aerial image, SC aerial image, U internal area.

Claims

1. A detection unit that detects a three-dimensional position of a detection target in a virtual space; a projection unit that projects an aerial image into the virtual space; Equipped with The virtual space is composed of a plurality of operation spaces, An operation that can be performed by a user when the three-dimensional position of the detection target detected by the detection unit is included in each of the operational spaces is defined, a predetermined pointer movement operation for display information of a display device linked with a movement of the detection target within at least one of the plurality of operational spaces is defined; The aerial image projected by the projection unit indicates the boundary positions of the operation spaces in the virtual space.

1. An interface device comprising:

2. The operation includes at least one of a predetermined pointer movement operation on the display device, an operation for executing a predetermined command, or an operation performed using a mouse or a touch panel.

2. The interface device according to claim 1.

3. The aerial image projected into the virtual space is The image is formed at a position that suppresses a decrease in the detection accuracy of the three-dimensional position of the detection target by the detection unit.

2. The interface device according to claim 1.

4. The projection unit is The aerial image is formed in the virtual space so that the aerial image includes an angle of view of the detection unit.

2. The interface device according to claim 1.

5. The projection unit is A projection mode of the aerial image projected into the virtual space is changed in accordance with at least one of an operation space including the three-dimensional position of the detection target detected by the detection unit and a movement of the detection target in the operation space including the three-dimensional position of the detection target.

2. The interface device according to claim 1.

6. One or more of the aerial images are projected into the virtual space, and at least one of the aerial images indicates an outer frame or an outer surface of the virtual space to the user.

2. The interface device according to claim 1.

7. The projection unit is The present invention includes one or more imaging optical systems that form a real image from light emitted from a light source, the imaging optical system forming the real image from the light source as the aerial image.

2. The interface device according to claim 1.

8. The projection unit, The imaging optical system includes two or more imaging optical systems that form light emitted from the light source into two or more real images through different optical paths, The two or more imaging optical systems form the real image into two or more aerial images each of which indicates a boundary position of the operation space as a different point, line, or surface.

8. The interface device according to claim 7.

9. The projection unit, The apparatus includes two or more light sources for forming the two or more aerial images.

8. The interface device according to claim 7.

10. The imaging optical system includes: a beam splitter having a light bending surface that forms a plane along which an optical path of light emitted from the light source bends, and that separates the light emitted from the light source into transmitted light and reflected light; A retroreflective material that reflects the reflected light from the beam splitter in the direction of incidence when the reflected light is incident on the beam splitter.

8. The interface device according to claim 7.

11. The imaging optical system comprises: The optical system includes two or more pairs of the beam splitter and the retroreflector, Each pair of the beam splitter and the retroreflector forms two or more aerial images, each of which indicates a boundary of the operation space, using light emitted from the light source as a different point, line, or surface.

11. The interface device according to claim 10.

12. The imaging optical system has a ray bending surface constituting a plane where the optical path of the light emitted from the light source is bent, and forms a real image by the light source arranged on one side of the ray bending surface as the aerial image on the opposite side of the ray bending surface.

8. The interface device according to claim 7.

13. The imaging optical system has a ray bending surface that constitutes a plane in which the optical path of the light emitted from the light source is bent, The detection unit is An internal region of the imaging optical system, the imaging optical system having the light bending surface, the imaging optical system having the light bending surface, and the light bending surface are disposed on one side of the light bending surface.

8. The interface device according to claim 7.

14. The aerial image projected by the projection unit is viewable by a user together with the image information displayed on the display device.

2. The interface device according to claim 1.

15. A boundary surface that is a surface onto which the aerial image is projected in the virtual space; A control unit is provided for changing the angle at which the display surface of the display device intersects with the display device or for setting the angle at which the display surface does not intersect with the display device.

2. The interface device according to claim 1.

16. The light source includes two or more light sources, The light sources are arranged such that at least one of axes or planes in space formed by the light sources is non-parallel, and a real image is formed as the aerial image by a pair of a beam splitter and a retroreflector, The aerial images are formed parallel to each other on any plane onto which the aerial images are projected in the virtual space.

8. The interface device according to claim 7.

17. Two or more of the light sources are provided, Each of the light sources has a variable attitude, By changing the attitude of each of the light sources, the attitude of each of the aerial images changes, and the angle that the boundary surface onto which each of the aerial images is projected makes with respect to the horizontal plane also changes.

8. The interface device according to claim 7.

18. The two or more light sources are provided with a variable position or attitude, By changing the position or orientation of each of the light sources, the positional relationship or orientation between each of the formed aerial images and the user can be changed.

20. The interface device according to claim 17,

19. An interface device that enables an operation of an application displayed on a display device to be executed, comprising: A detection unit that detects a three-dimensional position of a detection target in a virtual space constituted by a plurality of operation spaces; A detection unit that detects a three-dimensional position of a detection target in a virtual space constituted by a plurality of operation spaces; At least one boundary definition portion formed of a line or a surface indicating a boundary position of each of the operational spaces; A boundary display unit that provides at least one visible boundary of each of the operational spaces, the boundary display unit being a point, a line, or a surface; Equipped with When a three-dimensional position of the detection target detected by the detection unit is included in the virtual space, a plurality of types of operations for the application associated with each of the operation spaces can be executed on the detection target; A predetermined pointer movement operation for display information on the display device linked with the movement of the detection target within at least one of the plurality of operational spaces is defined.

1. An interface device comprising:

20. The boundary display unit is a projection unit that projects an aerial image into the virtual space, a boundary position of each of the operation spaces in the virtual space is indicated by the aerial image projected by the projection unit, The aerial image projected by the projection unit is viewable by a user together with the image information displayed on the display device.

20. An interface device according to claim 19.

21. The projection unit is capable of projecting the aerial image so as to indicate only a boundary position of the operational space.

2. The interface device according to claim 1.

22. The boundary display unit is a projection unit that projects an aerial image into the virtual space, The projection unit is capable of projecting the aerial image so as to indicate only a boundary position of the operational space.

20. An interface device according to claim 19.

23. The operations having continuity with each other are associated with adjacent operation spaces among the operation spaces, The adjacent operation spaces are simultaneously recognizable to the user.

23. An interface device according to claim 1 or any one of claims 19 to 22.

24. A detection unit that detects a three-dimensional position of a detection target in a virtual space; a projection unit that projects an aerial image into the virtual space; A display device for displaying video information; Equipped with the virtual space is configured as a plurality of operation spaces, and operations that can be performed by a user are defined in the plurality of operation spaces when the three-dimensional position of the detection target detected by the detection unit is included in the plurality of operation spaces; a boundary position of the operational space is indicated by the aerial image projected by the projection unit, The aerial image projected by the projection unit is viewable by a user together with the image information displayed on the display device.

1. An interface system comprising:

25. a boundary surface that is a surface onto which the aerial image is projected in the virtual space; A control unit is provided for changing an angle at which the display surface of the display device intersects with the display device.

25. The interface system of claim 24.

26. A detection unit that detects a three-dimensional position of a detection target in a virtual space constituted by a plurality of operation spaces; an acquisition unit that acquires a three-dimensional position of the detection target detected by the detection unit; a projection unit that projects an aerial image indicating a boundary position of the operational space; a determination unit that determines an operation space that includes the three-dimensional position of the detection target based on the three-dimensional position of the detection target acquired by the acquisition unit and boundary positions of each of the operation spaces in the virtual space; an operation information output unit that uses at least a result of the determination by the determination unit to output operation information for executing a predetermined operation on an application displayed on a display device; Equipped with Each of the operational spaces corresponds to at least one of a plurality of types of operations performed on the application using a mouse or a touch panel; Adjacent operation spaces among the operation spaces are associated with successive different operations on the application.

1. An interface system comprising:

27. A detection unit that detects a three-dimensional position of a detection target in a virtual space constituted by a plurality of operation spaces; an acquisition unit that acquires a three-dimensional position of the detection target detected by the detection unit; a projection unit that projects an aerial image indicating a boundary position of the operational space; a determination unit that determines an operation space that includes the three-dimensional position of the detection target based on the three-dimensional position of the detection target acquired by the acquisition unit and boundary positions of each of the operation spaces in the virtual space; an operation information output unit that uses at least a result of the determination by the determination unit to output operation information for executing a predetermined operation on an application displayed on a display device; Equipped with The operation information output unit Identifying a movement of the detection target based on a three-dimensional position of the detection target; A movement of the detection target within each of the operational spaces or across each of the operational spaces is associated with at least one of a plurality of types of operations on the application using a mouse or a touch panel, and a predetermined operation on the application is linked to the movement of the detection target.

1. An interface system comprising:

28. In at least one of the plurality of operational spaces, a predetermined pointer movement operation for display information of the display device linked with the movement of the detection target within the operational space is defined.

28. An interface system according to any one of claims 24 to 27.