Interface system, control device, and operation assistance method

Abstract

An interface system (100) is provided with: a detection unit (21) that detects the three-dimensional position of a detection target in a virtual space (K) divided into a plurality of operation spaces; a position acquisition unit (41) that acquires the three-dimensional position of the detection target detected by the detection unit; a projection unit (20) that projects an aerial image (S) indicating the boundary position of each operation space in the virtual space; an operation space determination unit (43) that determines an operation space including the three-dimensional position of the detection object on the basis of the three-dimensional position of the detection object acquired by the position acquisition unit and the boundary position of each operation space in the virtual space; and an operation information output unit (51) that outputs operation information for executing a predetermined operation on the display device (1) using at least the determination result of the operation space determination unit.

Description

Technical Field

[0001] The present disclosure relates to an interface system, a control device, and an operation assistance method. Background Art

[0002] Conventionally, the following technique has been proposed as an operation input technique for an electronic device or the like: a user operates a virtual space set in space, and thus operation input can be performed in a non-contact manner. In relation to such a technique, Patent Document 1 discloses a display device having a function of controlling an operation input based on a remote operation of a user on a display screen.

[0003] The display device includes two cameras that photograph a range of a user including an audiovisual display screen, and based on the captured images of the cameras, detects a second point representing a user reference position, a third point representing a position of the user's hand / finger, relative to a first point representing a reference position of the camera. In space, a virtual plane space is set at a position of a specified length in a first direction from the second point, and based on the degree of entry of the user's hand / finger into the virtual plane space, a specified operation of the user is determined and detected. Then, the display device generates operation input information based on the determination and detection results, and controls the operation of the display device based on the generated information.

[0004] Here, the virtual plane space is a space that has no physical entity and is set as a three-dimensional space position coordinate by calculation by a processor or the like of the display device. The virtual plane space is configured as a substantially rectangular parallelepiped or flat plate-shaped space sandwiched by two virtual planes. The two virtual planes refer to a first virtual plane located on the near front side close to the user and a second virtual plane located inside thereof.

[0005] For example, when the point of the hand / finger position reaches the first virtual plane from a first space closer to the front than the first virtual plane and then enters the second space inside the first virtual plane, the display device automatically shifts to a state of accepting a specified operation and displays a cursor on the display screen. Further, when the point of the hand / finger position passes through the second space, reaches the second virtual plane, and then enters the third space inside the second virtual plane, the display device determines and detects a specified operation (for example, touch, click, slide, and zoom on the second virtual plane). When the display device detects a specified operation, it controls the operation of the display device including the display control of the GUI including the display screen based on the position coordinates of the point of the detected hand / finger position and the operation information representing the specified operation.

[0006] Prior Art Documents

[0007] Patent Documents

[0008] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2021-15637 Summary of the Invention

[0009] Problems to be Solved by the Invention

[0010] In the display device described in the above Patent Document 1 (hereinafter, also referred to as the "conventional device"), the mode for accepting a specified operation and the mode for determining and detecting a specified operation are switched according to the points of the user's hand / finger positions in the virtual plane space. However, in the above conventional device, it is difficult for the user to visually recognize at which position in the virtual plane space the above respective modes are switched. In other words, it is difficult to visually recognize the boundary positions of the respective spaces constituting the virtual plane space (the boundary position between the first space and the second space, and the boundary position between the second space and the third space).

[0011] The present disclosure has been completed to solve the above-described problems, and an object thereof is to provide a technique capable of visually recognizing the boundary positions of a plurality of operation spaces constituting a virtual space that is an operation object of a user.

[0012] Means for Solving the Problems

[0013] The interface system of the present disclosure is characterized by including: a detection unit that detects the three-dimensional position of a detection object in a virtual space divided into a plurality of operation spaces; an acquisition unit that acquires the three-dimensional position of the detection object detected by the detection unit; a projection unit that projects an aerial image indicating the boundary positions of the respective operation spaces in the virtual space; a determination unit that determines the operation space including the three-dimensional position of the detection object based on the three-dimensional position of the detection object acquired by the acquisition unit and the boundary positions of the respective operation spaces in the virtual space; and an operation information output unit that outputs operation information for executing a specified operation on the display device using at least the determination result of the determination unit.

[0014] Advantages of the Invention

[0015] According to the present disclosure, since it is configured as described above, it is possible to visually recognize the boundary positions of a plurality of operation spaces constituting a virtual space that is an operation object of a user. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Figure 1 A is a perspective view showing a structural example of the interface system according to Embodiment 1, Figure 1 B is a side view showing a structural example of the interface system according to Embodiment 1.

[0017] Figure 2 Figure 2 A is a perspective view showing a structural example of the projection device in Embodiment 1, Figure 2 B is a side view showing a structural example of the projection device in Embodiment 1.

[0018] Figure 3 ​​​​​is a diagram showing a basic operation example of the interface system in Embodiment 1.

[0019] Figure 4 is a diagram showing an example of the functional blocks of the interface system of Embodiment 1.

[0020] Figure 5 is a flowchart showing an operation example in the "A. Aerial Image Projection Phase" of the interface system of Embodiment 1.

[0021] Figure 6 is a flowchart showing an operation example in the "B. Control Execution Phase" of the interface system of Embodiment 1.

[0022] Figure 7 is a flowchart showing an operation example in the "Spatial Processing A" of the interface system of Embodiment 1.

[0023] Figure 8 is a flowchart showing an operation example in the "Spatial Processing B" of the interface system of Embodiment 1.

[0024] Figure 9 is a diagram explaining the cursor movement in Embodiment 1.

[0025] Figure 10 is a diagram explaining the cursor movement in Embodiment 1.

[0026] Figure 11 is a diagram explaining the cursor fixation in Embodiment 1.

[0027] Figure 12 is a diagram explaining the left click in Embodiment 1.

[0028] Figure 13 is a diagram explaining the right click in Embodiment 1.

[0029] Figure 14 is a diagram explaining the double left click in Embodiment 1.

[0030] Figure 15 Figure 15 A~ Figure 15 D are diagrams explaining the continuous movement operation of the pointer in Embodiment 1.

[0031] Figure 16 Figure 16 A is a diagram explaining the continuous movement operation of the pointer of the existing device, Figure 16 B is a diagram explaining the continuous movement operation of the pointer in Embodiment 1.

[0032] Figure 17 Figure 17 A、 Figure 17 ​​​​​​​​​​​​​​​​​B is a diagram for explaining the scrolling operation in Embodiment 1.

[0033] Figure 18 is a flowchart showing another example of an operation in the "B. Control Execution Phase" of the interface system of Embodiment 1.

[0034] Figure 19 is a flowchart showing an example of an operation in the "Spatial Processing AB" of the interface system of Embodiment 1.

[0035] Figure 20 Figure 20 A is a diagram for explaining the left drag operation in Embodiment 1, Figure 20 B is a diagram for explaining the right drag operation in Embodiment 1.

[0036] Figure 21 Figure 21 A, Figure 21 B is a diagram showing an example of the hardware configuration of the device control device in Embodiment 1. Detailed Embodiment

[0037] Hereinafter, with reference to the drawings, the embodiments will be described in detail.

[0038] Embodiment 1.

[0039] Figure 1 A and Figure 1 B are diagrams showing an example of the configuration of the interface system 100 of Embodiment 1. For example, as Figure 1 A and Figure 1 B show, the interface system 100 is configured to include a display device 1 and an interface device 2. In addition, Figure 1 A is a perspective view showing an example of the configuration of the interface system 100, Figure 1 B is a side view showing an example of the configuration of the interface device 2.

[0040] <Display Device 1>

[0041] For example, as Figure 1 A shows, the display device 1 is configured to include a display 10 and a display control device 11.

[0042] The display 10, for example, receives the control of the display control device 11 and displays various screens represented by a specified operation screen R on which a pointer P that can be operated by the user is displayed. The display 10 is composed of, for example, a liquid crystal display and a plasma display.

[0043] The display control device 11, for example, performs control for causing the display 10 to display various screens. The display control device 11 is composed of, for example, a PC (Personal Computer) and a server. ​​​​​​

[0044] In Embodiment 1, the user uses the interface device 2 described below to perform various operations on the display device 1. For example, the user uses the interface device 2 described below to operate the pointer P on the operation screen displayed on the display 10, or execute various instructions for the display device 1.

[0045] <Interface device 2>

[0046] The interface device 2 is a non-contact device that allows the user to input operations for the display device 1 without direct contact. For example, as Figure 1 A and Figure 1 B show, the interface device 2 is configured to include a projection device 20 and a detection device 21.

[0047] <Projection device 20>

[0048] The projection device 20 projects one or more aerial images S into the virtual space K using an imaging optical system. The imaging optical system is, for example, an optical system having a light bending surface that forms a plane for bending the optical path of the light emitted from the light source.

[0049] For example, as Figure 1 B shows, the virtual space K is a space without a physical entity set within the detectable range of the detection device 21, and is a space divided into a plurality of operation spaces. In addition, in Figure 1 B, an example is shown in which the virtual space K is set in a posture slightly inclined with respect to the detection direction of the detection device 21 ( Figure 1 the vertical direction in B), however, the virtual space K is not limited to this, and can also be set in any posture.

[0050] In addition, in the following description, for ease of understanding, an example in which the virtual space K is divided into two operation spaces (here, operation space A and operation space B) will be described. At this time, in Embodiment 1, for example, as Figure 1 B shows, the boundary position between the operation space A and the operation space B that make up the virtual space K is shown by the aerial image S projected by the projection device 20.

[0051] Next, with reference to Figure 2 A and Figure 2 B, a specific structural example of the projection device 20 will be described. Figure 2 A and Figure 2 B show an example in which the imaging optical system mounted on the projection device 20 is configured to include a beam splitter 202 and a retroreflective material 203. In addition, reference numeral 201 is a light source. Figure 2 A is a perspective view showing a structural example of the projection device 20, Figure 2 B is a side view showing a structural example of the projection device 20.

[0052] The light source 201 is composed of a display device that emits incoherent diffused light. The light source 201 is composed of, for example, a display device having a liquid crystal element and a backlight such as a liquid crystal display, a display device of a self-luminous device using an organic EL element and an LED element, or a projection device using a projector and a screen.

[0053] The beam splitter 202 is an optical element that separates incident light into transmitted light and reflected light, and its element surface functions as the above-mentioned light bending surface. The beam splitter 202 is composed of, for example, an acrylic plate and a glass plate. When the beam splitter 202 is composed of an acrylic plate and a glass plate or the like, generally, the intensity of their transmitted light is higher than that of the reflected light. Therefore, the beam splitter 202 can also be composed of a semi-reflective mirror in which a metal is added to an acrylic plate and a glass plate or the like to increase the reflection intensity.

[0054] In addition, the beam splitter 202 can also be composed of a reflective polarizing plate in which the reflection behavior and the transmission behavior change according to the polarization state of the incident light by a liquid crystal element and a thin film element. In addition, the beam splitter 202 can also be composed of a reflective polarizing plate in which the ratio of the transmittance to the reflectance changes according to a liquid crystal element and a thin film element in the polarization state of the incident light.

[0055] The retroreflective material 203 is a sheet-like optical element having a retroreflective property of directly reflecting the incident light in the incident direction. Among the optical elements that achieve retroreflection, there are a bead-type optical element in which small glass beads are paved in a mirror shape, a microprism-type optical element in which convex minute triangular pyramids each having a mirror surface on each face or a shape obtained by cutting off the central part of the triangular pyramid are paved, and the like.

[0056] In the projection device 20 having the imaging optical system configured as described above, for example, the light (diffused light) emitted from the light source 201 is specularly reflected on the surface of the beam splitter 202, and the reflected light is incident on the retroreflective material 203. The retroreflective material 203 retroreflects the incident light and is incident on the beam splitter 202 again. The light incident on the beam splitter 202 passes through the beam splitter 202 and reaches the user. Then, along the above optical path, the light emitted from the light source 201 is re-converged and re-diffused to a position symmetric to the light source 201 with the beam splitter 202 as the boundary. As a result, the user can perceive the aerial image S in the virtual space K.

[0057] In addition, in Figure 2 A and Figure 2 B, an example in which the aerial image S is projected in a star shape is shown, but the shape of the aerial image S is not limited to this, and it can be any shape.

[0058] In addition, in the above description, an example in which the imaging optical system included in the projection device 20 is configured to include a beam splitter 202 and a retroreflective material 203 has been described. However, the structure of the imaging optical system is not limited to the above example.

[0059] For example, the imaging optical system may also be configured to include a dihedral reflector array element. The dihedral reflector array element is an element formed by arranging a plurality of two mirror elements (mirrors) perpendicular to each other on a flat plate (substrate), for example.

[0060] The dihedral reflector array element has the following function: the light incident from the light source 201 disposed on one side of the plate is reflected by one of the two mirror elements, and then the reflected light is reflected by the other mirror element and passes through to the other side of the plate. If the path of the light is observed from the side, the incident path and the exit path of the light are symmetric with respect to the plate. That is, the element surface of the dihedral reflector array element functions as the above-described light bending surface, and the real image based on the light source 201 located on one side of the plate is imaged as the aerial image S at the symmetric position on the other side.

[0061] When the imaging optical system is composed of a dihedral reflector array element, in the structure in the case where the above-described retroreflective material 203 is used, the dihedral reflector array element is disposed at the position where the beam splitter 202 is disposed. In addition, in this case, the retroreflective material 203 is omitted.

[0062] In addition, the imaging optical system may also be configured to include a lens array element, for example. The lens array element is an element formed by arranging a plurality of lenses on a flat plate (substrate), for example. In this case, the element surface of the lens array element functions as the above-described light bending surface, and the real image based on the light source 201 disposed on one side of the plate is imaged as the aerial image S at the symmetric position on the other side. In addition, in this case, the distance from the light source 201 to the element surface is approximately proportional to the distance from the element surface to the aerial image S.

[0063] In addition, the imaging optical system may also be configured to include a holographic element, for example. In this case, the element surface of the holographic element functions as the above-described light bending surface. By projecting the light from the light source 201 as the reference light onto the holographic element, the holographic element outputs in a manner that reproduces the phase information of the light stored in the element. As a result, the holographic element forms the real image based on the light source 201 disposed on one side of the element as the aerial image S at the symmetric position on the other side.

[0064] <Detection device 21>

[0065] The detection device 21 detects, for example, the three-dimensional position of a detection object (for example, a user's hand) existing in the simulation space K.

[0066] As a detection method of the detection device 21 for the detection object, for example, the following method can be cited: irradiate the detection object with infrared rays, and calculate the position of the detection object existing in the depth direction within the imaging field angle of the detection device 21 by detecting its time of flight (ToF: Time of Flight) and the infrared pattern. In the first embodiment, the detection device 21 is constituted by, for example, a three-dimensional camera sensor or a two-dimensional camera sensor capable of detecting infrared wavelengths. In this case, the detection device 21 can calculate the position of the detection object existing in the imaging field angle in the depth direction and can detect the three-dimensional position of the detection object.

[0067] In addition, the detection device 21 can also be constituted by a device such as a linear sensor that detects the position in the depth direction in one dimension. Further, when the detection device 21 is constituted by a linear sensor, the three-dimensional position of the detection object can be detected by arranging a plurality of linear sensors according to the detection range.

[0068] In addition, for example, the detection device 21 can also be constituted by a stereo camera device composed of a plurality of cameras. In this case, the detection device 21 performs triangulation based on the feature points detected within the imaging field angle to detect the three-dimensional position of the detection object.

[0069] <Virtual space K>

[0070] Next, with reference to Figure 3 , a specific structural example of the virtual space K will be described.

[0071] As described above, the virtual space K is a space without a physical entity set within the detectable range of the detection device 21, and is a space divided into an operation space A and an operation space B. For example, as Figure 3 shown, the virtual space K is a space that is set as a rectangular parallelepiped shape as a whole and is divided into two operation spaces (operation space A and operation space B). In the following description, the operation space A is also referred to as the "first operation space", and the operation space B is also referred to as the "second operation space".

[0072] In this case, the aerial image S projected onto the virtual space K by the projection device 20 represents the boundary position between the two operation spaces, that is, the operation space A and the operation space B. In Figure 3 , two aerial images S are projected. These aerial images S are projected onto a closed plane (hereinafter, this plane is particularly referred to as the "boundary plane") that divides the operation space A and the operation space B. In addition, in Figure 3In FIG. 1 , an example in which two aerial images S are projected is shown, but the number of aerial images S is not limited thereto, and may be, for example, one, or more than three. Figure 3 As shown, the short side direction of the boundary surface is defined as the X-axis direction, the long side direction is defined as the Y-axis direction, and the direction perpendicular to the X-axis direction and the Y-axis direction is defined as the Z-axis direction.

[0073] In addition, the operation space A and the operation space B correspond to operations that can be performed by the user when the three-dimensional position of the detection object detected by the detection device 21 is included in each operation space. In addition, in the following description, for easy understanding of the description, the case where the detection object of the detection device 21 is the user's hand is used as an example. In this case, it is assumed that the detection device 21 detects the three-dimensional position of the user's hand in the virtual space K, especially the three-dimensional positions of the five fingers of the user's hand in the virtual space K.

[0074] For example, the operation space A corresponds to an operation of the pointer P as an operation that the user can perform. Specifically, for example, when the user puts his hand into the operation space A, that is, when the three-dimensional positions of the five fingers of the user's hand detected by the detection device 21 are all included in the operation space A, when the user moves his hand in the operation space A, the pointer P displayed on the operation screen R of the display 10 can be moved in conjunction with the movement ( Figure 3 In addition, Figure 3 On the left side of FIG. 1 , as a schematic diagram, a pointer P is shown on the operation space A; however, in reality, the pointer P displayed on the operation screen R of the display 10 moves.

[0075] In the following description, “the three-dimensional position of the user's hand is included in the operation space A” means “the three-dimensional positions of the five fingers of the user's hand are all included in the operation space A”. In the following description, “the user operates the operation space A” means “the user moves his hand while the three-dimensional position of the user's hand is included in the operation space A”. In the following description, the operation mode of the interface system 100 when the user puts his hand in the operation space A is also referred to as “pointer operation mode”.

[0076] Furthermore, when the user places his hand from the operation space A across the boundary position (boundary surface) into the operation space B, that is, when the three-dimensional positions of the five fingers of the user's hand detected by the detection device 21 are all included in the operation space B, the movement of the pointer P displayed on the operation screen R in the display 10 is fixed ( Figure 3 In addition, Figure 3 On the right side, square brackets displayed at the four corners of the pointer P indicate that the movement of the pointer P has been fixed.

[0077] At this time, even if the user moves their hand in the operation space B, the pointer P does not move. On the other hand, when the user moves their hand in a specified pattern in the operation space B, instructions (such as left click and right click) corresponding to the movement (gesture) can be executed.

[0078] In addition, in the following description, "the three-dimensional position of the user's hand is included in the operation space B" means "the three-dimensional positions of the five fingers of the user's hand are all included in the operation space B". Further, in the following description, "the user operates the operation space B" means "the user moves their hand in a state where the three-dimensional position of the user's hand is included in the operation space B". In addition, in the following description, the operation mode of the interface system 100 when the user puts their hand into the operation space B is also referred to as the "instruction execution mode".

[0079] In addition, the range of the operation space A is, for example, in Figure 3 the Z-axis direction, from the position of the boundary surface of the projected aerial image S to the upper limit position of the detectable range of the detection device 21. In addition, the range of the operation space B is, for example, in Figure 3 the Z-axis direction, from the position of the boundary surface of the projected aerial image S to the lower limit position of the detectable range of the detection device 21.

[0080] In addition, on the Figure 3 right side, the aerial image SC is the aerial image projected by the projection device 20 when the user puts their hand from the operation space A across the boundary position (boundary surface) into the operation space B. The aerial image SC represents the lower limit position of the detectable range of the detection device 21 and represents the reference position for dividing the operation space B into left and right spaces when viewed from the user side. The aerial image SC is projected by the projection device 20 near the lower limit position of the detectable range of the detection device 21 and near the approximate center of the operation space B in the X-axis direction. The aerial image S exists in the plane (boundary surface) with a Z-axis coordinate position of 0. In contrast, the aerial image SC exists in the region with a negative Z-axis coordinate position. Thus, the user can easily grasp to what extent the hand can be lowered in the operation space B and can execute instructions that require specifying left and right, such as left click and right click. The input method for instructions such as left click and right click will be described later.

[0081] Next, the functional blocks of the interface system 100 in Embodiment 1 will be described. Figure 4 An example of the functional block diagram of the interface system 100 in Embodiment 1 is shown.

[0082] As Figure 4As shown, the interface system 100 includes an aerial image projection unit 31, a position detection unit 32, a position acquisition unit 41, a boundary position recording unit 42, an operation space determination unit 43, a pointer operation information output unit 44, a pointer position control unit 45, an instruction determination unit 46, an instruction recording unit 47, an instruction output unit 48, an instruction generation unit 49, and an aerial image generation unit 50.

[0083] The aerial image projection unit 31 acquires data representing the aerial image S generated by the aerial image generation unit 50, and projects the aerial image S based on the acquired data into the virtual space K. The aerial image projection unit 31 is constituted by, for example, the above-described projection device 20. Additionally, the aerial image projection unit 31 may also acquire data representing the above-described aerial image SC generated by the aerial image generation unit 50, and project the aerial image SC based on the acquired data into the virtual space K.

[0084] The position detection unit 32 detects the three-dimensional position of a detection object (here, the user's hand) in the virtual space K. The position detection unit 32 is constituted by, for example, the above-described detection device 21. The position detection unit 32 outputs the detection result of the three-dimensional position of the detection object (hereinafter, also referred to as "position detection result") to the position acquisition unit 41.

[0085] Furthermore, the position detection unit 32 may also detect the three-dimensional position of the aerial image S projected into the virtual space K, and record the data representing the detected three-dimensional position of the aerial image S into the boundary position recording unit 42.

[0086] In addition, when the aerial image projection unit 31 is constituted by the above-described projection device 20 and the position detection unit 32 is constituted by the above-described detection device 21, the functions of the aerial image projection unit 31 and the position detection unit 32 are realized by the above-described interface device 2.

[0087] The position acquisition unit 41 acquires the position detection result output from the position detection unit 32. The position acquisition unit 41 outputs the acquired position detection result to the operation space determination unit 43.

[0088] The boundary position recording unit 42 records data representing the three-dimensional position of the aerial image S, which is the boundary position between the operation space A and the operation space B constituting the virtual space K. The boundary position recording unit 42 is constituted by, for example, an HDD (Hard Disc Drive), an SSD (Solid State Drive), etc.

[0089] For example, when the aerial image S is composed of Figure 3In the case of a linear (straight line) figure shown, the boundary position recording unit 42 records data representing the three-dimensional position of at least one point (pixel) among the points constituting the aerial image S of the line. For example, the boundary position recording unit 42 may record data representing the three-dimensional positions of any three points among the points constituting the aerial image S of the line, or may record data representing the three-dimensional positions of all the points among the points constituting the aerial image S of the line. In addition, since the aerial image S is projected onto Figure 3 the side surface shown, the coordinate positions of the respective points recorded in the boundary position recording unit 42 in the Z-axis direction are all the same coordinate positions.

[0090] The operation space determination unit 43 acquires the position detection result output from the position acquisition unit 41. In addition, the operation space determination unit 43 determines the operation space where the user's hand exists based on the acquired position detection result and the boundary positions of the respective operation spaces in the virtual space K. The operation space determination unit 43 outputs the above-determined result (hereinafter, also referred to as "space determination result") to the aerial image generation unit 50. In addition, the operation space determination unit 43 outputs the space determination result and the position detection result acquired from the position acquisition unit 41 to the operation information output unit 51.

[0091] The operation information output unit 51 outputs operation information for performing a specified operation on the display device 1 by using at least the space determination result of the operation space determination unit 43. The operation information output unit 51 is configured to include a pointer operation information output unit 44, an instruction determination unit 46, and an instruction output unit 48.

[0092] The pointer operation information output unit 44 acquires the space determination result and the position detection result output from the operation space determination unit 43. When the acquired space determination result indicates that the user's hand exists in the operation space A, the pointer operation information output unit 44 generates information (hereinafter, also referred to as "movement control information") for causing the pointer P displayed on the operation screen R of the display 10 to move corresponding to the movement of the user's hand in the operation space A. In addition, let "the movement of the user's hand" include information related to movement such as the movement amount of the user's hand. For example, the pointer operation information output unit 44 calculates the movement amount of the user's hand based on the position detection result output from the operation space determination unit 43. The movement amount of the user's hand includes information related to the direction in which the user's hand moves and the distance by which the user's hand moves in that direction.

[0093] Then, the pointer operation information output unit 44 generates information (movement control information) for causing the pointer P displayed on the operation screen R of the display 10 to move corresponding to the movement of the user's hand in the operation space A based on the calculated movement amount. The pointer operation information output unit 44 outputs the above operation information including the generated movement control information to the pointer position control unit 45.

[0094] In addition, when the spatial determination result obtained above indicates that the user's hand exists in the operation space B, the pointer operation information output unit 44 generates information for fixing the pointer P displayed on the operation screen R of the display 10 (hereinafter, also referred to as "fixing control information"). The pointer operation information output unit 44 outputs the above operation information including the generated fixing control information to the pointer position control unit 45.

[0095] In addition, the pointer operation information output unit 44 may also be based on the distance in the direction perpendicular to the boundary surface between the three-dimensional position of the user's hand included in the operation space A and the boundary surface of the virtual space K represented by the aerial image S ( Figure 3 in the Z-axis direction of the boundary surface), and include information for making the movement amount or movement speed of the pointer P displayed on the screen of the display device 1 variable in the operation information and output it.

[0096] The pointer position control unit 45 obtains the operation information output from the pointer operation information output unit 44. When the operation information obtained from the pointer operation information output unit 44 includes movement control information, the pointer position control unit 45 moves the pointer P on the operation screen R displayed on the display 10 corresponding to the movement of the user's hand according to the movement control information. For example, the pointer position control unit 45 moves it by an amount equivalent to the movement amount of the user's hand, in other words, moves it by the distance included in the movement amount in the direction included in the movement amount.

[0097] In addition, when the operation information obtained from the pointer operation information output unit 44 includes fixing control information, the pointer position control unit 45 fixes the pointer P on the operation screen R displayed on the display 10 according to the fixing control information.

[0098] The instruction determination unit 46 obtains the spatial determination result and the position detection result output from the operation space determination unit 43. When the spatial determination result obtained above indicates that the user's hand exists in the operation space B, the instruction determination unit 46 determines the movement (gesture) of the user's hand according to the position detection result output from the operation space determination unit 43.

[0099] The instruction recording unit 47 pre-records instruction information. The instruction information is information that associates the movement (gesture) of the user's hand with the instructions that the user can execute. The instruction recording unit 47 is composed of, for example, an HDD (Hard Disc Drive) or an SSD (Solid State Drive).

[0100] The instruction determination unit 46 determines an instruction corresponding to the movement (gesture) of the user's hand determined above based on the instruction information recorded in the instruction recording unit 47. The instruction determination unit 46 outputs the determined instruction to the instruction output unit 48 and the aerial image generation unit 50.

[0101] The instruction output unit 48 acquires the instruction output from the instruction determination unit 46. The instruction output unit 48 outputs the above operation information including the information indicating the acquired instruction to the instruction generation unit 49.

[0102] The instruction generation unit 49 receives the operation information output from the instruction output unit 48 and generates the instruction included in the received operation information. Thus, in the interface system 100, an instruction corresponding to the movement (gesture) of the user's hand is executed.

[0103] The aerial image generation unit 50 generates data representing the aerial image S projected by the aerial image projection unit 31 into the virtual space K. The aerial image generation unit 50 outputs the generated data representing the aerial image S to the aerial image projection unit 31.

[0104] In addition, the aerial image generation unit 50 may acquire the space determination result output from the operation space determination unit 43 and regenerate the data representing the aerial image S projected in a manner corresponding to the acquired space determination result. In addition, the aerial image generation unit 50 may output the regenerated data representing the aerial image S to the aerial image projection unit 31.

[0105] For example, when the space determination result indicates that the user's hand exists in the operation space A, the aerial image generation unit 50 may regenerate the data representing the aerial image S projected in blue. In addition, when the space determination result indicates that the user's hand exists in the operation space B, the aerial image generation unit 50 may regenerate the data representing the aerial image S projected in red. In addition, when the space determination result indicates that the user's hand exists in the operation space B, the aerial image generation unit 50 may generate the data representing the above aerial image SC, and output the generated data representing the aerial image SC to the aerial image projection unit 31.

[0106] In addition, the aerial image generation unit 50 may acquire the instruction output from the instruction determination unit 46 and regenerate the data representing the aerial image S projected in a manner corresponding to the acquired instruction. In addition, the aerial image generation unit 50 may output the regenerated data representing the aerial image S to the aerial image projection unit 31.

[0107] For example, when the instruction obtained from the instruction determination unit 46 is a left click, the aerial image generation unit 50 may also regenerate data of the aerial image S representing one flash. Further, when the instruction obtained from the instruction determination unit 46 is a double left click, the aerial image generation unit 50 may also regenerate data of the aerial image S representing two consecutive flashes.

[0108] In addition, the above operation information output unit 51 may also include a sound information output unit (not shown). When the operation information including the fixed control information is output from the pointer operation information output unit 44 to the pointer position control unit 45, the sound information output unit generates information for outputting a sound corresponding to the fixation of the pointer P (a sound notifying the fixation of the pointer P), and includes the generated information in the above operation information for output. In this case, when the pointer position control unit 45 fixes the pointer P according to the fixed control information, a sound corresponding to the fixation of the pointer P is output. Therefore, the user can easily grasp that the pointer P has been fixed by listening to this sound.

[0109] In addition, the above sound information output unit may also generate information for outputting a sound corresponding to the instruction determined by the instruction determination unit 46, and include the generated information in the above operation information for output. In this case, when the instruction generation unit 49 generates an instruction, a sound corresponding to the instruction is output. Therefore, the user can easily grasp that the instruction has been generated by listening to this sound.

[0110] In addition, the above sound information output unit may also generate information for outputting a sound corresponding to the three-dimensional position of the user's hand in the operation space A or a sound corresponding to the movement of the user's hand in the operation space A, and include the generated information in the above operation information for output. For example, the above sound information output unit may generate information for outputting a sound corresponding to the three-dimensional position of the user's hand in the operation space A detected by the position detection unit 32, and include the generated information in the above operation information for output. In this case, for example, when the user brings the hand close to the boundary surface in the operation space A, a sound whose volume increases as the user's hand approaches the boundary surface is output. The user can easily grasp that the hand has approached the boundary surface by listening to this sound.

[0111] In addition, for example, the above sound information output unit may generate information for outputting a sound corresponding to the movement amount of the user's hand calculated by the pointer operation information output unit 44, and include the generated information in the above operation information for output. In this case, for example, the larger the user moves the hand in the operation space A (the larger the movement amount of the hand), the louder the output sound. The user can easily grasp that the hand has moved significantly by listening to this sound. In this way, the user can easily grasp the three-dimensional position of the hand or the movement of the hand in the operation space A by listening to the sound.

[0112] In addition, in Embodiment 1, the above-mentioned position acquisition unit 41, boundary position recording unit 42, operation space determination unit 43, pointer operation information output unit 44, pointer position control unit 45, instruction determination unit 46, instruction recording unit 47, instruction output unit 48, instruction generation unit 49, and aerial image generation unit 50 are mounted on the display control device 11, for example. In this case, the device control device 12 is constituted by including the position acquisition unit 41, boundary position recording unit 42, operation space determination unit 43, pointer operation information output unit 44, instruction determination unit 46, instruction recording unit 47, instruction output unit 48, and aerial image generation unit 50. The device control device 12 controls the interface device 2.

[0113] In addition, in the above description, an example in which the boundary position recording unit 42 and the instruction recording unit 47 are mounted on the device control device 12 is described. However, the boundary position recording unit 42 and the instruction recording unit 47 are not limited thereto, and may be provided outside the device control device 12.

[0114] Next, with reference to Figures 5 - 8 the flowchart shown, an operation example of the interface system 100 of Embodiment 1 will be described. Here, for easy understanding of the description, the operation example of the interface system 100 is divided into "A. Aerial image projection stage" and "B. Control execution stage" for description.

[0115] <A. Aerial image projection stage>

[0116] First, with reference to Figure 5 the flowchart shown, the aerial image projection stage will be described. In the aerial image projection stage, the aerial image S is projected in the virtual space K. In addition, when the interface system 100 is started, the aerial image projection stage is executed at least once.

[0117] First, the aerial image generation unit 50 generates data representing the aerial image S projected by the aerial image projection unit 31 into the virtual space K (step A001). The aerial image generation unit 50 outputs the generated data representing the aerial image S to the aerial image projection unit 31.

[0118] Next, the aerial image projection unit 31 acquires the data representing the aerial image S generated by the aerial image generation unit 50, and projects the aerial image S based on the acquired data into the virtual space K (step A002).

[0119] Next, the position detection unit 32 detects the three-dimensional position of the aerial image S projected into the virtual space K, and records the data representing the detected three-dimensional position of the aerial image S in the boundary position recording unit 42 (step A003).

[0120] In addition, in the above description, the following example was described: First, the aerial image projection unit 31 projects the aerial image S. Then, the position detection unit 32 detects the three-dimensional position of the aerial image S, and records the data representing the detected three-dimensional position of the aerial image S in the boundary position recording unit 42. However, step A003 is not an essential process and can also be omitted. For example, in the interface system 100, it can also be that, first, the user pre-records the data representing the three-dimensional position of the aerial image S in the boundary position recording unit 42, and the aerial image projection unit 31 projects the aerial image S onto the three-dimensional position represented by this data. In this case, step A003 can also be omitted.

[0121] <B. Control Execution Phase>

[0122] Next, referring to Figure 6 the flowchart shown, the control execution phase will be described. In the control execution phase, the user uses the interface device 2 to execute the control of the display control device 11 and the device control device 12. In addition, after the above-mentioned aerial image projection phase is completed, the control execution phase is repeatedly executed at regular intervals.

[0123] First, when the user puts their hand into the virtual space K, the position detection unit 32 detects the three-dimensional position of the user's hand in the virtual space K (step B001). The position detection unit 32 outputs the detection result (position detection result) of the three-dimensional position of the user's hand to the position acquisition unit 41.

[0124] Next, the position acquisition unit 41 acquires the position detection result output from the position detection unit 32 (step B002). The position acquisition unit 41 outputs the acquired position detection result to the operation space determination unit 43.

[0125] Next, the operation space determination unit 43 acquires the detection result output from the position acquisition unit 41, and determines the operation space where the user's hand exists based on the acquired position detection result and the boundary positions of each operation space in the virtual space K.

[0126] For example, the operation space determination unit 43 compares Figure 3 the position coordinates of the five fingers of the user's hand in the Z-axis direction shown in

[0127] Next, the operation space determination unit 43 confirms whether it is determined that the user's hand exists in the operation space A (step B003). When it is determined that the user's hand exists in the operation space A (step B003; YES), the operation space determination unit 43 outputs the determined result (space determination result) to the airborne image generation unit 50 (step B004). In addition, the operation space determination unit 43 outputs the space determination result together with the position detection result obtained from the position acquisition unit 41 to the pointer operation information output unit 44 (step B004). Then, the process proceeds to step B005 (space process A).

[0128] On the other hand, when it is determined in step B003 that the user's hand does not exist in the operation space A (step B003; NO), the operation space determination unit 43 confirms whether it is determined that the user's hand exists in the operation space B (step B006). When it is determined that the user's hand exists in the operation space B (step B006; YES), the operation space determination unit 43 outputs the determined result (space determination result) to the airborne image generation unit 50 (step B007). In addition, the operation space determination unit 43 outputs the space determination result together with the position detection result obtained from the position acquisition unit 41 to the pointer operation information output unit 44 and the instruction determination unit 46 (step B007). Then, the process proceeds to step B008 (space process B).

[0129] On the other hand, when it is determined in step B006 that the user's hand does not exist in the operation space B (step B006; NO), the interface system 100 ends the process.

[0130] <Space process A>

[0131] Next, with reference to Figure 7 the flowchart shown, the space process A of step B005 will be described.

[0132] First, the airborne image generation unit 50 acquires the space determination result that the user's hand exists in the operation space A output from the operation space determination unit 43, and regenerates data representing the airborne image S projected in a manner corresponding to the acquired space determination result (step C001). For example, the airborne image generation unit 50 regenerates data representing the airborne image S projected in blue as the airborne image S indicating that the user's hand exists in the operation space A. The airborne image generation unit 50 outputs the regenerated data representing the airborne image S to the airborne image projection unit 31.

[0133] Next, the aerial image projection unit 31 obtains the data representing the aerial image S regenerated by the aerial image generation unit 50, and re-projects the aerial image S based on the obtained data to the virtual space K (step C002). That is, the aerial image projection unit 31 updates the aerial image S projected to the virtual space K. As a result, for example, the color of the aerial image S changes to blue, and the user can easily understand that the hand has entered the operation space A (becoming the pointer operation mode). In addition, steps C001 and C002 are not essential processes and can be omitted.

[0134] Next, the pointer operation information output unit 44 determines whether the user's hand has moved based on the position detection result output from the operation space determination unit 43 (step C003). If it is determined that the user's hand has not moved (step C003; No), the process returns. On the other hand, if it is determined that the user's hand has moved (step C003; Yes), the process transfers to step C004.

[0135] In step C004, the pointer operation information output unit 44 determines the movement of the user's hand based on the position detection result output from the operation space determination unit 43. Then, the pointer operation information output unit 44 generates information (movement control information) for moving the pointer P displayed on the operation screen R of the display 10 in accordance with the movement of the user's hand in the operation space A (step C004). In addition, the pointer operation information output unit 44 outputs operation information including the generated movement control information to the pointer position control unit 45 (step C005).

[0136] Next, the pointer position control unit 45 controls the pointer P according to the movement control information included in the operation information output from the pointer operation information output unit 44 (step C006). Specifically, the pointer position control unit 45 moves the pointer P on the operation screen R displayed on the display 10 in accordance with the movement of the user's hand according to the movement control information. In more detail, the pointer position control unit 45 moves the pointer P on the operation screen R displayed on the display 10 by an amount corresponding to the movement amount of the user's hand, in other words, moves the pointer P in the direction included in the movement amount by a distance included in the movement amount. Thus, the pointer P moves in conjunction with the movement of the user's hand. Then, the process returns.

[0137] <Spatial Processing B>

[0138] Next, refer to Figure 8 The flowchart shown illustrates the spatial processing B of step B008.

[0139] First, the aerial image generation unit 50 acquires the space determination result output from the operation space determination unit 43 that the user's hand exists in the operation space B, and regenerates data representing the aerial image S projected in a manner corresponding to the acquired space determination result (step D001). For example, the aerial image generation unit 50 regenerates data representing the aerial image S projected in red as the aerial image S indicating that the user's hand exists in the operation space B. The aerial image generation unit 50 outputs the regenerated data representing the aerial image S to the aerial image projection unit 31.

[0140] Next, the aerial image projection unit 31 acquires the data representing the aerial image S regenerated by the aerial image generation unit 50, and re-projects the aerial image S based on the acquired data onto the virtual space K (step D002). That is, the aerial image projection unit 31 updates the aerial image S projected onto the virtual space K. Thus, for example, the color of the aerial image S becomes red, and the user can easily grasp that the hand has entered the operation space B (entering the instruction execution mode). In addition, steps D001 and D002 are not essential processes and can also be omitted.

[0141] Next, the pointer operation information output unit 44 generates control information (fixing control information) for fixing the pointer P displayed on the operation screen R of the display 10 (step D003). In addition, the pointer operation information output unit 44 outputs operation information including the generated fixing control information to the pointer position control unit 45 (step D004).

[0142] Next, the pointer position control unit 45 fixes the pointer P on the operation screen R displayed on the display 10 according to the fixing control information included in the operation information output from the pointer operation information output unit 44 (step D005).

[0143] Next, the instruction determination unit 46 determines whether the user's hand has moved based on the position detection result output from the operation space determination unit 43 (step D006). If the result is that it is determined that the user's hand has not moved (step D006; no), the process returns. On the other hand, if it is determined that the user's hand has moved (step D006; yes), the process proceeds to step D007.

[0144] In step D007, the instruction determination unit 46 determines the movement (gesture) of the user's hand based on the position detection result output from the operation space determination unit 43 (step D007).

[0145] Next, the instruction determination unit 46 refers to the instruction information recorded in the instruction recording unit 47 and determines whether the movement corresponding to the determined hand movement is included in the instruction information (step D008). If the result of the determination is that the movement corresponding to the determined hand movement is not included in the instruction information (step D008; No), the process returns. On the other hand, if it is determined that the movement corresponding to the determined hand movement is included in the instruction information (step D008; Yes), the instruction determination unit 46 determines the instruction corresponding to this movement in the instruction information (step D009). The instruction determination unit 46 outputs the determined instruction to the instruction output unit 48.

[0146] Next, the instruction output unit 48 outputs operation information including information indicating the instruction obtained from the instruction determination unit 46 to the instruction generation unit 49 (step D010).

[0147] Next, the instruction generation unit 49 receives the operation information output from the instruction output unit 48 and generates the instruction included in the received operation information (step D011). Thus, in the interface system 100, an instruction corresponding to the movement (gesture) of the user's hand is executed.

[0148] In addition, although not shown in the above flowchart, in step D009, the instruction determination unit 46 may also output the determined instruction to the aerial image generation unit 50. Then, the aerial image generation unit 50 may also obtain the instruction output from the instruction determination unit 46 and regenerate the data representing the aerial image S projected in a manner corresponding to the obtained instruction. In addition, the aerial image generation unit 50 may output the regenerated data representing the aerial image S to the aerial image projection unit 31.

[0149] In addition, the aerial image projection unit 31 may obtain the data representing the aerial image S regenerated by the aerial image generation unit 50 and re-project the aerial image S based on the obtained data onto the virtual space K. That is, the aerial image projection unit 31 may update the aerial image S projected onto the virtual space K. Thus, for example, the aerial image S blinks once, and the user can easily grasp that the left click instruction has been executed.

[0150] Next, refer to Figures 9 - 17 , and explain the control example of the interface system 100 of Embodiment 1. The interface system 100 of Embodiment 1 operates as described above and can perform the following control, for example.

[0151] (1) Pointer movement

[0152] When the user's hand is in the operation space A, the pointer P moves on the operation screen R of the display 10 according to the movement amount of the user's hand in the virtual space K (XYZ coordinate system) (refer toFigure 9 )。In addition, in Figure 9 , as a schematic diagram, a pointer P is shown on the operation space A, but actually, it is the pointer P displayed on the operation screen R of the display 10 that moves.

[0153] In addition, in the above case, the pointer operation information output unit 44 can also generate the following movement control information: even for the same amount of movement of the user's hand, the amount of movement or the movement speed of the pointer P varies depending on how far the three-dimensional position of the user's hand is from the boundary surface (XY plane) of the virtual space represented by the aerial image S in the direction perpendicular to the boundary surface (i.e., the Z-axis direction).

[0154] For example, as Figure 10 shown, if the three-dimensional position of the user's hand is far from the boundary surface (XY plane) in the Z-axis direction, the pointer operation information output unit 44 can also generate movement control information that makes the pointer P move by a distance approximately the same as the distance the user's hand moves or at a speed approximately the same as the speed of the user's hand movement ( Figure 10 mark W1). On the other hand, even if the amount of movement of the user's hand is the same as above, if the three-dimensional position of the user's hand is close to the boundary surface (XY plane) in the Z-axis direction, the pointer operation information output unit 44 can also generate movement control information that makes the pointer P move by a distance approximately half of the distance the user's hand moves or at a speed approximately half of the speed of the user's hand movement ( Figure 10 mark W2).

[0155] That is, the pointer operation information output unit 44 can also multiply the amount of movement or the movement speed of the user's hand projected onto the boundary surface (XY plane) where the aerial image S is projected by a coefficient corresponding to the distance in the Z-axis direction between the three-dimensional position of the user's hand and the boundary surface (XY plane) to generate movement control information.

[0156] In this case, if the user moves the hand at a position far from the boundary surface (XY plane) where the aerial image S is projected in the Z-axis direction, the pointer P can be made to move by an amount equivalent to the amount of hand movement or at the same speed as the hand movement. On the other hand, if the user moves the hand at a position close to the boundary surface (XY plane) where the aerial image S is projected in the Z-axis direction, the pointer P can be made to move finely (slowly). In particular, assume that when the user transfers from the pointer movement mode to the instruction execution mode, the user moves the hand near the boundary surface where the aerial image S is projected. At this time, since the user can move the pointer P finely or slowly, the position of the pointer P when executing the instruction can be specified finely, improving convenience.

[0157] In addition, the following examples are described herein: If the three-dimensional position of the user's hand is away from the boundary surface (XY plane) in the Z-axis direction, the pointer operation information output unit 44 generates movement control information that causes the pointer P to move at the same distance as the distance the user's hand moves or at a speed similar to the speed at which the user's hand moves. If the three-dimensional position of the user's hand is close to the boundary surface (XY plane) in the Z-axis direction, the pointer operation information output unit 44 generates movement control information that causes the pointer P to move at approximately half the distance the user's hand moves or at approximately half the speed at which the user's hand moves. However, conversely, it can also be that if the three-dimensional position of the user's hand is away from the boundary surface (XY plane) in the Z-axis direction, the pointer operation information output unit 44 generates movement control information that causes the pointer P to move at approximately half the distance the user's hand moves or at approximately half the speed at which the user's hand moves. If the three-dimensional position of the user's hand is close to the boundary surface (XY plane) in the Z-axis direction, the pointer operation information output unit 44 generates movement control information that causes the pointer P to move at the same distance as the distance the user's hand moves or at the same speed as the speed at which the user's hand moves.

[0158] (2) Pointer fixation

[0159] In the case where the user's hand crosses the position of the virtual image in the air (boundary position) from the operation space A and enters the operation space B, the pointer P is fixed on the operation screen R of the display 10 (see Figure 11 ). Then, even if the user's hand moves in the operation space B, the pointer P remains fixed on the operation screen R of the display 10. In addition, at this time, the virtual image S can also be updated. For example, the color of the virtual image S can be changed from blue to red. Thereby, the user can easily grasp that the hand has entered the operation space B (changed to the instruction execution mode). In addition, at this time, the virtual image SC can also be projected by the projection device 20 to a position near the lower limit position of the detectable range of the detection device 21 and approximately at the center in the X-axis direction in the virtual space K.

[0160] (3) Left click

[0161] For example, in the operation space B, when the user moves the hand in the -Y direction and the hand reaches a preset left click generation area, the instruction determination unit 46 determines the movement (gesture) of the hand. The left click generation area is, for example, a specified area on the left side (-X direction side) of the virtual image SC in the operation space B and on the inner side (-Y direction side) when viewed from the user.

[0162] This movement (gesture) corresponds to the instruction of "left click" in the instruction information. Therefore, the instruction determination unit 46 determines the instruction of "left click" and executes the left click (see Figure 12)。In addition, at this time, it can also be that the aerial image generation unit 50, for example, regenerates the data of the aerial image S representing one blink, and the aerial image projection unit 31 projects the aerial image S based on the regenerated data. Thus, in the interface system 100, the aerial image S blinks once, and the user can easily grasp that a left click has been executed. Also, at this time, the interface system 100 can also output a sound such as "click" as the sound corresponding to the left click. Thus, by hearing this sound, the user can more easily grasp that a left click has been executed.

[0163] (4) Right click

[0164] For example, in the operation space B, when the user moves the hand in the -Y direction and the hand reaches a preset right click generation area, the instruction determination unit 46 determines the movement (gesture) of the hand. The right click generation area is, for example, a specified area in the operation space B that is on the right side (+X direction side) of the aerial image SC and inward (-Y direction side) when viewed from the user.

[0165] This movement (gesture) is associated with the instruction of "right click" in the instruction information. Therefore, the instruction determination unit 46 determines the instruction of "right click" and executes the right click (refer to Figure 13 )。In addition, at this time, it can also be that the aerial image generation unit 50, for example, regenerates the data of the aerial image S representing one blink, and the aerial image projection unit 31 projects the aerial image S based on the regenerated data. Thus, in the interface system 100, the aerial image S blinks once, and the user can easily grasp that a right click has been executed.

[0166] (5) Left double click

[0167] For example, in the operation space B, when the user moves the hand in the -Y direction and then, while the hand is in the state of reaching the preset left click generation area, the user moves the hand continuously in the +Y direction and -Y direction, the instruction determination unit 46 determines the movement (gesture) of the hand. This movement (gesture) is associated with the instruction of "left double click" in the instruction information. Therefore, the instruction determination unit 46 determines the instruction of "left double click" and executes the left double click (refer to Figure 14 )。In addition, at this time, it can also be that the aerial image generation unit 50, for example, regenerates the data of the aerial image S representing two consecutive blinks, and the aerial image projection unit 31 projects the aerial image S based on the regenerated data. Thus, in the interface system 100, the aerial image S blinks twice continuously, and the user can easily grasp that a left double click has been executed. Also, at this time, the interface system 100 can also output continuous sounds such as "click", "click" as the sound corresponding to the left double click. Thus, by hearing this sound, the user can more easily grasp that a left double click has been executed.

[0168] (6) Continuous movement operation of the pointer

[0169] When the user moves the hand in the +Y direction in the operation space A, the pointer P also moves in the +Y direction in linkage with this movement (refer to Figure 15 A). Here, the user moves the hand once to the operation space B to fix the pointer P (refer to Figure 15 B). In this state, when the user moves the hand in the -Y direction, the pointer P remains fixed (refer to Figure 15 C).

[0170] Then, when the user moves the hand from the operation space B across the boundary position (boundary surface) to the operation space A, the pointer P moves again in linkage with the movement of the user's hand (refer to Figure 15 D). By repeating the above operations, the user can continuously move the pointer P only by moving the hand in a limited space such as the operation space A and the operation space B.

[0171] Regarding this point, in the above-described conventional device, for example, as Figure 16 shown in A, in operations accompanied by continuity such as long-distance movement and scrolling of the pointer P, the movement amount of the user's hand becomes large, and a space large enough to enable such a large movement is required. In contrast, in the first embodiment, for example, as Figure 16 shown in B, the user's hand moves back and forth at the boundary position (boundary surface), whereby the correlation between the pointer P and the user's hand can be reset. Therefore, the user repeatedly moves the hand with a short movement distance, and thus, even in a limited space such as the operation space A and the operation space B, operations with continuity such as long-distance movement and scrolling of the pointer P can be achieved.

[0172] (7) Scroll operation

[0173] When the user starts to move (gesture) the hand, such as rotating the hand, within a range that does not reach the left-click generation area or the right-click generation area in the operation space B, the instruction determination unit 46 determines this movement (gesture) of the hand. This movement (gesture) is associated with the instruction of "scroll operation" in the instruction information. Therefore, in the interface system 100, the instruction determination unit 46 determines the instruction of "scroll operation" and executes the scroll operation (refer to Figure 17 A). In addition, at this time, it may also be that the aerial image generation unit 50, for example, regenerates data of an aerial image SE obtained by adding a specified graphic to the current aerial image S, and the aerial image projection unit 31 projects the aerial images S and SE based on the regenerated data (refer to Figure 17 B). Thereby, the aerial images S and SE to which the specified graphic is added are projected, and the user can easily grasp that the scroll operation can be executed.

[0174] Next, refer to Figure 18The flowchart shown illustrates an example of an application operation during the control execution phase of the interface system 100 according to Embodiment 1. In this example of the application operation, an example is described in which the user operates both the operation space A and the operation space B using both the left and right hands.

[0175] First, when the user places a hand in the virtual space K, the position detection unit 32 detects the three-dimensional position of the user's hand in the virtual space K (step E001). The position detection unit 32 outputs the detection result of the three-dimensional position of the user's hand (position detection result) to the position acquisition unit 41.

[0176] Next, the position acquisition unit 41 acquires the position detection result output from the position detection unit 32 (step E002). The position acquisition unit 41 outputs the acquired position detection result to the operation space determination unit 43.

[0177] Next, the operation space determination unit 43 acquires the detection result output from the position acquisition unit 41, and determines the operation space in which the user's hand exists based on the acquired position detection result and the boundary positions of each operation space in the virtual space K.

[0178] Next, the operation space determination unit 43 confirms whether it is determined that the user's hand exists in both the operation space A and the operation space B (step E003). If it is determined that the user's hand does not exist in both the operation space A and the operation space B (step E003; NO), the process proceeds to step B003 of the above Figure 6 flowchart.

[0179] On the other hand, if it is determined that the user's hand exists in both the operation space A and the operation space B (step E003; YES), the operation space determination unit 43 outputs the determined result (space determination result) to the aerial image generation unit 50. In addition, the operation space determination unit 43 outputs the space determination result and the position detection result acquired from the position acquisition unit 41 to the pointer operation information output unit 44 and the instruction determination unit 46 (step E004). Then, the process proceeds to step E005 (space processing AB).

[0180] <Space Processing AB>

[0181] Next, with reference to Figure 19 the flowchart shown, the space processing AB in step E005 is described.

[0182] First, the aerial image generation unit 50 acquires the space determination result output from the operation space determination unit 43, indicating that the user's hand exists in both the operation space A and the operation space B, and regenerates the data representing the aerial image S projected in a manner corresponding to the acquired space determination result (step F001). For example, the aerial image generation unit 50 regenerates the data representing the aerial image S projected in green as the aerial image S indicating that the user's hand exists in both the operation space A and the operation space B. The aerial image generation unit 50 outputs the regenerated data representing the aerial image S to the aerial image projection unit 31.

[0183] Next, the aerial image projection unit 31 acquires the data representing the aerial image S regenerated by the aerial image generation unit 50, and re-projects the aerial image S based on the acquired data into the virtual space K (step F002). That is, the aerial image projection unit 31 updates the aerial image S projected into the virtual space K. As a result, for example, the color of the aerial image S becomes green, and the user can easily grasp that the hand has entered both the operation space A and the operation space B. In addition, steps F001 and F002 are not essential processes and can also be omitted.

[0184] Next, the pointer operation information output unit 44 determines whether the user's hand has moved based on the position detection result output from the operation space determination unit 43 (step F003). If the result is that it is determined that the user's hand has not moved (step F003; no), the process returns. On the other hand, if it is determined that the user's hand has moved (step F003; yes), the process proceeds to step F004.

[0185] In step F004, the instruction determination unit 46 determines the movement (gesture) of the user's hand based on the position detection result output from the operation space determination unit 43 (step F004). In this case, the movement (gesture) of the user's hand is a movement that combines the movement of the hand existing in the operation space A and the movement of the hand existing in the operation space B.

[0186] Next, the instruction determination unit 46 refers to the instruction information recorded in the instruction recording unit 47 and determines whether the movement corresponding to the determined movement of the hand is in the instruction information (step F005). If the result is that the movement corresponding to the determined movement of the hand is not in the instruction information (step F005; no), the process returns.

[0187] On the other hand, if it is determined that the movement corresponding to the determined movement of the hand is in the instruction information (step F005; yes), the instruction determination unit 46 determines the instruction corresponding to the movement in the instruction information (step F006). The instruction determination unit 46 outputs the determined instruction to the instruction output unit 48.

[0188] Next, the instruction output unit 48 outputs the above operation information including the information representing the instruction obtained from the instruction determination unit 46 to the instruction generation unit 49 (step F007).

[0189] Next, the instruction generation unit 49 receives the operation information output from the instruction output unit 48 and generates the instruction included in the received operation information (step F008). Thus, in the interface system 100, an instruction corresponding to the movement (gesture) of the user's hand is executed.

[0190] By operating as described above, the interface system 100 according to the first embodiment can perform control as described below, for example.

[0191] (8) Left drag operation

[0192] The user moves the left hand to the left click generation area in the operation space B and moves the right hand in the operation space A. Then, in the interface system 100, the instruction determination unit 46 determines the movement (gesture) of the left and right hands. This movement (gesture) corresponds to the instruction of "left drag operation" in the instruction information. Therefore, in the interface system 100, the instruction determination unit 46 determines the instruction of "left drag operation" and performs a left drag operation linked to the movement of the user's right hand (see Figure 20 A).

[0193] (9) Right drag operation

[0194] The user moves the right hand to the right click generation area in the operation space B and moves the left hand in the operation space A. Then, in the interface system 100, the instruction determination unit 46 determines the movement (gesture) of the left and right hands. This movement (gesture) corresponds to the instruction of "right drag operation" in the instruction information. Therefore, in the interface system 100, the instruction determination unit 46 determines the instruction of "right drag operation" and performs a right drag operation linked to the movement of the user's left hand (see Figure 20 B).

[0195] In addition, in the above description, an example in which the user performs a left drag operation and a right drag operation by moving the left and right hands is described. However, this is just an example, and the instructions executed by the combination of the movements of the user's left and right hands are not limited to the above examples. In this way, by associating the combination of the movements of the user's left and right hands with instructions, the variety of instructions that the user can execute can be increased in the interface system 100.

[0196] In addition, in the above description, for ease of understanding, the operation examples in the spatial processing AB and the operation examples in the above spatial processing B are described separately. However, these processes can also be executed continuously. For example, in the interface system 100, it can also be that first, in the spatial processing B, after the pointer position control unit 45 fixes the pointer P on the operation screen R according to the fixed control information generated by the pointer operation information output unit 44, the above spatial processing AB is executed. That is, for example, the user can also fix the pointer P on the operation screen R by putting one of the left and right hands into the operation space B, and while maintaining this state, move the left and right hands in the operation space A and the operation space B, thereby performing the above left drag operation and right drag operation. In this case, in the interface system 100, the spatial processing B and the spatial processing AB are executed continuously. Thereby, in the interface system 100, it is possible to balance the accurate pointing operation of the user and the expansion of the instruction changes that the user can execute.

[0197] As described above, in the interface system 100 of the first embodiment, the aerial image S indicating the boundary position between the operation space A and the operation space B constituting the virtual space K is projected onto the virtual space K. Thus, the user can visually recognize the boundary position between the operation space A and the operation space B in the virtual space K, and can easily grasp at which position the operation space (mode) is switched.

[0198] Regarding this point, in the above-mentioned conventional device, it is difficult for the user to visually recognize at which position in the virtual plane space the mode is switched. In other words, it is difficult to visually recognize the boundary positions of the respective spaces constituting the virtual plane space (the boundary position between the first space and the second space, and the boundary position between the second space and the third space). The user needs to move the hand to some extent while grasping their positions. In addition, therefore, when the user does not move the hand to some extent, the user cannot grasp the correlation between the pointer and the hand, and it sometimes takes time until the operation starts.

[0199] On the other hand, in the first embodiment, as described above, the user can visually recognize the boundary position between the operation space A and the operation space B in the virtual space K, and can easily grasp at which position the operation space (mode) is switched. In addition, thereby, the user does not need to move the hand to grasp the boundary position for switching the operation space, and can start the operation more quickly than the conventional device.

[0200] In addition, in the existing non-contact pointing system represented by the existing device, the position in the virtual space corresponding to the pressing of the button in the operation screen displayed on the display is difficult for the user to understand, so sometimes it is necessary to add auxiliary display on the operation screen. Or, in order to reliably press the button on the operation screen according to the touch operation in the virtual space, sometimes it is necessary to increase the size of the button on the operation screen. That is, in the existing non-contact pointing system, sometimes it is necessary to reorganize the existing operation screen display software.

[0201] Furthermore, in the existing non-contact pointing system, even if the user keeps his hand still in the air and performs an action (gesture) such as pressing, it is sometimes difficult to specify an accurate position on the operation screen due to the position of the pointer shifting when pressing. Furthermore, in the existing non-contact pointing system, in the operation with continuity such as long-distance movement and scrolling of the pointer, the amount of movement of the user's hand becomes large, and sometimes a large space is required.

[0202] In this regard, in Embodiment 1, as described above, the virtual space K is divided into the operation space A and the operation space B, and in the operation space A, the pointer P can be moved in conjunction with the movement of the user's hand, while in the operation space B, the pointer P is fixed, and the movement (gesture) of the user's hand that generates the instruction is recognized in the state where the pointer P is fixed. Thus, in Embodiment 1, the position of the pointer P is prevented from being shifted during the execution of the movement (gesture) of the hand that generates the instruction. Therefore, the user can not only perform accurate pointing operations when executing instructions, but also directly operate a small operation screen such as a button generated for a PC mouse operation, and there is no need to reorganize the software for displaying the operation screen.

[0203] In addition, in embodiment 1, the user can operate the display device represented by the operation of the pointer P in a non-contact manner. Therefore, even in a working environment where hygiene is important, such as when the user's hands are dirty or the user does not want to dirty their hands, the user can operate in a non-contact manner.

[0204] In addition, in Embodiment 1, the user can execute instructions by moving the hand regardless of the shape of the finger, so there is no need to remember specific finger gestures. In addition, in Embodiment 1, the detection object of the detection device 21 is not limited to the user's hand, so if the detection object is an object other than the user's hand, the user can perform operations even when holding an object, for example.

[0205] Finally, refer to Figure 21, an example of the hardware structure of the device control device 12 included in the interface system 100 of Embodiment 1 is described. Each function of the position acquisition unit 41, the operation space determination unit 43, the pointer operation information output unit 44, the instruction determination unit 46, the instruction output unit 48, and the aerial image generation unit 50 in the device control device 12 is implemented by a processing circuit. The processing circuit can be, as Figure 21 shown in A, dedicated hardware, or can be, as Figure 21 shown in B, a CPU (also referred to as a Central Processing Unit (central processing unit), a central processing device, a processing device, an arithmetic device, a microprocessor, a microcomputer, a processor, or a DSP (Digital Signal Processor: digital signal processor)) 62 that executes a program stored in the memory 63.

[0206] When the processing circuit is dedicated hardware, the processing circuit 61 is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit: application specific integrated circuit), an FPGA (Field Programmable Gate Array: field programmable gate array), or a component formed by combining them. The functions of each part of the position acquisition unit 41, the operation space determination unit 43, the pointer operation information output unit 44, the instruction determination unit 46, the instruction output unit 48, and the aerial image generation unit 50 can be implemented separately by the processing circuit 61, or the functions of each part can be implemented uniformly by the processing circuit 61.

[0207] When the processing circuit is the CPU 62, the functions of the position acquisition unit 41, the operation space determination unit 43, the pointer operation information output unit 44, the instruction determination unit 46, the instruction output unit 48, and the aerial image generation unit 50 are implemented by software, firmware, or a combination of software and firmware. The software and firmware are described as programs and stored in the memory 63. The processing circuit realizes the functions of each part by reading and executing the programs stored in the memory 63. That is, the device control device 12 has a memory for storing, when executed by the processing circuit, the result is to execute, for example Figures 5 - 8 and Figures 18 - 19The programs of the steps shown. In addition, these programs can also be said to be the processes and methods for causing a computer to execute the position acquisition unit 41, the operation space determination unit 43, the pointer operation information output unit 44, the instruction determination unit 46, the instruction output unit 48, and the aerial image generation unit 50. Here, as the memory 63, for example, it is a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically EPROM), a magnetic disk, a floppy disk, an optical disk, a high-density disk, a mini disk, or a DVD (Digital Versatile Disc).

[0208] In addition, regarding the respective functions of the position acquisition unit 41, the operation space determination unit 43, the pointer operation information output unit 44, the instruction determination unit 46, the instruction output unit 48, and the aerial image generation unit 50, a part can be implemented by dedicated hardware and a part can be implemented by software or firmware. For example, regarding the position acquisition unit 41, its function can be implemented by a processing circuit as dedicated hardware, and regarding the operation space determination unit 43, the pointer operation information output unit 44, the instruction determination unit 46, the instruction output unit 48, and the aerial image generation unit 50, their functions can be implemented by the processing circuit reading and executing the programs stored in the memory 63.

[0209] In this way, the processing circuit can implement the above-mentioned respective functions through hardware, software, firmware, or a combination thereof.

[0210] As described above, according to the first embodiment, the interface system 100 has: a detection unit 21 that detects the three-dimensional position of a detection object in the virtual space K divided into a plurality of operation spaces; a position acquisition unit 41 that acquires the three-dimensional position of the detection object detected by the detection unit 21; a projection unit 20 that projects an aerial image S representing the boundary positions of the respective operation spaces in the virtual space K; an operation space determination unit 43 that determines the operation space including the three-dimensional position of the detection object based on the three-dimensional position of the detection object acquired by the position acquisition unit 41 and the boundary positions of the respective operation spaces in the virtual space K; and an operation information output unit 51 that outputs operation information for performing a specified operation on the display device 1 by at least using the determination result of the operation space determination unit 43. Thus, in the interface system 100 of the first embodiment, the boundary positions of the plurality of operation spaces constituting the virtual space K as the operation object of the user can be visually recognized.

[0211] The interface system 100 includes an aerial image generator 50 that generates data representing an aerial image S, and the projection unit 20 projects the aerial image S based on the data generated by the aerial image generator 50. Thus, the interface system 100 according to the first embodiment can project the aerial image S generated based on the data.

[0212] Furthermore, the aerial image generation unit 50 regenerates data representing the aerial image S projected in a projection manner corresponding to the determination result of the operation space determination unit 43, and the projection unit 20 projects the aerial image S based on the data regenerated by the aerial image generation unit 50. Thus, the interface system 100 of Embodiment 1 can project the aerial image S projected in a projection manner corresponding to the determination result of the operation space determination unit 43, and the user can easily understand in which operation space the three-dimensional position of the detection object is included.

[0213] In addition, the operation information output unit 51 includes a pointer operation information output unit 44. When the operation space determination unit 43 determines that the operation space including the three-dimensional position of the detection object is the first operation space, the pointer operation information output unit 44 generates information for moving the pointer P displayed on the screen of the display device 1 in correspondence with the movement of the detection object in the first operation space, and outputs operation information including the generated information. Thus, in the interface system 100 of the first embodiment, the user can move the pointer P in conjunction with the movement of the detection object in the first operation space.

[0214] In addition, the pointer operation information output unit 44 includes information that makes the movement amount or movement speed of the pointer P displayed on the screen of the display device 1 variable in the operation information according to the distance between the three-dimensional position of the detection object included in the first operation space and the boundary surface of the virtual space K represented by the aerial image S in the direction perpendicular to the boundary surface. As a result, the interface system 100 of the first embodiment can make the movement amount or movement speed of the pointer P variable according to the above distance, and the convenience of the user is improved.

[0215] In addition, the operation information output unit 51 includes a sound information output unit, which generates information for outputting a sound corresponding to the three-dimensional position of the detection object in the first operation space or a sound corresponding to the movement of the detection object in the first operation space, and outputs the sound included in the operation information. Thus, the interface system 100 of embodiment 1 can output a sound corresponding to the three-dimensional position of the detection object in the first operation space or the movement of the detection object, and the user can easily grasp the three-dimensional position or movement of the detection object in the first operation space.

[0216] In addition, when the operation space determination unit 43 determines that the operation space including the three-dimensional position of the detection target is the second operation space, the pointer operation information output unit 44 generates information on the pointer P displayed on the screen of the fixed display device 1 and outputs the generated information included in the operation information. Thus, in the interface system 100 of the first embodiment, even if the user moves the detection target to generate an instruction in the second operation space, since the position of the pointer P remains fixed, an accurate pointing operation can be performed when executing the instruction. In addition, an existing operation screen can be directly operated, and there is no need to reorganize the software for operation screen display.

[0217] In addition, the operation information output unit 51 includes: an instruction determination unit 46 that, when the operation space determination unit 43 determines that the operation space including the three-dimensional position of the detection target is the second operation space, determines the movement of the detection target in the second operation space and determines an instruction corresponding to the determined movement of the detection target; and an instruction output unit 48 that outputs operation information including information representing the instruction determined by the instruction determination unit 46. Thus, in the interface system 100 of the first embodiment, the user can execute an instruction corresponding to the movement of the detection target in the second operation space.

[0218] In addition, when the operation space determination unit 43 determines that the operation space including the three-dimensional position of the detection target is the second operation space, the aerial image generation unit 50 generates data of an aerial image SC that represents the lower limit position of the detectable range of the detection unit 21 and represents dividing the second operation space into left and right spaces, and the projection unit 20 projects the aerial image SC based on the data generated by the aerial image generation unit 50. Thus, in the interface system 100 of the first embodiment, the user can easily grasp to what extent the hand can be lowered in the second operation space and can execute an instruction that requires left / right designation.

[0219] In addition, the operation information output unit 51 includes a sound information output unit that generates information for outputting a sound corresponding to the fixation of the pointer P and includes the information in the operation information for output. Thus, the interface system 100 of the first embodiment can output a sound corresponding to the fixation of the pointer P, and the user can easily grasp that the pointer P has been fixed.

[0220] In addition, the operation information output unit 51 includes: an instruction determination unit 46 that, when the operation space determined by the operation space determination unit 43 to include the three-dimensional position of the detection object is the second operation space, determines the movement of the detection object in the second operation space and determines an instruction corresponding to the determined movement of the detection object; and an instruction output unit 48 that outputs information indicating the instruction determined by the instruction determination unit 46 as operation information. The pointer operation information output unit 44 outputs operation information including information on the pointer P displayed on the screen of the fixed display device 1. After the pointer is fixed, the detection unit 21 detects the three-dimensional position of the first detection object and the three-dimensional position of the second detection object. When the operation space determined by the operation space determination unit 43 to include the three-dimensional position of the first detection object is the first operation space and the operation space determined to include the three-dimensional position of the second detection object is the second operation space, the instruction determination unit 46 determines the movement of the first detection object in the first operation space and the movement of the second detection object in the second operation space, and determines an instruction corresponding to the combination of the two determined movements. The instruction output unit 48 outputs operation information including information indicating the instruction corresponding to the combination of the two movements determined by the instruction determination unit 46. Thus, the interface system 100 according to the first embodiment can, after fixing the pointer P, determine an instruction corresponding to the combination of the movement of the first detection object and the movement of the second detection object, can implement an accurate pointing operation by the user, and can increase the variety of instructions that the user can execute.

[0221] In addition, the operation information output unit 51 has: an instruction determination unit 46 that, when the operation space determined by the operation space determination unit 43 to include the three-dimensional position of the detection object is the second operation space, determines the movement of the detection object in the second operation space and determines an instruction corresponding to the determined movement of the detection object; and an instruction output unit 48 that outputs operation information including information indicating the instruction determined by the instruction determination unit 46. Thus, in the interface system 100 according to the first embodiment, the user can execute an instruction corresponding to the movement of the detection object in the second operation space.

[0222] In addition, the aerial image generation unit 50 regenerates data representing the aerial image S projected in a projection manner corresponding to the instruction determined by the instruction determination unit 46, and the projection unit 20 projects the aerial image S based on the data regenerated by the aerial image generation unit 50 when the instruction determined by the instruction determination unit 46 is executed. Thus, the interface system 100 according to the first embodiment can project the aerial image S projected in a projection manner corresponding to the instruction determined by the instruction determination unit 46, and the user can easily grasp which instruction has been determined.

[0223] In addition, the operation information output unit 51 includes a sound information output unit that generates information for outputting a sound corresponding to the instruction determined by the instruction determination unit 46 and outputs the information in the operation information. Accordingly, the interface system 100 according to Embodiment 1 can output a sound corresponding to the instruction determined by the instruction determination unit 46, and the user can easily grasp which instruction has been determined.

[0224] In addition, the three-dimensional position of the first detection object and the three-dimensional position of the second detection object are detected by the detection unit 21. When the operation space determination unit 43 determines that the operation space including the three-dimensional position of the first detection object is the first operation space and the operation space including the three-dimensional position of the second detection object is the second operation space, the instruction determination unit 46 determines the movement of the first detection object in the first operation space and the movement of the second detection object in the second operation space, and determines an instruction corresponding to the combination of the two determined movements. The instruction output unit 48 outputs operation information including information indicating the instruction corresponding to the combination of the two movements determined by the instruction determination unit 46. Accordingly, the interface system 100 according to Embodiment 1 can determine an instruction corresponding to the combination of the movement of the first detection object and the movement of the second detection object, and can increase the variety of instructions that the user can execute.

[0225] In addition, according to Embodiment 1, the device control device 12 controls the interface device 2 having: a detection unit 21 that detects the three-dimensional position of a detection object in a virtual space K divided into a plurality of operation spaces; and a projection unit 20 that projects an aerial image S indicating the boundary positions of the respective operation spaces in the virtual space K. The device control device 12 includes: a position acquisition unit 41 that acquires the three-dimensional position of the detection object detected by the detection unit 21; an operation space determination unit 43 that determines the operation space including the three-dimensional position of the detection object based on the three-dimensional position of the detection object acquired by the position acquisition unit 41 and the boundary positions of the respective operation spaces in the virtual space K; and an operation information output unit 51 that outputs operation information for performing a specified operation on the display device 1 using at least the determination result of the operation space determination unit 43. Accordingly, in the device control device 12 according to Embodiment 1, the boundary positions of the plurality of operation spaces constituting the virtual space K, which is the operation object of the user, can be visually recognized.

[0226] In addition, any structural element of the embodiments of the present disclosure can be deformed or any structural element of the embodiments can be omitted. For example, in the above description, the case where the aerial image S is composed of a linear (straight line) - shaped figure has been described as an example. However, the aerial image S is not limited thereto and can be composed of any figure. In addition, the aerial image S is not limited to a figure and can be composed of any character or character string. Further, regarding the color, blinking pattern (number of blinks) of the aerial image S, and the sound output when an instruction is executed, the examples given above are merely examples, and it can also be a color, blinking pattern, and sound other than those described above. In addition, the control of the color of the aerial image S and the control of the blinking pattern (number of blinks) can also be executed in units of pixels of the aerial image S. For example, the projection device 20 can change the color or brightness of the entire aerial image S (all pixels of the aerial image S) in the same manner, or can change the color or brightness of any part (pixels of any part of the aerial image S) of the aerial image S. Additionally, by changing the color or brightness of any part of the aerial image S, the projection device 20 can increase the variation in the projection method of the aerial image S, such as adding any gradient to the aerial image S.

[0227] Industrial Applicability

[0228] The present disclosure can visually recognize the boundary positions of a plurality of operation spaces that constitute a virtual space which is an object of user operation, and is suitable for use in an interface system.

[0229] Reference Numeral Explanation

[0230] 1: Display device; 2: Interface device; 10: Display; 11: Display control device; 12: Device control device (control device); 20: Projection device (projection unit); 21: Detection device (detection unit); 31: Aerial image projection unit; 32: Position detection unit; 41: Position acquisition unit (acquisition unit); 42: Boundary position recording unit; 43: Operation space determination unit (determination unit); 44: Pointer operation information output unit; 45: Pointer position control unit; 46: Instruction determination unit; 47: Instruction recording unit; 48: Instruction output unit; 49: Instruction generation unit; 50: Aerial image generation unit; 51: Operation information output unit; 61: Processing circuit; 62: CPU; 63: Memory; 100: Interface system; 201: Light source; 202: Beam splitter; 203: Retroreflective material; A: Operation space; B: Operation space; K: Virtual space; P: Pointer; R: Operation screen; S: Aerial image; SC: Aerial image; SE: Aerial image.

Claims

1. An interface system, wherein, the interface system has: a detection unit that detects the three-dimensional position of a detection object in a virtual space divided into a plurality of operation spaces; an acquisition unit that acquires the three-dimensional position of the detection object detected by the detection unit; a projection unit that projects an aerial image representing the boundary positions of the respective operation spaces in the virtual space; a determination unit that determines the operation space containing the three-dimensional position of the detection object based on the three-dimensional position of the detection object acquired by the acquisition unit and the boundary positions of the respective operation spaces in the virtual space; and an operation information output unit that outputs operation information for performing a specified operation on a display device, using at least the determination result of the determination unit.

2. The interface system according to claim 1, wherein, the interface system has an aerial image generation unit that generates data representing the aerial image, and the projection unit projects an aerial image based on the data generated by the aerial image generation unit.

3. The interface system according to claim 2, wherein, the aerial image generation unit regenerates data representing an aerial image projected in a projection manner corresponding to the determination result of the determination unit, and the projection unit projects an aerial image based on the data regenerated by the aerial image generation unit.

4. The interface system according to claim 2 or 3, wherein, the operation information output unit includes a pointer operation information output unit that, when the operation space determined by the determination unit to contain the three-dimensional position of the detection object is the first operation space, generates information for causing a pointer displayed on the screen of the display device to move corresponding to the movement of the detection object in the first operation space, and outputs the operation information including the generated information.

5. The interface system according to claim 4, wherein, the pointer operation information output unit includes, in the operation information output, information for making the movement amount or movement speed of the pointer displayed on the screen of the display device variable, based on the distance in a direction perpendicular to the boundary surface of the virtual space represented by the aerial image, between the three-dimensional position of the detection object included in the first operation space and the boundary surface.

6. The interface system according to claim 4, wherein, the operation information output unit includes a sound information output unit that generates information for outputting a sound corresponding to the three-dimensional position of the detection object in the first operation space or a sound corresponding to the movement of the detection object in the first operation space, and includes the information in the operation information output.

7. The interface system according to claim 4, wherein, when the operation space determined by the determination unit to contain the three-dimensional position of the detection object is the second operation space, the pointer operation information output unit generates information for fixing the pointer displayed on the screen of the display device, and includes the generated information in the operation information output.

8. The interface system according to claim 7, wherein, The operation information output unit includes: An instruction determination unit that, when the operation space including the three-dimensional position of the detection object is determined by the determination unit to be the second operation space, determines the movement of the detection object in the second operation space and determines an instruction corresponding to the determined movement of the detection object; And An instruction output unit that outputs the operation information including information representing the instruction determined by the instruction determination unit.

9. The interface system according to claim 7, wherein when the operation space including the three-dimensional position of the detection object is determined by the determination unit to be the second operation space, the aerial image generation unit generates data of an aerial image representing the lower limit position of the detectable range of the detection unit and representing the division of the second operation space into left and right spaces, and the projection unit projects the aerial image based on the data generated by the aerial image generation unit.

10. The interface system according to claim 7, wherein the operation information output unit includes a sound information output unit that generates information for outputting a sound corresponding to the fixation of the pointer and outputs it included in the operation information.

11. The interface system according to claim 7, wherein the operation information output unit includes: An instruction determination unit that, when the operation space including the three-dimensional position of the detection object is determined by the determination unit to be the second operation space, determines the movement of the detection object in the second operation space and determines an instruction corresponding to the determined movement of the detection object; And An instruction output unit that outputs the operation information including information representing the instruction determined by the instruction determination unit, after the operation information including information for fixing the pointer displayed on the screen of the display device is output by the pointer operation information output unit and the pointer is fixed, the three-dimensional positions of the first detection object and the second detection object are detected by the detection unit, when the operation space including the three-dimensional position of the first detection object is determined by the determination unit to be the first operation space and the operation space including the three-dimensional position of the second detection object is determined to be the second operation space, the instruction determination unit determines the movement of the first detection object in the first operation space and the movement of the second detection object in the second operation space, and determines an instruction corresponding to the combination of the two determined movements, and the instruction output unit outputs the operation information including information representing the instruction corresponding to the combination of the two movements determined by the instruction determination unit.

12. The interface system according to claim 4, wherein the operation information output unit includes: An instruction determination unit that, when the operation space including the three-dimensional position of the detection object is determined by the determination unit to be the second operation space, determines the movement of the detection object in the second operation space and determines an instruction corresponding to the determined movement of the detection object; And An instruction output unit that outputs the operation information including information representing an instruction determined by the instruction determination unit.

13. The interface system according to claim 12, wherein, the aerial image generation unit regenerates data representing an aerial image projected in a projection manner corresponding to the instruction determined by the instruction determination unit, the projection unit projects an aerial image based on the data regenerated by the aerial image generation unit when the instruction determined by the instruction determination unit is executed.

14. The interface system according to claim 12, wherein, the operation information output unit includes a sound information output unit that generates information for outputting a sound corresponding to the instruction determined by the instruction determination unit and outputs it included in the operation information.

15. The interface system according to claim 12, wherein, the three-dimensional positions of the first detection object and the second detection object are detected by the detection unit, when the determination unit determines that the operation space including the three-dimensional position of the first detection object is the first operation space and the operation space including the three-dimensional position of the second detection object is the second operation space, the instruction determination unit determines the movement of the first detection object in the first operation space and the movement of the second detection object in the second operation space, and determines an instruction corresponding to the combination of the two determined movements, the instruction output unit outputs the operation information including information representing the instruction corresponding to the combination of the two movements determined by the instruction determination unit.

16. A control device that controls an interface device having: a detection unit that detects the three-dimensional position of a detection object in a virtual space divided into a plurality of operation spaces; and a projection unit that projects an aerial image representing the boundary positions of the respective operation spaces in the virtual space, wherein, the control device has: an acquisition unit that acquires the three-dimensional position of the detection object detected by the detection unit; a determination unit that determines the operation space including the three-dimensional position of the detection object based on the three-dimensional position of the detection object acquired by the acquisition unit and the boundary positions of the respective operation spaces in the virtual space; and an operation information output unit that outputs operation information for performing a prescribed operation on a display device at least using the determination result of the determination unit.

17. An operation assistance method executed by an interface system, wherein, the operation assistance method has the following steps: the projection unit projects an aerial image representing the boundary positions of the respective operation spaces in a virtual space divided into a plurality of operation spaces; the detection unit detects the three-dimensional position of a detection object in the virtual space; the acquisition unit acquires the three-dimensional position of the detection object detected by the detection unit; the determination unit determines the operation space including the three-dimensional position of the detection object based on the three-dimensional position of the detection object acquired by the acquisition unit and the boundary positions of the respective operation spaces in the virtual space; and The operation information output unit outputs operation information for performing a specified operation on the display device, using at least the determination result of the determination unit.

Images