Operation input device, operation input method, program, and recording medium

Abstract

To provide an operation input device that is simple, immersive, and enables intuitive operation. [Solution] An operation input device 10 comprising: an image display unit 11 having a display surface 11a for displaying an image; a 3D plate 12 that enables the image to be projected as a real image onto an image-forming surface 16 in the air; a detection unit 13 that sets a three-dimensional sensing area 17 including the air image-forming surface 16 and detects the movement of the operator's hand within the three-dimensional sensing area 17; and a control unit that changes the image according to the movement of the operator's hand.

Description

Technical Field

[0001] The present invention relates to an operation input device, an operation input method, a program, and a recording medium.

Background Art

[0002] A medical image display device has been proposed that displays a three-dimensional medical image on a tablet terminal and accepts a user's operation input for the three-dimensional image (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The technique described in Patent Document 1 is useful because it can be easily used anywhere in a medical facility such as a hospital without using a large-scale device. On the other hand, if it can further provide a sense of presence and intuitive operations, it will be more useful and practical.

[0005] Therefore, an object of the present invention is to provide an operation input device, an operation input method, a program, and a recording medium that can simply realize operations with a sense of presence and intuitive operations.

Means for Solving the Problems

[0006] To achieve the above object, the operation input device of the present invention includes a video display unit having a display surface for displaying a video, a 3D plate that can form a real image of the video on an aerial imaging surface, a detection unit that sets a three-dimensional sensing area including the aerial imaging surface and detects the movement of the hand of an operator (user) within the three-dimensional sensing area, A control unit that changes the image in accordance with the movements of the operator, Includes.

[0007] The operation input method of the present invention is A video display process in which an image is displayed on the display surface of the video display unit, An aerial imaging process in which the aforementioned image is formed as a real image on an imaging surface in the air using a 3D plate, A detection step includes setting a three-dimensional sensing region including the aerial imaging surface and detecting the movement of the operator's hand within the three-dimensional sensing region, A control step that changes the image in accordance with the hand movements of the operator, Includes.

[0008] The program of the present invention causes a computer to execute the operation input method of the present invention.

[0009] The recording medium of the present invention is a computer-readable recording medium characterized by having the program of the present invention recorded on it. [Effects of the Invention]

[0010] According to the present invention, the function of the 3D plate allows for the display of an image on an aerial imaging surface that is actually a planar (two-dimensional) image but is perceived as three-dimensional by the operator. By detecting the operator's hand movements and changing the image accordingly, it is possible to provide an operation input device, operation input method, program, and recording medium that enable simple, realistic, and intuitive operation. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1(A) is a block diagram showing an example of the configuration of the operation input device of the present invention, and Figure 1(B) is a flowchart showing an example of the operation input method of the present invention. [Figure 2] Figure 2 is a schematic cross-sectional view showing an example of the configuration of the operation input device of the present invention. [Figure 3] Figure 3 shows an example of a spherical image displayed on the video display surface of the present invention. [Figure 4] Figure 4 is a diagram showing an example of operation input by the movement of the operator's palm in the present invention. [Figure 5] Figure 5 is a diagram showing an example of a drag operation in the present invention. [Figure 6] Figure 6 is a diagram showing another example of a drag operation in the present invention. [Figure 7] Figure 7 is a diagram showing another example of operation input by the movement of the operator's palm in the present invention. [Figure 8] Figure 8 is a diagram showing yet another example of operation input by the movement of the operator's palm in the present invention. [Figure 9] Figure 9 is a diagram showing yet another example of operation input by the movement of the operator's palm in the present invention. [Figure 10] Figure 10 is a diagram showing yet another example of operation input by the movement of the operator's palm in the present invention. [Figure 11] Figure 11 is a schematic cross-sectional view showing another example of the configuration of the operation input device of the present invention. [Figure 12] Figure 12 is a diagram showing an example of operation input by the movement of the operator's fingers in the present invention. [Figure 13] Figure 13 is a diagram showing another example of operation input by the movement of the operator's fingers in the present invention. [Figure 14] Figure 14 is a diagram showing an example of the movement of the operator's fingers in the present invention. [Figure 15] Figure 15 is a diagram showing an example of a rotation operation input in the present invention.

Embodiments for Carrying Out the Invention

[0012] The operation input device, operation input method, program, and recording medium of the present invention will be described with reference to examples. However, the present invention is not limited to the following examples. In Figures 1 to 15 below, the same parts are denoted by the same reference numerals, and their descriptions may be omitted. Also, in the drawings, for the sake of explanation, the structure of each part may be shown in a simplified manner, and the dimensional ratios of each part may be shown schematically and may differ from the actual dimensions. Furthermore, unless otherwise specified, the descriptions of each embodiment can be used interchangeably.

[0013] (Embodiment 1: Operation input device and operation input method) Figure 1(A) is a block diagram showing an example of the configuration of the operation input device of the present invention. As shown in Figure 1(A), the operation input device 10 of this embodiment includes a video display unit 11, a 3D plate 12, a detection unit 13, and a control unit 14.

[0014] Referring to the schematic cross-sectional view in Figure 2, an example of the configuration of the operation input device 10 will be described. As shown in Figure 2, the operation input device 10 in this example includes a video display unit 11, a 3D plate 12, a detection unit 13, and a housing 15. The housing 15 is a hollow rectangular parallelepiped with a rectangular opening at the top. The operation input device 10 also includes a control unit 14 (not shown) located inside the housing 15. In this example of the operation input device 10, the presence or absence of the housing 15 is optional, and the operation input device 10 does not need to have one.

[0015] The video display unit 11 has a display surface 11a for displaying video. For example, a conventionally known video display device such as a liquid crystal display can be used as the video display unit 11.

[0016] The 3D plate 12 enables the image to be projected as a real image onto the image-forming surface 16 in the air. The 3D plate 12 is, for example, a flat plate made of a glass material or a light-transmitting resin material having a mirror structure with two layers arranged perpendicularly to each other, and is placed in the upper opening of the housing 15. A commercially available product may be used as the 3D plate 12. An example of such a commercially available product is the ASKA3D plate manufactured by Asukanet Co., Ltd.

[0017] The angle θ1 between the 3D plate 12 and the aerial image plane 16 is not particularly limited and can be, for example, 30° to 60°, 40° to 50°, or 45°. Similarly, the angle θ2 between the display surface 11a of the video display unit 11 and the 3D plate 12 is not particularly limited and can be, for example, approximately the same as angle θ1. The video display unit 11 is fixed inside the housing 15 so that the angle θ2 is of a desired size.

[0018] The detection unit 13 sets a three-dimensional sensing area 17 that includes the aerial imaging surface 16 and detects the operator's hand movements within the three-dimensional sensing area 17. Since the three-dimensional sensing area 17 is not visible to the operator, its boundary is shown with a dashed line in Figure 2. Examples of the detection unit 13 include a 3D camera, a 3D TOF sensor, and other range sensors.

[0019] The 3D sensing area 17 may have any shape as long as it includes the aerial imaging surface 16, for example, it may be a rectangular parallelepiped, cubic, cylindrical, or elliptical shape. The size (range) of the 3D sensing area 17 is preferably about 1.1 to 1.5 times each side of the aerial imaging surface 16 in the planar direction parallel to the aerial imaging surface 16, and about 20 mm to 100 mm on the operator side and the opposite side in the depth direction perpendicular to the aerial imaging surface 16.

[0020] Figure 2 shows an example in which the detection unit 13 is installed approximately 30 cm above the 3D plate 12. However, the installation position of the detection unit 13 is not limited to the example shown in Figure 2. The detection unit 13 may be installed at any position as long as it is possible to set a three-dimensional sensing area 17 including the aerial imaging surface 16.

[0021] The control unit 14 changes the image in accordance with the operator's hand movements. Specifically, the control unit 14 recognizes and controls the operation of the image display unit 11 and the detection unit 13 so as to change the image in accordance with the operator's hand movements. The control unit 14 may be, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an APU (Accelerated Processing Unit), a microprocessor, a microcontroller (MCU), or a combination of several of these.

[0022] Figure 1(B) shows a flowchart of the operation input method in this embodiment.

[0023] First, an image is displayed on the display surface 11a of the video display unit 11 (step S1).

[0024] Next, the 3D plate 12 is used to project the image onto the aerial imaging surface 16 as a real image (step S2).

[0025] Next, the detection unit 13 sets a three-dimensional sensing region 17 that includes the aerial imaging surface 16, and detects the operator's hand movements within the three-dimensional sensing region 17 (step S3).

[0026] Next, the control unit 14 changes the image in accordance with the operator's hand movements (step S4).

[0027] From here on, the details of the aforementioned detection process (step S3) and control process (step S4) will be explained with examples.

[0028] As shown in Figure 2, the detection unit 13 and the control unit 14 may set a filtering surface 18 on the operator side of the 3D sensing area 17. The filtering surface 18 is not visible to the operator and is therefore shown as a dashed line in Figure 2. The filtering surface 18 may be set so that a part of it extends into the operator-side portion of the 3D sensing area 17, for example, as shown in Figure 2. When a part of the operator other than the hand enters the 3D sensing area 17 beyond the filtering surface 18, the control unit 14 excludes that part from detection. This suppresses erroneous operation inputs caused by the detection of parts other than the operator's hand and reduces the computational processing load on the control unit 14, such as the CPU.

[0029] Next, the detection process (step S3) and control process (step S4) described above will be explained in more detail, separately for cases where the operator's hand movement within the 3D sensing area 17 is a palm movement and cases where it is a finger movement.

[0030] First, let's explain the case where the operator's hand movement is a palm movement. When the operator's hand movement is a palm movement, the control unit 14 may control the drive of the video display unit 11 so that the image displayed on the display surface 11a of the video display unit 11 includes a spherical image 19 to assist in operation, as shown in Figure 3. Due to the function of the 3D plate 12 described above, the image is formed as a real image on the aerial image forming surface 16. From here on, we will explain using the case where the spherical image 19 is included in the image as an example, but the operator can choose whether or not to include the spherical image 19 in the image. Even if the operator chooses not to include the spherical image 19 in the image, they can still perform the same operation input as when the spherical image 19 is included, as will be explained later, by moving their palm.

[0031] The size of the spherical image 19 is not particularly limited, but for example, as shown in Figure 3(A), it may be initially set to a diameter of about 20 cm so that the entire image fits on the display surface 11a (airborne image plane 16) of the video display unit 11, and the operator may choose to enlarge it so that part of it does not fit on the display surface 11a (airborne image plane 16) of the video display unit 11, as shown in Figure 3(B), or conversely, reduce it, as shown in Figure 3(C).

[0032] Next, we will explain in more detail the cases where the palm movement is performed by one hand and when it is performed by both hands.

[0033] First, let's explain the case where the movement of the palm within the three-dimensional sensing area is performed by the operator's one hand. As shown in Figure 4, in this case, the control unit 14 may provide a neutral area 20a in the center of the spherical image 19 where operation input becomes difficult, excluding that area from detection. The size of the neutral area 20a is, for example, about 1 cm to 5 cm in diameter. The control unit 14 may also provide a neutral area 20b on the outer periphery of the spherical image 19, excluding that area from detection and allowing the operator's hand (palm) to be withdrawn from the three-dimensional sensing area, for the purpose of suppressing erroneous operation input. The neutral area 20b may be, for example, in the range of about 1 cm to 5 cm from the outer edge of the spherical image 19. Since the neutral areas 20a and 20b are not visible to the operator, their boundaries are shown with dashed lines in Figure 4. Alternatively, the control unit 14 may control the drive of the video display unit 11 so that the neutral area 20a and the neutral area 20b are colored gray or the like on its display surface 11a so that they can be seen by the operator.

[0034] When the palm movement is performed by the operator's one hand, the operation input begins when the detection unit 13 detects that the operator's hand (palm of one hand) has been inserted into the 3D sensing area. This insertion of the operator's hand may be performed from any direction, such as the left, right, top, bottom, or front (operator) side of the 3D sensing area, and this point is the same thereafter. When the control unit 14 recognizes that the detection unit 13 has detected that the operator's palm of one hand has been inserted into the 3D sensing area, it controls the drive of the image display unit 11 so that a yellow spherical image 19 is displayed on its display surface 11a. As a result, due to the function of the 3D plate 12, the operator perceives the spherical image 19 projected onto the aerial image plane 16 in three dimensions.

[0035] Then, as shown by reference numerals 21a, 22a, or 23a in Figure 4, when the control unit 14 recognizes that the detection unit 13 has detected that the palm of one of the operator's hands is in a position to touch the spherical image 19, it controls the operation of the video display unit 11 to change the color of the spherical image 19 to blue. The operator understands that dragging is possible by the change in color of the spherical image 19 projected onto the aerial imaging surface 16. In Figure 4, within the three-dimensional sensing area, the state where the operator is touching the right side and top of the spherical image 19 (21a and 23a) shows the palm of the operator's right hand, and the state where the operator is touching the left side of the spherical image 19 (22a) shows the palm of the operator's left hand. The operator can choose to use either the palm of their right hand or the palm of their left hand depending on their dominant hand, the direction of hand insertion, etc.

[0036] Referring to Figures 5 and 6, the drag operation will be explained using the example of a case where the image displayed on the display surface 11a of the image display unit 11 and formed on the aerial image-forming surface 16 is an image of the brain. Hereafter, the left-right direction in each drawing may be referred to as the X-axis direction, the up-down direction as the Y-axis direction, and the depth direction (direction perpendicular to the plane of the paper) as the Z-axis direction. Alternatively, in the present invention, within the three-dimensional sensing area, the direction parallel to the ground on which the operation input device 10 is installed may be referred to as the X-axis direction, the direction perpendicular to the ground as the Y-axis direction, and the depth direction perpendicular to them as the Z-axis direction.

[0037] When the operator has the palm of one hand touching the right side of the spherical image 19, and moves the palm in an arc from the back (opposite the operator) to the front (operator) as shown from the top to the bottom of Figure 5 (movement from 21a1 to 21a2), or moves the palm in an arc from the front (operator) to the back (opposite the operator) as shown from the bottom to the top of Figure 5 (movement from 21a2 to 21a1), the control unit 14 controls the drive of the image display unit 11 so that the image of the brain displayed on its display surface 11a rotates around the Y axis as if dragged by the movement of the palm. As a result, due to the function of the 3D plate 12, the operator perceives the image of the brain formed on the aerial image plane 16 rotating around the Y axis in three dimensions. Furthermore, when the palm of one hand is touching the right side of the spherical image 19, the control unit 14 controls the drive of the image display unit 11 so that when the operator moves the palm in an arc from top to bottom (movement from 21a3 to 21a4) as if moving from the top to the bottom of Figure 6, or when the palm moves from bottom to top (movement from 21a4 to 21a3) as if moving from the bottom to the top of Figure 6, the image display unit 11 rotates the brain image displayed on its display surface 11a counterclockwise or clockwise around the Z axis, as if being dragged along by the movement of the palm. Note that the state when the palm of one hand is touching the left side of the spherical image 19 is the same as the state when it is touching the right side described above, so the explanation is omitted.

[0038] Although not shown in the diagram, when the palm of one hand is touching the upper part of the spherical image 19, the control unit 14 controls the drive of the image display unit 11 so that when the operator moves the palm in an arc from left to right or from right to left in an arc, the image of the brain displayed on its display surface 11a rotates around the X-axis in accordance with the movement of the palm. As a result, the function of the 3D plate 12 allows the operator to perceive the rotation of the image of the brain projected onto the aerial image plane 16 around the X-axis in three dimensions. It is also possible to control the image in the same way as when the palm of one hand is touching the upper part of the spherical image 19.

[0039] Furthermore, when the control unit 14 recognizes that the detection unit 13 has detected that the palm of one hand has moved away from the spherical image 19 without touching it, as shown by reference numerals 21b, 22b, or 23b in Figure 4, the control unit 14 controls the operation of the video display unit 11 so that the color of the spherical image 19 displayed on its display surface 11a changes from blue back to yellow. In this state, if the operator moves the palm of one hand in the same way as when touching the spherical image 19, the brain image is controlled to rotate continuously around the X, Y, or Z axis by an amount (angle) corresponding to the magnitude of the palm movement.

[0040] For convenience, the explanation so far has used independent rotations around each axis as examples. However, depending on the movement of the operator's hand, the system can be controlled to simultaneously apply two or three rotations to the image: rotation around the X axis, rotation around the Y axis, and rotation around the Z axis.

[0041] Next, with reference to Figures 7 to 10, we will explain the case where the palm movements within the three-dimensional sensing area are performed by both hands of the operator.

[0042] When the palm movements are from both hands of the operator, the operation input begins when the detection unit 13 detects that the operator's hands (both palms) have been inserted into the three-dimensional sensing area. Then, as shown by reference numerals 24R and 24L in the upper diagram of Figure 7, when the control unit 14 recognizes that the detection unit 13 has detected that the operator's palms are in a position to touch the spherical image 19, it controls the operation of the video display unit 11 so that it displays the blue spherical image 19 on its display surface 11a. Furthermore, as shown from the upper diagram to the lower diagram of Figure 7, when the operator moves the right palm to the right (movement from 24R to 25R) and the left palm to the left (movement from 24L to 25L) to widen the distance between the palms of both hands, the control unit 14 controls the operation of the video display unit 11 so that it enlarges the brain image displayed on its display surface 11a in accordance with the movement of the palms. As a result, the 3D plate 12 allows the operator to perceive the image of the brain projected onto the aerial imaging surface 16 as expanding in three dimensions. When the control unit 14 recognizes that the detection unit 13 has detected that the operator's palms have moved away from the spherical image 19 without touching it, as shown in the lower diagram of Figure 7, the control unit 14 controls the operation of the image display unit 11 to change the color of the spherical image 19 to yellow. On the other hand, as shown from the lower diagram to the upper diagram of Figure 7, when the operator moves the right palm to the left (movement from 25R to 24R) and the left palm to the right (movement from 25L to 24L) to reduce the distance between the palms, the control unit 14 controls the operation of the image display unit 11 to reduce the brain image displayed on its display surface 11a in accordance with the movement of the palms. As shown in Figure 7, in these cases, a bar (rod) whose length changes according to the scaling of the image may be displayed at the bottom of the display surface 11a (aerial image plane 16) of the image display unit 11.

[0043] Furthermore, as shown in Figure 8 from the top to the bottom, the control unit 14 controls the operation of the image display unit 11 so that when the operator moves the right palm to the left (movement from 26R to 27R) and the left palm to the left (movement from 26L to 27L) without changing the distance between the palms of both hands, the image display unit 11 moves to the left in accordance with the movement of the palms. As a result, due to the function of the 3D plate 12, the operator perceives the image of the brain projected onto the aerial image plane 16 moving to the left in three dimensions. Meanwhile, the control unit 14 controls the operation of the video display unit 11 so that, as shown from the lower diagram to the upper diagram in Figure 8, when the operator moves the right palm to the right (movement from 27R to 26R) and the left palm to the right (movement from 27L to 26L) without changing the distance between the palms of both hands, the video display unit 11 moves the image of the brain displayed on its display surface 11a to the right in accordance with the movement of the palms.

[0044] Then, as shown from the top diagram to the bottom diagram in Figure 9, when the operator moves the right palm downward (from 28R to 29R) and the left palm downward (from 28L to 29L) without changing the distance between the palms of both hands, the control unit 14 controls the drive of the image display unit 11 so that the image of the brain displayed on its display surface 11a moves downward in accordance with the movement of the palms. As a result, due to the function of the 3D plate 12, the operator perceives the image of the brain formed on the aerial imaging surface 16 moving downward in three dimensions. On the other hand, as shown from the bottom diagram to the top diagram in Figure 9, when the operator moves the right palm upward (from 29R to 28R) and the left palm upward (from 29L to 28L) without changing the distance between the palms of both hands, the control unit 14 controls the drive of the image display unit 11 so that the image of the brain displayed on its display surface 11a moves upward in accordance with the movement of the palms.

[0045] Furthermore, as indicated by reference numerals 30R and 30L in the upper diagram of Figure 10, when the control unit 14 recognizes that the detection unit 13 has detected that the operator's palms are in a position away from the spherical image 19 without touching it, the control unit 14 controls the drive of the video display unit 11 so that it displays the yellow spherical image 19 on its display surface 11a. When the operator moves both palms clockwise in an arc (movement from 30R and 30L to 31R and 31L) as shown from the upper diagram to the lower diagram of Figure 10, or moves both palms counterclockwise in an arc (movement from 31R and 31L to 30R and 30L) as shown from the lower diagram to the upper diagram, the control unit 14 controls the drive of the video display unit 11 so that the brain image displayed on its display surface 11a rotates around the Y axis, dragged along by the movement of the palms. As a result, the function of the 3D plate 12 allows the operator to perceive the brain image projected onto the aerial imaging surface 16 as rotating around the Y-axis in three dimensions. Although not shown in the illustration, when the operator moves both palms in an arc from top to bottom or from bottom to top while the palms of both hands are not touching the spherical image 19, the brain image is controlled to rotate left or right around the Z-axis in accordance with the movement of the palms. The operator can also control rotation around the X-axis using the aforementioned movement of one hand.

[0046] As explained above, according to the present invention, the function of the 3D plate 12 allows for the display of an image on the aerial imaging surface 16 that is actually a planar (two-dimensional) image but is perceived as three-dimensional by the operator. By detecting the operator's hand movements and changing the image accordingly, it is possible to achieve simple, realistic, and intuitive operation. Furthermore, in conventional operation input devices such as the medical image display device described in Patent Document 1 (Patent No. 5747007), the image (video) of the object to be operated may be obscured by the user's (operator's) hand, making it difficult for the operator to see. In contrast, according to the present invention, by utilizing aerial imaging technology, the target image can be operated without being obscured by the operator's hand. This effect can also be obtained when the operator's hand movements are finger movements, as described later, but it is particularly noticeable when the operator's hand movements are palm movements.

[0047] Next, we will explain the case where the operator's hand movements are finger movements.

[0048] When the operator's hand movement is a finger movement, the control unit 14 may set a hover plane 40 parallel to the aerial image plane 16 on the operator's side relative to the aerial image plane 16, as shown in Figure 11. The hover plane 40 is not visible to the operator and is therefore shown with a dashed line in Figure 11. The distance between the aerial image plane 16 and the hover plane 40 can be freely set by the operator, for example, between 5 cm and 10 cm.

[0049] As shown in Figure 12, the operator can perform the following operations using their fingers (fingertips) 41. As mentioned above, the hover surface 40 is not visible to the operator, so its boundary is shown with a dashed line in Figure 12. In Figure 12, the fingers (fingertips) 41 of the operator's right hand are shown, but the operator can also use the fingers (fingertips) of their left hand depending on their dominant hand, the direction of hand insertion, etc.

[0050] First, as a pointing operation, when the control unit 14 recognizes that the detection unit 13 has detected that the operator's fingertip 41 is in contact with the hover surface 40, the control unit 14 controls the drive of the video display unit 11 so that the cursor (gray circle) is displayed on its display surface 11a at a position corresponding to directly below the operator's fingertip 41 on the aerial image projection surface 16.

[0051] Then, the operator presses their fingertip 41 from the hover surface 40 to the aerial image-forming surface 16 and quickly pulls it back to perform a touch (click) operation. That is, when the control unit 14 recognizes that the detection unit 13 has detected the operator making the aforementioned hand (fingertip 41) movement, it changes the color of the cursor displayed on the display surface 11a of the video display unit 11 to green and controls the drive of the video displayed on the display surface 11a so that the cursor is in the center of the display surface 11a. The operation input device 10 may also emit a touch sound (click sound) when this touch (click) operation is performed. Conventional known means may be used to generate the touch sound (click sound).

[0052] On the other hand, if the operator keeps their fingertip 41 in contact with the aerial image plane 16 for a predetermined time (for example, 0.8 seconds), the drag operation becomes possible. That is, when the control unit 14 recognizes that the detection unit 13 has detected the operator stopping their fingertip 41, it controls the operation of the video display unit 11 so that the color of the cursor displayed on its display surface 11a changes to blue. Subsequently, the control unit 14 controls the operation of the video display unit 11 so that the image displayed on its display surface 11a moves in accordance with the movement of the operator's fingertip 41 detected by the detection unit 13.

[0053] Furthermore, the operator can perform the following operations using two-finger movements.

[0054] As shown in Figure 13, when the control unit 14 recognizes that the detection unit 13 has detected that the operator's fingertips (two fingers) 42 are touching the hover surface 40, the control unit 14 controls the operation of the video display unit 11 so that the cursor (gray bar) 44 is displayed on its display surface 11a at a position corresponding to directly below the operator's fingertips (two fingers) 42 on the aerial image projection surface 16. As mentioned above, the hover surface 40 is not visible to the operator, so its boundary is shown with a dashed line in Figure 13. Then, in this state, as shown in Figure 14(A) or Figure 14(B), when the operator moves either of the two fingers up or down, the control unit 14 controls the operation of the video display unit 11 so that the image displayed on its display surface 11a rotates left or right around the Z axis. As a result, the 3D plate 12 allows the operator to perceive the image projected onto the aerial imaging surface 16 as rotating around the Z-axis in three dimensions. However, the central region of the hover surface 40 is a neutral region 43, similar to the spherical image in the aforementioned operation using the operator's palm movements. Even if the operator moves their fingertips within this neutral region 43, the image does not change. The size of the neutral region 43 is, for example, about 1cm x 1cm to 5cm x 5cm. Since the neutral region 43 is not visible to the operator, its boundary is shown with a dashed line in Figure 13. In Figures 13 and 14, the operator's left hand fingertips (two fingers) are shown, but the operator can also use the fingertips (two fingers) of their right hand depending on their dominant hand, the direction of hand insertion, etc.

[0055] Meanwhile, when the control unit 14 recognizes that the detection unit 13 has detected that the operator has pressed their fingertips (two fingers) 42 from the hover surface 40 to the aerial image-forming surface 16, the control unit 14 controls the operation of the image display unit 11 so that the color of the cursor displayed on its display surface 11a changes to green. Furthermore, in this state, as shown in Figure 14(A) or Figure 14(B), if the operator moves either of the two fingers up or down, the control unit 14 controls the operation of the image display unit 11 so that the image displayed on its display surface 11a is enlarged or reduced. As a result, due to the function of the 3D plate 12, the operator perceives the enlargement or reduction of the image formed on the aerial image-forming surface 16 in three dimensions.

[0056] Furthermore, when the control unit 14 recognizes that the detection unit 13 has detected that the operator's fingertips (two fingers) 42 have moved from the center of the hover surface 40 or the aerial image-forming surface 16 in the left-right or up-down direction, the control unit 14 controls the drive of the image display unit 11 so that the image displayed on its display surface 11a rotates around the Y-axis or the X-axis. As a result, the function of the 3D plate 12 allows the operator to perceive the image projected onto the aerial image-forming surface 16 as rotating around the Y-axis or the X-axis in three dimensions.

[0057] In the rotation input of the operation input device and operation input method of the present invention, for example, as shown in Figure 15, an image 50 that rotates by the same angle as the rotation angle of the displayed image (in Figure 15, an image of the brain) may be displayed in the upper right corner of the display surface 11a (aerial image forming surface 16) of the image display unit 11, so that the operator can easily grasp how much the displayed image has rotated. In Figure 15, the image 50 is shown as a spherical image and three arrows indicating the axis of rotation, but the image 50 is not limited to this, and for example, it may be just three arrows indicating the axis of rotation that do not include a spherical image.

[0058] The operation input device and operation input method of the present invention are useful, for example, in medical settings for simulating surgical movements by displaying images of various organs such as the brain, heart, lungs, stomach, liver, small intestine, large intestine, pancreas, kidneys, and blood vessels, but are not limited to that and can be used in a wide range of applications, such as rotating images of exhibits at trade shows.

[0059] (Embodiment 2: Program and recording medium) The program of this embodiment is characterized by causing a computer to execute the operation input method of the present invention. The program of this embodiment may be recorded, for example, on a computer-readable recording medium.

[0060] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the configuration and details of the present invention can be made that will be understood by those skilled in the art within the scope of the present invention. [Explanation of Symbols]

[0061] 10. Operation Input Device 11. Video display unit 11a Display surface 12 3D plates 13 Detection unit 14 Control Unit 15 cabinets 16. Aerial imaging plane 17. 3D sensing area 18 Filtering surface 19 Spherical images 20a, 20b, 43 Neutral Region 21a, 21b, 21a1, 21a2, 21a3, 21a4, 22a, 22b, 23a, 23b, 24R, 24L, 25R, 25L, 26R, 26L, 27R, 27L, 28R, 28L, 29R, 29L, 30R, 30L, 31R, 31L Operator's palm 40 hover planes 41, 42 The operator's fingers 44 Cursor

Claims

1. A video display unit having a display surface for displaying images, A 3D plate that enables the aforementioned image to be projected as a real image onto an image-forming surface in the air, A detection unit sets a three-dimensional sensing region including the aerial imaging surface and detects the movement of the operator's hand within the three-dimensional sensing region. A control unit that changes the image in accordance with the hand movements of the operator, An operating input device, including one.

2. The detection unit and the control unit set a filtering surface on the operator side with respect to the three-dimensional sensing area. The control unit excludes any part other than the operator's hand from detection when that part enters the three-dimensional sensing area beyond the filtering surface, according to claim 1.

3. The operation input device according to claim 1 or 2, wherein the control unit displays a spherical image on the video to assist in operation when the operator's hand movement is the operator's palm movement.

4. The control unit sets a hover plane parallel to the aerial image plane on the operator side with respect to the aerial image plane when the operator's hand movement is the operator's finger movement, according to claim 1 or 2.

5. A video display process in which an image is displayed on the display surface of the video display unit, An aerial imaging process in which the image is formed as a real image on an imaging surface in the air using a 3D plate, A detection step includes setting a three-dimensional sensing region including the aerial imaging surface and detecting the movement of the operator's hand within the three-dimensional sensing region, A control step that changes the image in accordance with the hand movements of the operator, Operation input methods, including those mentioned above.

6. A program characterized by causing a computer to execute the operation input method described in claim 5.

7. A computer-readable recording medium characterized by recording the program described in claim 6.

Images