Information processing device, control method for information processing device, and program

Abstract

The objective is to provide an information processing device, a control method for the information processing device, and a program that can improve the operability of the display device. [Solution] The information processing device 103 is communicably connected to a display device 100 that displays a spatial image including an image of at least one of the real space and the virtual space. The information processing device 103 includes a first detection means (processing calculation unit 109) that detects the hand movements of a user US who is viewing the spatial image displayed on the display device 100, a second detection means (processing calculation unit 109) that detects the position of the hand that has been detected in accordance with the detection by the first detection means, and a switching means (processing calculation unit 109) that switches the function to be executed by the display device 100 according to the detection result of the first detection means and the detection result of the second detection means.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for the information processing apparatus, and a program.

Background Art

[0002] There is known a Mixed Reality (MR) technology for performing a movement operation or the like on a virtual object included in a virtual space, for example, on an image of a composite reality that combines a real space and a virtual space. In this MR technology, an experiencer experiencing the composite reality can perform a movement operation or the like on a virtual object by making a gesture with his or her own hand without gripping and operating a controller. Further, the image of the composite reality is displayed, for example, on an HMD (Head Mounted Display) worn on the head of the experiencer or on a handheld display gripped by the hand of the experiencer. In the case of a handheld display, when the experiencer makes a gesture with a hand, for example, the experiencer grips the handheld display with the right hand and makes a gesture with the left hand. For example, Patent Document 1 discloses an electronic device that selects an operation mode based on the positional relationship between a face and a hand in a video. Further, Patent Document 2 discloses an information processing apparatus that calculates the three-dimensional position of a subject. The calculation of the three-dimensional position of this subject can be applied to the acquisition of the positional relationship between a face and a hand in Patent Document 1.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] When the electronic device described in Patent Document 1 is applied to a handheld display, the user of the handheld display would, for example, grasp the handheld display with their right hand and perform gestures with their left hand. Furthermore, if the user extends their right or left arm further, they would be unable to stably grasp the handheld display, and the usability when selecting an operation mode would be reduced.

[0005] This invention has been made in view of the above-mentioned problems. The object of this invention is to provide an information processing device, a control method for the information processing device, and a program that can improve the operability of a display device. [Means for solving the problem]

[0006] To achieve the above objective, the present invention provides an information processing device that is communicably connected to a display device that displays a spatial image including an image of at least one of real space and virtual space, and processes information communicated with the display device, and is characterized by comprising: a first detection means for detecting the hand movements of a user who views the spatial image displayed on the display device using the display device; a second detection means for detecting the position of the hand that has been detected in accordance with the detection by the first detection means; and a switching means for switching the function to be executed by the display device according to the detection result of the first detection means and the detection result of the second detection means. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the operability of the display device. [Brief explanation of the drawing]

[0008] [Figure 1] This block diagram shows an example of the functional configuration of an information processing system according to the first embodiment. [Figure 2] This is a block diagram showing an example of the hardware configuration of an information processing system. [Figure 3]This figure shows a handheld display device being held while a gesture is being performed. [Figure 4] This is a flowchart showing the processes performed by the information processing device. [Figure 5] This flowchart shows the process of setting the threshold used in the decision-making step S403 of the flowchart shown in Figure 4. [Figure 6] This diagram shows the relationship between the position where a gesture is performed between the proximal and distal ends and the function performed by the display device. [Figure 7] This figure shows examples of the proximal end, distal end, and the location where the gesture is performed. [Figure 8] This figure shows the state in which the user's hand is reflected in the image captured by the image capture unit of the display device according to the second embodiment. [Figure 9] This figure shows an example of an image displayed on the image display unit of the display device according to the third embodiment. [Figure 10] This figure shows the state in which a gesture is performed while gripping the display device according to the fourth embodiment. [Figure 11] This figure shows a notification device that can be attached to and detached from a finger according to the fifth embodiment. [Modes for carrying out the invention]

[0009] The embodiments of the present invention will be described in detail below with reference to the drawings. However, the configurations described in the following embodiments are merely illustrative, and the scope of the present invention is not limited to the configurations described in each embodiment. For example, each part constituting the present invention can be replaced with any configuration that can perform a similar function. In addition, any configuration may be added. Furthermore, any two or more configurations (features) from each embodiment can be combined.

[0010] <First Embodiment> The first embodiment will be described below with reference to Figures 1 to 6. Figure 1 is a block diagram showing an example of the functional configuration of the information processing system according to the first embodiment. Figure 2 is a block diagram showing an example of the hardware configuration of the information processing system. As shown in Figures 1 and 2, the information processing system 1000 has a display device 100 and an information processing device 103, and these devices are connected to each other so as to be able to communicate with each other. The communication between the display device 100 and the information processing device 103 may be by wired communication or by wireless communication. In this embodiment, the display device 100 and the information processing device 103 are composed of separate devices, but this is not limited to this configuration, and for example, the information processing device 103 may be built into the display device 100.

[0011] The display device 100 has an image capture unit (imaging means) 101 and an image display unit 102. The display device 100 is not particularly limited, and for example, a handheld display or a head-mounted display (HMD) can be used. A "handheld display" is a device that is held in at least one of the left hand LH and right hand RH of a user US who uses the display device 100 to view the spatial image displayed on the display device 100 (see Figure 3). A "head-mounted display" is a device that is worn on the user's head. In this embodiment, the display device 100 will be described as a handheld display.

[0012] The image capture unit 101 captures an image of the real space in the direction of the user's line of sight, for example, when the user US holds the display device 100 (handheld display) with both hands in front of their face or chest and faces forward. This image of the real space is displayed as live video on the image display unit 102 using a video see-through method. The image display unit 102 is not limited to displaying images of the real space; for example, it can display images of the virtual space, and can also display composite images that include images of both the real and virtual spaces. Thus, the image display unit 102 is configured to display spatial images that include images of at least one of the real and virtual spaces. Furthermore, the images of the real space and composite images include the user US's left hand LH or right hand RH, which are necessary for switching control to switch the functions performed by the display device 100, which will be described later. When the image display unit 102 displays images of the virtual space, the image capture unit 101 is used to capture an image of the user US's left hand LH or right hand RH, which are necessary for switching control. The image capture unit 101 captures images of the real space at a predetermined frame rate (e.g., 30 frames / second). This image capture unit 101 is composed of a stereo camera having a camera corresponding to the user US's left eye and a camera corresponding to the right eye. Hereinafter, the left and right pair of images captured by each camera may be referred to as "stereo camera images". A display device 100 with such a configuration can perform multiple types of functions. The functions that the display device 100 can perform are not particularly limited, and examples include a menu screen display function, an image enlargement function (zoom function), and an image reduction function. The menu screen display function is a function that displays a menu screen on the image display unit 102. The image enlargement function is a function that enlarges the spatial image displayed on the image display unit 102. The image reduction function is a function that reduces the spatial image displayed on the image display unit 102.

[0013] The information processing device 103 is a device that processes information communicated with the display device 100. The information processing device 103 includes an image capture storage unit 104, a contour point extraction unit 105, a three-dimensional position calculation unit 106, a relative position and orientation calculation unit 107, an image drawing unit 108, and a processing calculation unit 109. The information processing device 103 also includes a CPU 200, RAM 201, and ROM 202 as hardware. The information processing device 103 is not particularly limited, and for example, a desktop or notebook personal computer, a tablet terminal, or a smartphone can be used. The CPU 200 is a computer (control means) that controls the entire information processing system 1000. In this embodiment, the CPU 200 functions as the image capture storage unit 104 to the processing calculation unit 109, but is not limited to this. For example, a part having any of the functions of the image capture storage unit 104 to the processing calculation unit 109 may be provided separately from the CPU 200. The RAM 201 stores images captured by the image capture unit 101 under the control of the captured image storage unit 104. The ROM 202 stores various programs in advance. These programs include, for example, programs that cause the CPU 200 to execute each process (control method for the information processing device) described later.

[0014] The contour point extraction unit 105 extracts the contour points of the fingers from the image stored in the RAM 201 and including the left hand LH or the right hand RH of the user US by image recognition processing. The three-dimensional position calculation unit 106 calculates the three-dimensional position in the real space of the contour points extracted by the contour point extraction unit 105. The x coordinate and y coordinate of the contour point are acquired with the lower left corner of the VRAM area of the image including the contour point as the reference (0, 0). Note that the reference coordinates are not limited to the lower left corner of the VRAM area of the image including the contour point and can be arbitrary. Also, the z coordinate (coordinate in the depth direction) of the contour point is acquired from the stereo camera image. Further, in order to correctly display the depth of the hand, the area of the hand may be extracted from the video of the hand captured by the stereo camera, and the depth value of the hand with respect to the stereo camera may be calculated. In this case, the depth value is obtained by triangulation for all the corresponding points of the stereo images of the extracted hand contour line (see, for example, Patent Document 2 mentioned above). In addition, "LeapMotion (registered trademark)" of LeapMotion can also be used. With LeapMotion, the area of the hand can be detected from the stereo camera (image capturing unit 101). Also, a depth sensor may be installed in the display device 100, and the position and posture of the fingers may be estimated by the depth sensor. In the present embodiment, an arbitrary one point is determined from among the plurality of contour points extracted by the contour point extraction unit 105, and the three-dimensional positions of the other contour points with respect to the one point are calculated. The one point determined here may be stored as the reference position and may be updated periodically as the fingers move.

[0015] The relative position and orientation calculation unit 107 calculates the position and orientation of the hand in an image stored in the RAM 201 and including the left hand LH or the right hand RH of the user US. For calculating the position and orientation, for example, a method created using deep learning, a method using an existing publicly available library, or the like is used. The processing operation unit 109 associates the type of gesture with the change in the coordinate values calculated by the three-dimensional position calculation unit 106. The relationship between the type of gesture and the function executed by the display device 100 is stored in the ROM 202 in advance. Note that the addition of the type of gesture to the ROM 202 can be performed as appropriate. This addition may be possible, for example, on a setting menu. Also, the coordinate values calculated by the three-dimensional position calculation unit 106 include errors. In this case, the allowable error may be set to, for example, an error rate of 5 percent. The image drawing unit 108 draws an image to be displayed on the image display unit 102 of the display device 100 based on the processing result of the processing operation unit 109.

[0016] FIG. 3 is a diagram showing a state in which a gesture is being made while gripping a display device that is a handheld display. FIG. 3(a) is a diagram showing a state in which the user US grips the display device 100 with the right hand RH and makes a gesture (moves the left hand LH). FIG. 3(b) is a diagram showing a state in which the user US grips the display device 100 with the right hand RH and makes a gesture with the left hand LH extended more than the state shown in FIG. 3(a). As shown in FIG. 3, the index finger IF of the left hand LH is proximal to the display device 100 in the state shown in FIG. 3(a) and distal to the display device 100 in the state shown in FIG. 3(b). As a result, the separation distance of the index finger IF (left hand LH) from the display device 100 changes.

[0017] Figure 4 is a flowchart showing the processes performed by the information processing device. As shown in Figure 4, in step S400, the three-dimensional position calculation unit 106 of the information processing device 103 calculates the three-dimensional position (coordinates) of the contour points extracted by the contour point extraction unit 105. The processing calculation unit 109 determines whether it has recognized the movement of the user US's left hand LH and right hand RH, or the movement of any of the fingers of the left hand LH and right hand RH, based on the change in the three-dimensional position calculated by the three-dimensional position calculation unit 106. Note that when a person is awake, their hands and fingers do not completely stop moving. Therefore, the information processing device 103 can determine whether or not the hands and fingers are being moved intentionally by setting a threshold for determining the presence or absence of hand and finger movement. This threshold can be arbitrarily set and changed. The threshold can be changed, for example, by operating the keyboard or other means (change means) of the information processing device 103. If, as a result of the determination in step S400, it is determined that hand and finger movement has been recognized, the process proceeds to step S401. On the other hand, if the judgment in step S400 determines that the movement of the hand or fingers has not been recognized, the process remains in a waiting state at step S400.

[0018] In step S401, the processing unit 109 determines whether the hand and finger movements (hereinafter typically referred to as "fingers") determined in step S400 are included in the types of gestures pre-stored in the ROM 202 (first detection step). If the determination in step S401 is found to be included in the types of gestures, the process proceeds to step S402. On the other hand, if the determination in step S401 is found to be not included in the types of gestures, the process returns to step S400 and executes the subsequent steps in order. Thus, in this embodiment, the processing unit 109 also functions as a first detection means for detecting gestures as hand and finger movements of the user US. Note that in the information processing device 103, the part that functions as the first detection means may be provided separately from the processing unit 109.

[0019] In step S402, the processing unit 109 acquires the distance from the display device 100 to the finger based on the three-dimensional position of the finger that was identified as the target of determination (detection target) in step S400 (second detection step). This distance is calculated using the finger depth value obtained using a stereo camera image that includes the finger. The distance in step S402 may also be acquired, for example, by using an image captured by a TOF (Time of Flight) camera as a distance map. Thus, in this embodiment, the processing unit 109 also functions as a second detection means that detects the distance (position) of the detected finger from the display device 100 in conjunction with the gesture detection in step S401. In the information processing device 103, the part that functions as the second detection means may be provided separately from the processing unit 109.

[0020] In step S403, the processing unit 109 determines whether the distance obtained in step S402 is less than a threshold, as described later. If the determination in step S403 is found to be less than the threshold, the process executes step S404. On the other hand, if the determination in step S403 is found to be greater than or equal to the threshold, the process executes step S405.

[0021] In step S404, the processing unit 109 causes the display device 100 to execute a first function. The first function is not particularly limited; for example, one of the multiple functions that the display device 100 can execute can be assigned the menu screen display function.

[0022] In step S405, the processing unit 109 causes the display device 100 to execute a second function. The second function is not particularly limited; for example, an image magnification function can be assigned. Thus, in this embodiment, the processing unit 109 also functions as a switching means (switching step) to switch the function executed by the display device 100 according to the judgment result (detection result) in step S401 and the judgment result (detection result) in step S403. Specifically, if the processing unit 109 determines that a gesture is detected in step S401 and that the distance is less than a threshold in step S403, it switches to the menu screen display function (one function) from among several types of functions. If the processing unit 109 determines that a gesture is detected in step S401 and that the distance is not less than a threshold in step S403, it switches to an image magnification function (another function) different from the menu screen display function. Note that in the information processing device 103, the part that functions as a switching means may be provided separately from the processing unit 109. Furthermore, in this embodiment, two functions, namely the menu screen display function and the image enlargement function, are assigned to one gesture, but this is not limited to this. For example, three or more functions may be assigned to one gesture. If three functions are assigned to one gesture, two thresholds will be required.

[0023] Figure 5 is a flowchart showing the process of setting the threshold used in the decision made in step S403 of the flowchart shown in Figure 4. This process can be executed in the information processing device 103. As shown in Figure 5, in step S501, the processing unit 109 of the information processing device 103 acquires the distance R2 between the display device 100 and the left hand LH (or right hand RH) with the user US extending their left hand LH (or right hand RH) to its maximum extent. This distance R2 can be acquired in the same way as the distance acquired in step S402. In the following, the position of the left hand LH at distance R2 will be referred to as the "distal end".

[0024] In step S502, the processing unit 109 acquires the distance R1 between the display device 100 and the left hand LH, with the user US extending their left hand LH to the position closest to the display device 100 within the field of view of the image capture unit 101 of the display device 100. This distance R1 can be acquired in the same way as the distance acquired in step S402. In the following, the position of the left hand LH at distance R1 will be referred to as the "proximal end".

[0025] In step S503, the processing unit 109 sets the number of functions to be assigned to a single gesture as the number of divisions of the hand movement area. The difference between distance R1 and distance R2 represents the range in which functions can be executed by the gesture. For example, distance R1 is calculated using the following equation (1), with the position of the display device 100 as the starting point. In equation (1), "X1", "Y1", and "Z1" are the coordinates of the proximal end. Distance R2 is calculated using the following equation (2), with the position of the display device 100 as the starting point (X0, Y0, Z0). In equation (2), "X2", "Y2", and "Z2" are the coordinates of the distal end.

[0026]

number

[0027] In step S504, the processing unit 109 divides the difference between distance R1 and distance R2 by the number of divisions set in step S503. Based on the calculation result in step S504, a threshold (distance R1 + distance Ra) can be set.

[0028] Figure 6 shows the relationship between the position where a gesture is performed between the proximal and distal ends and the function executed by the display device. Figure 7 shows examples of the proximal end, distal end, and the position where the gesture is performed. For example, suppose the functions assigned to gesture A are the first function and the second function. If the distance R3 from the display device 100 to the point where gesture A is performed is less than the threshold (distance R1 + distance Ra), the first function is executed. If the distance from the display device 100 to the point where gesture A is performed is greater than or equal to the threshold (distance R1 + distance Ra), the second function is executed. Distance Ra is calculated using the following formula (3). Formula (3) is an example where the number of divisions set in step S503 is "2", and the difference between distance R1 and distance R2 is divided by "2". Distance R3 is calculated using the following formula (4). In formula (4), "X3", "Y3", and "Z3" are the coordinates of the point where gesture A is performed.

[0029]

number

[0030] In the information processing system 1000 (information processing device 103) configured as described above, a user US can, for example, grasp the display device 100 with their right hand RH and perform a gesture with their left hand LH. The function executed by the display device 100 can be changed according to the position of the gesture by the left hand LH relative to the display device 100. This improves the operability of the display device 100. The threshold may be determined, for example, based on the average human arm length or a previously measured arm length.

[0031] <Second Embodiment> The second embodiment will now be described with reference to Figure 8, focusing on the differences from the previously described embodiment, and omitting explanations of similar matters. Figure 8 shows the state in which the user's hand is reflected in the image captured by the image capture unit of the display device according to the second embodiment. Figure 8(a) shows the state in which the index finger IF of the user's left hand LH is reflected. Figure 8(b) shows the state in which the area from the index finger IF to the wrist of the user's left hand LH is reflected. Figure 8(c) shows the state in which the area from the index finger IF to the forearm of the user's left hand LH is reflected. For example, the state shown in Figure 8(b) can be considered the proximal end, and the state shown in Figure 8(c) can be considered the distal end. In this embodiment, the proximal end and distal end can be set according to the state in which the hand is reflected in the image captured by the image capture unit 101 of the display device 100, that is, according to the length (size) of the hand. Based on these proximal and distal ends, a threshold can be determined as described above.

[0032] <Third Embodiment> The third embodiment will now be described with reference to Figure 9, focusing on the differences from the previously described embodiment, and similar matters will be omitted from the description. Figure 9 is a diagram showing an example of an image displayed on the image display unit of the display device according to the third embodiment. The image 900 shown in Figure 9 includes the left hand (LH), the first function display unit 901, and the second function display unit 902. The first function display unit 901 indicates that when the left hand (LH) is positioned proximal, the function executed by the display device 100 can be switched to the first function. The second function display unit 902 indicates that when the left hand (LH) is positioned distal, the function executed by the display device 100 can be switched to the second function. The display control of the image 900 on the image display unit 102 of the display device 100 is performed by the CPU 200 of the information processing device 103.

[0033] Furthermore, when the left hand LH is positioned proximal, i.e., when switching to the second function is possible, the index finger IF of the left hand LH may be enclosed in a dashed circle 903. Also, when the left hand LH is positioned distal, i.e., when switching to the second function is possible, the index finger IF of the left hand LH may be enclosed in a solid circle 903. Such enclosures allow the user US to more reliably understand which of the first and second functions the display device 100 is capable of switching to. Note that the emphasis on function switching is not limited to enclosing the finger in a circle, but may also be done by adding color. Also, the image 900 may include a boundary line 905. The boundary line 905 is the line that the left hand LH crosses when switching from one of the first and second functions to the other. The boundary line 905 allows the user US to understand the timing of the function switching performed by the display device 100. Note that the boundary line 905 is not limited to a dashed line.

[0034] <Fourth Embodiment> The fourth embodiment will now be described with reference to Figure 10, focusing on the differences from the previously described embodiments, and similar matters will be omitted from the explanation. Figure 10 shows a state in which a gesture is performed while gripping the display device according to the fourth embodiment. Figure 10(a) shows a state in which the user US grips the display device 100 with their right hand RH and performs a relatively small gesture with their left hand LH. Figure 10(b) shows a state in which the user US grips the display device 100 with their right hand RH, extends their left hand LH more than in the state shown in Figure 10(a), and performs a larger gesture than in the state shown in Figure 10(a). Both the gesture shown in Figure 10(a) and the gesture shown in Figure 10(b) are assigned, for example, an image magnification function. For the user US, extending their left hand LH further makes it easier to perform larger gestures. Also, to judge relatively small (fine) gestures, it is preferable that the left hand LH is closer to the display device 100. In this embodiment, the type of gesture can be linked (combined) with the function executed by the display device 100, taking into consideration the ease of performing the gesture for the user US and the stability of gesture judgment.

[0035] <Fifth Embodiment> The fifth embodiment will now be described with reference to Figure 11, focusing on the differences from the previously described embodiments, and similar matters will be omitted. Figure 11 shows a notification device that can be attached to and removed from a finger according to the fifth embodiment. As shown in Figure 11, the communication device 1100 is a ring-type device that is attached, for example, to the index finger IF of the left hand LH. The communication device 1100 has, for example, a GPS 1101 built in and is communicably connected to the information processing device 103. This allows for more accurate detection of the judgment in step S401, i.e., the gesture of the left hand LH, based on the communication information from the communication device 1100. Also, the distance in step S401, i.e., the position of the left hand LH, can be detected more accurately based on the communication information from the communication device 1100. In this embodiment, the communication device 1100 is a ring-type device, but is not limited to this, and may be, for example, a glove-type device that can be attached to and removed from the hand.

[0036] While preferred embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of its gist. The present invention supplies a program that implements one or more functions of the embodiments described above to a system or device via a network or storage medium. It can also be implemented by one or more general-purpose processors (ASICs) of the computer of the system or device reading and executing the program. Furthermore, the present invention can also be implemented by a dedicated processor (e.g., an ASIC or FPGA) that implements one or more functions. Moreover, the present invention can also be implemented by a combination of a general-purpose processor and a dedicated processor. Here, "processor" refers to a processor in a broad sense and includes both general-purpose processors and dedicated processors. Furthermore, the process of implementing the present invention may be executed by only one processor, or by the cooperation of multiple processors located at physically separate locations. Furthermore, although the display device 100 is a handheld display in each embodiment, it is not limited to this, and can also be a head-mounted display.

[0037] Each embodiment of the disclosure includes the following configurations, methods, and programs. (Configuration 1) An information processing device that is communicably connected to a display device that displays a spatial image including an image of at least one of real space and virtual space, and processes information communicated with the display device, A first detection means for detecting the hand movements of a user viewing the spatial image displayed on the display device, A second detection means detects the position of the hand that has been detected by the first detection means, An information processing apparatus comprising a switching means for switching the function to be performed by the display device according to the detection result of the first detection means and the detection result of the second detection means. (Configuration 2) The information processing device according to Configuration 1, characterized in that the first detection means detects a gesture as the hand movement. (Configuration 3) The information processing apparatus according to Configuration 1 or 2, characterized in that the second detection means detects the distance between the hand and the display device as the position. (Configuration 4) The functions performed by the display device include multiple types of functions, The first detection means detects a gesture as the hand movement, The second detection means detects the distance between the hand and the display device as the position, The information processing apparatus according to Configuration 1, characterized in that the switching means switches to one of the multiple types of functions when the gesture is detected by the first detection means and the distance between the two persons detected by the second detection means is less than a threshold, and switches to another function different from the one of the multiple types of functions when the gesture is detected by the first detection means and the distance between the two persons detected by the second detection means is greater than or equal to the threshold. (Configuration 5) The information processing apparatus according to Configuration 4, characterized by comprising a means for changing the threshold. (Configuration 6) The information processing apparatus according to Configuration 4 or 5, characterized in that it includes a control means that displays a message to the display device when the switching means becomes capable of switching to one of the functions, and displays a message to the display device when the switching means becomes capable of switching to the other function. (Configuration 7) An information processing apparatus according to any one of Configurations 4 to 6, characterized in that it includes control means for causing the display device to show the boundary line that the hand crosses when switching from one of the two functions to the other function. (Configuration 8) The information processing device according to Configuration 7, wherein the control means displays the boundary line when the hand is moving. (Configuration 9) The display device is equipped with imaging means capable of capturing an image of the hand, The first detection means detects the movement of the hand based on the image of the hand, The information processing apparatus according to any one of configurations 1 to 8, characterized in that the second detection means detects the position of the hand based on the image of the hand. (Configuration 10) The hand is fitted with a communication device that is connected to the information processing device in a communicative manner, The first detection means detects the hand movement based on the communication information from the communication device, The information processing device according to any one of configurations 1 to 9, characterized in that the second detection means detects the position of the hand based on communication information from the communication device. (Configuration 11) The information processing device according to any one of Configurations 1 to 10, characterized in that the information processing device is communicably connected to a head-mounted display used by being worn on the user's head, or a handheld display used by being held in at least one of the user's left and right hands, as the display device. (Method 1) A method for controlling an information processing device that is communicatively connected to a display device that displays a spatial image including an image of at least one of real space and virtual space, and that processes information communicated with the display device, A first detection step involves using the display device to detect the hand movements of a user who views the spatial image displayed on the display device, Following the detection in the first detection step, a second detection step is performed to detect the position of the hand that was the target of the detection, A control method for an information processing device, characterized by comprising a switching step, which switches the function to be performed by the display device according to the detection result in the first detection step and the detection result in the second detection step. (Program 1) A program characterized by causing a computer to execute the control method described in Method 1. [Explanation of symbols]

[0038] 100 display device 103 Information Processing Device 106 3D position calculation section 109 Processing Unit 200 CPU

Claims

1. An information processing device that is communicatively connected to a display device that displays a spatial image including an image of at least one of real space and virtual space, and processes information communicated with said display device, A first detection means for detecting the hand movements of a user viewing the spatial image displayed on the display device, A second detection means detects the position of the hand that has been detected by the first detection means, An information processing apparatus comprising a switching means for switching the function to be performed by the display device according to the detection result of the first detection means and the detection result of the second detection means.

2. The information processing device according to claim 1, characterized in that the first detection means detects a gesture as the hand movement.

3. The information processing apparatus according to claim 1, characterized in that the second detection means detects the distance between the hand and the display device as the position.

4. The functions performed by the aforementioned display device include multiple types of functions, The first detection means detects a gesture as the hand movement, The second detection means detects the distance between the hand and the display device as the position, The information processing apparatus according to claim 1, characterized in that the switching means switches to one of the multiple types of functions when the gesture is detected by the first detection means and the distance between the two persons detected by the second detection means is less than a threshold, and switches to another function different from the one of the multiple types of functions when the gesture is detected by the first detection means and the distance between the two persons detected by the second detection means is greater than or equal to the threshold.

5. The information processing apparatus according to claim 4, characterized in that it comprises a means for changing the threshold.

6. The information processing apparatus according to claim 4, further comprising control means for displaying a message on the display device when the switching means becomes capable of switching to one of the functions, and for displaying a message on the display device when the switching means becomes capable of switching to the other of the functions.

7. The information processing apparatus according to claim 4, further comprising control means for causing the display device to display the boundary line that the hand crosses when switching from one of the aforementioned functions to the other function.

8. The information processing device according to claim 7, characterized in that the control means displays the boundary line when the hand is moving.

9. The display device includes an imaging means capable of capturing an image of the hand, The first detection means detects the movement of the hand based on the image of the hand, The information processing apparatus according to claim 1, characterized in that the second detection means detects the position of the hand based on the image of the hand.

10. The hand is equipped with a communication device that is connected to the information processing device in a manner that enables communication with it. The first detection means detects the hand movement based on the communication information from the communication device, The information processing apparatus according to claim 1, characterized in that the second detection means detects the position of the hand based on communication information from the communication device.

11. The information processing device according to claim 1, characterized in that the information processing device is communicably connected to a display device, which is a head-mounted display used by being worn on the user's head, or a handheld display used by being held in at least one of the user's left and right hands.

12. A method for controlling an information processing device that is communicatively connected to a display device that displays a spatial image including an image of at least one of real space and virtual space, and that processes information communicated with the display device, A first detection step involves using the display device to detect the hand movements of a user who views the spatial image displayed on the display device, Following the detection in the first detection step, a second detection step is performed to detect the position of the hand that was the target of the detection, A control method for an information processing device, characterized by comprising a switching step, which switches the function to be performed by the display device according to the detection result in the first detection step and the detection result in the second detection step.

13. A program characterized by causing a computer to execute the control method described in claim 12.

Images