Information processing system, information processing method, and program

Abstract

To provide a technique which detects movement of a hand to allow a virtual interface to be appropriately operated.SOLUTION: An information processing system includes: determination means which determines movement of a hand of a user on the basis of inertia information on a controller held by or worn on the hand; control means which controls display means so as to represent a virtual space where a virtual interface is arranged in a real space; and operation means which operates the virtual interface on the basis of the movement of the hand determined by the determination means if a position of the virtual interface and a position of the hand have a specific relation in the virtual space.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] Conventionally, in Mixed Reality (MR) technology and Virtual Reality (VR) technology, a technology that allows a user to operate a CG model has been considered. In addition, such technology often uses a head mounted display (HMD).

[0003] Patent Document 1 describes a method for moving a CG model by motion capture using an image of a hand captured by an imaging device, SLAM (Stimultaneous Localization and Mapping), etc. Patent Document 2 describes a method for operating a switch on a CG model by tracking the movement of a user's hand using an image of the hand captured by a device mounted on an HMD. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-129167 A [Patent Document 2] JP 2018-206340 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the above-mentioned technology has a problem in that if the hand is not captured in the captured image, it is not possible to appropriately detect the hand movement and operate the virtual interface.

[0006] Therefore, an object of the present invention is to provide a technique for detecting hand movements and enabling appropriate operation of a virtual interface. [Means for solving the problem]

[0007] One aspect of the present invention is a method for producing a composition comprising the steps of: A determination means for determining a movement of a hand based on inertial information of a controller held by or attached to a user's hand; a control means for controlling a display means so as to display a virtual space in which the virtual interface is arranged in a real space; an operating means for operating the virtual interface based on the movement of the hand determined by the determining means when a position of the virtual interface and a position of the hand have a specific relationship in the virtual space; The information processing system is characterized by having the following features.

[0008] One aspect of the present invention is a method for producing a composition comprising the steps of: a determining step of determining a movement of the hand based on inertial information of a controller held by or attached to a user's hand; a control step of controlling a display means to display a virtual space in which the virtual interface is arranged in a real space; When the position of the virtual interface and the position of the hand have a specific relationship in the virtual space, the virtual interface is moved based on the movement of the hand determined in the determining step. Operation steps for operating the interface; The information processing method is characterized by having the following features. Effect of the Invention

[0009] According to the present invention, hand movements can be detected to appropriately operate a virtual interface. [Brief description of the drawings]

[0010] [Figure 1] 1 is an external view of an information processing system according to a first embodiment. [Diagram 2]1 is a diagram showing each configuration of an information processing system according to a first embodiment. [Diagram 3] FIG. 2 is a diagram showing the relationship between a real space and a synthetic image according to the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating an operation of a virtual dial according to the first embodiment. [Diagram 5] 4 is a flowchart of a virtual dial operation process according to the first embodiment. [Figure 6] FIG. 2 is a diagram for explaining an operation acceptance range according to the first embodiment. [Figure 7] FIG. 4 is a diagram for explaining calculation of shift information according to the first embodiment. [Figure 8] 4A to 4C are diagrams illustrating the operation of a virtual slider according to the first embodiment. [Figure 9] 11 is a flowchart of a virtual slider operation process according to the first embodiment. [Figure 10] FIG. 2 is a diagram showing an interface using a model according to the first embodiment. [Figure 11] 13 is a flowchart of a virtual dial operation process according to the second embodiment. [Figure 12] 13A to 13C are diagrams illustrating the operation of a virtual dial according to a third embodiment. [Figure 13] 13 is a flowchart of a virtual dial operation process according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0012] <Embodiment 1> An information processing system 1 according to the first embodiment will be described with reference to Fig. 1. The information processing system 1 includes an HMD 100, an image processing device 110, and a controller 120.

[0013] The HMD 100 is a head-mounted display device (electronic device) that is worn on the head of a user. The HMD 100 displays a composite image that combines a captured image of the area in front of the user captured by the HMD 100 with content such as CG (computer graphics) in a form that corresponds to the posture of the HMD 100.

[0014] The image processing device 110 is a control device (information processing device; electronic device) that controls the HMD 100. The image processing device 110 is, for example, a smartphone, a tablet terminal, or a PC (personal computer). The image processing device 110 is connected to the HMD 100 wirelessly or by wire. The image processing device 110 generates a composite image by combining a captured image with CG (such as a virtual operation interface), and transmits the composite image to the HMD 100. Note that each component of the image processing device 110 may be included in the HMD 100.

[0015] The controller 120 is a device for performing various controls of the HMD 100. The controller 120 accepts user operations on a virtual operation interface (such as a virtual dial or a virtual slider) expressed by CG. The controller 120 has a finger ring shape that can be worn on a user's finger, as shown in FIG. 1, for example. If the controller 120 can be worn on a user's finger, the user can move his or her hand freely while holding the controller 120. The controller 120 performs wireless communication with the image processing device 110 via Bluetooth.

[0016] (Internal structure of HMD) The internal configuration of the HMD 100 will be described with reference to FIG. 201 , an imaging unit 202 , an image display unit 203 , and an orientation sensor unit 204 .

[0017] The HMD control unit 201 controls each component of the HMD 100. When the HMD control unit 201 acquires a composite image (an image obtained by combining a captured image of the space in front of the user captured by the imaging unit 202 with CG) from the image processing device 110, the HMD control unit 201 displays the composite image on the image display unit 203. Therefore, by wearing the HMD 100, the user can view the composite image displayed on the image display unit 203. The user can experience various mixed realities (virtual spaces) in which CG is blended with real space.

[0018] The imaging unit 202 includes two cameras (imaging devices). The two cameras are disposed near the positions of the left and right eyes of the user when wearing the HMD 100 in order to capture an image of a space similar to the space the user normally sees. Images of a subject (a range in front of the user) captured by the two cameras are output to the image processing device 110. In addition, the two cameras in the imaging unit 202 can acquire information on the distance from the two cameras to the subject as distance information by measuring distances with a stereo camera.

[0019] The image display unit 203 displays a composite image representing a virtual space in which CG is placed in a real space. The image display unit 203 has, for example, a liquid crystal panel or an organic EL panel. When the user is wearing the HMD 100, an organic EL panel is placed in front of each of the user's eyes.

[0020] The attitude sensor unit 204 detects the attitude (and position) of the HMD 100. Then, the attitude sensor unit 204 detects (acquires) the attitude of the user (the user wearing the HMD 100) that corresponds to the attitude (and position) of the HMD 100. The attitude sensor unit 204 has an inertial measurement unit (IMU). The attitude sensor unit 204 outputs information on the attitude of the user (attitude information) to the image processing device 110.

[0021] (Internal configuration of image processing device) The internal configuration of the image processing device 110 will be described with reference to Fig. 2. The image processing device 110 includes a control unit 211, a content DB 212, and a communication unit 213.

[0022] The control unit 211 controls each component of the image processing device 110. The control unit 211 can also control the HMD 100 and the controller 120.

[0023] The control unit 211 receives an image (captured image) acquired by the imaging unit 202 and orientation information acquired by the orientation sensor unit 204 from the HMD 100. The control unit 211 performs image processing on the captured image to cancel aberrations in the optical system of the imaging unit 202 and the optical system of the image display unit 203. Then, the control unit 211 generates a composite image by combining the captured image with any CG. That is, the control unit 211 generates a composite image representing a virtual space in which CG is placed in a real space (a space represented by the captured image). The control unit 211 transmits the composite image to the HMD control unit 201 in the HMD 100.

[0024] The control unit 211 can also control the position, orientation, and size of the CG in the composite image based on the information (distance information and orientation information) acquired by the HMD 100. For example, when the control unit 211 places a virtual object represented by the CG near a specific object existing in real space in the space represented by the composite image, the control unit 211 makes the virtual object (CG) larger as the distance between the specific object and the imaging unit 202 is closer. By controlling the position, orientation, and size of the CG in this way, the control unit 211 can generate a composite image in which a CG object that is not placed in real space appears as if it were placed in real space.

[0025] The content DB 212 is a storage unit that stores information such as CG, etc. The control unit 211 can switch the CG read from the content DB 212 (that is, the CG used to generate a composite image).

[0026] (Internal structure of the controller) The internal configuration of the controller 120 will be described with reference to Fig. 2. The controller 120 includes a controller control unit 221, a detection unit 222, and a communication unit 223.

[0027] The controller control unit 221 controls each component of the controller 120 .

[0028] The detection unit 222 includes an acceleration sensor (a sensor that acquires acceleration information) and / or a gyro sensor (a sensor that acquires angular velocity information). The detection unit 222 acquires inertial information (such as acceleration information or angular velocity information) of the controller 120. The detection unit 222 can also detect the movement of the user's hand based on the inertial information. The movement of the hand can be represented by, for example, a movement vector (movement information) of the hand or a rotation angle of the hand. In the first embodiment, since the controller 120 is worn on the hand, the movement of the controller 120 can be said to be the movement of the hand.

[0029] The communication unit 223 performs wireless communication with the image processing device 110 (the communication unit 213). The communication unit 223 transmits, for example, to the image processing device 110, inertial information of the controller 120 or information on the movement of the user's hand detected from the inertial information.

[0030] The image processing device 110 (information processing device) may have a part of the configuration of the controller 120 or a part of the configuration of the HMD 100. For example, the controller 120 may have only a configuration for acquiring inertial information of the detection unit 22, and the image processing device 110 may have a configuration for detecting (determining) hand movements from the inertial information.

[0031] With reference to Figs. 3 to 6B, an example of operation of a virtual operation interface (virtual interface) such as a dial or slider will be described. Here, an example of operation of a virtual dial 300 and a virtual slider 301 for adjusting volume will be described. The virtual dial 300 can be operated in response to rotation of the hand (wrist) in an xy plane, which will be described later. The volume is controlled in response to operation of the virtual dial 300. The virtual slider 301 can be operated in response to movement of the hand in an xy plane, which will be described later. The volume is controlled in response to operation of the virtual slider 301. Note that the following description is applicable not only to the virtual dial 300 and the virtual slider 301, but also to any virtual operation interface (such as a virtual lever, for example) that should be operated in response to hand movement.

[0032] 3 shows the relationship between the real space and the composite image displayed on the image display unit 203. Here, the coordinate system of the real space is represented by coordinates (x, y, z), and the coordinate system of the captured image (two-dimensional image) acquired by the imaging unit 202 in the HMD 100 is represented by coordinates (h, v). In the composite image, a virtual dial 300 for adjusting the volume and a virtual slider 301 for adjusting the volume are superimposed on the captured image (image of the real space) of the house and the user's hands captured by the imaging unit 202.

[0033] (Virtual dial operation) 4A, 4B, and 5, the operation of the virtual dial 300 will be described. In the following, the coordinates of the position of the controller 120 in the real space at time t will be expressed as coordinates (x(t), y(t), z(t)).

[0034] In the following, the control unit 211 controls the image capturing unit 202 in the HMD 100 to capture the image. The virtual dial 300 is operated based on information on the position of the user's hand that is being input and information on the movement of the user's hand obtained from the detection unit 222 of the controller 120.

[0035] First, the flow of operations of the virtual dial 300 will be described with reference to the flowchart of Fig. 5. The process of the flowchart of Fig. 5 starts when the user selects the volume adjustment item in the menu.

[0036] In step S501, the control unit 211 displays the virtual dial 300 on the image display unit 203 of the HMD 100. That is, the control unit 211 displays a composite image representing a virtual space in which the virtual dial 300 is placed in real space on the image display unit 203. At this time, the control unit 211 acquires the coordinates (xc(0), yc(0), zc(0)) of the center position of the virtual dial 300 in real space, as shown in FIG. 4A.

[0037] In step S502, the control unit 211 sets an operation reception range 600 of the virtual dial 300 in real space, as shown in Fig. 6A. For example, the control unit 211 sets a circle having a specific radius centered on the coordinates (xc(0), yc(0), zc(0)) of the center position of the virtual dial 300. Then, the control unit 211 sets the range included in the circle as the operation reception range 600.

[0038] As shown in Fig. 6B, when there are multiple virtual operation interfaces around one virtual dial 300, the control unit 211 judges whether the distance between the virtual dial 300 and the other virtual operation interfaces is closer than a specific distance. When the distance between the virtual dial 300 and the other virtual operation interfaces is closer than a specific distance, the control unit 211 narrows the operation acceptance range 600 of the virtual dial 300. As a result, even when the distance between the multiple virtual operation interfaces is short, the possibility that the multiple operation acceptance ranges 600 overlap each other can be reduced by narrowing the operation acceptance range 600. This makes it possible to smoothly switch the operation target between the multiple virtual operation interfaces.

[0039] In step S503, the control unit 211 determines whether or not the user's finger has touched the virtual dial 300 based on the captured image. At this time, first, the control unit 211 converts the coordinates (h, v) of the position of the finger (controller 120) in the captured image into coordinates (x, y, z) in real space. After that, the control unit 211 determines whether or not the finger has touched the virtual dial 300 based on whether or not the coordinates (x, y, z) are included in the operation reception range 600. This can prevent erroneous detection of the presence or absence of contact with the virtual dial 300. Note that, for example, distance information acquired by the imaging unit 202 may also be used for the conversion from the coordinates (h, v) to the coordinates (x, y, z). In addition, whether or not the finger has touched the virtual dial 300 may be determined based on whether or not the distance between the coordinates of the finger and the coordinates of the center position of the virtual dial 300 is shorter than a predetermined distance.

[0040] Then, in step S503, if it is determined that the user's finger has touched the virtual dial 300 (if it is determined that the finger position is included in the operation reception range 600), the process proceeds to step S504. If it is determined that the user's finger has not touched the virtual dial 300 (if it is determined that the finger position is not included in the operation reception range 600), the process of step S503 is repeated.

[0041] In step S504, the control unit 211 determines the coordinates (x(0), y(0), z(0)) of the initial position of the controller 120 (the finger wearing the controller 120) and the coordinates (xt(0), yt(0), zt(0)) of the initial position of the thumb based on the captured image. In step S504, the initial position is the position at time t=0 when it is determined that the finger has touched the virtual dial 300. Furthermore, the control unit 211 determines the position of the virtual dial 300 in the xy plane using the following formula 1: Then, the distance L between the center position of the wheel 300 and the controller 120 is calculated (see FIG. 4A).

number

[0042] Furthermore, the control unit 211 calculates the distance Lt between the thumb and the center position of the virtual dial 300 on the xy plane by the following equation 2 (see FIG. 4A).

number

[0043] Then, the control unit 211 calculates the angle θt formed by the line segment connecting the center position of the virtual dial 300 and the position of the thumb, and the line segment connecting the position of the controller 120 and the center position of the virtual dial 300, using the following equation 3 (see FIG. 4A).

number

[0044] In step S505, the control unit 211 determines the current position of the controller 120 (the finger on which the controller 120 is worn). The control unit 211 determines the coordinates (x(t), y(t), z(t)) of the current position of the controller 120 based on acceleration information (ax(t), ay(t), az(t)) of the controller 120. Note that when the imaging unit 202 is capturing an image of a finger, the control unit 211 may determine the coordinates (x(t), y(t), z(t)) based on the coordinates (h(t), v(t)) of the position of the controller 120 obtained from the captured image in addition to the acceleration information.

[0045] First, the control unit 211 controls the detection unit 222 to detect (determine) the movement of the hand. Specifically, the detection unit 222 calculates (detects) movement information (Δx(t), Δy(t), Δz(t)) from the coordinates (x(0), y(0), z(0)) of the initial position of the controller 120 as information representing the movement of the hand.

[0046] To calculate the movement information (Δx(t), Δy(t), Δz(t)), the detection unit 222 integrates the detected acceleration twice. For example, the detection unit 222 can calculate the movement information Δx(t) in the x-axis direction from time t=0 to time t=T using the acceleration ax(t) in the x-axis direction according to the following formula 4. Note that the calculation as shown in formula 4 is performed for each of the x-axis direction, the y-axis direction, and the z-axis direction.

number

[0047] Then, the control unit 211 adds the movement information (Δx(t), Δy(t), Δz(t)) to the coordinates of the initial position (x(0), y(0), z(0)) and determines the result as the coordinates (x(t), y(t), z(t)) of the current position of the controller 120. By calculating the current position according to the movement information, it becomes possible to appropriately operate the virtual dial 300 even if the finger (controller 120) cannot be detected in the captured image obtained from the imaging unit 202.

[0048] In step S506, the control unit 211 determines whether or not a hand is included in the captured image acquired by the imaging unit 202. If it is determined that a hand is included in the captured image, the control unit 211 The process proceeds to step S507. If it is determined that the captured image does not include a hand, the process proceeds to step S509.

[0049] In step S507, the control unit 211 determines how much the position of the fingers gripping the virtual dial 300 has shifted in the horizontal and vertical directions other than the rotation around the center position of the virtual dial 300, as shown in Fig. 7B. In other words, the control unit 211 determines how much the detected hand movement includes movements other than the hand rotation. Specifically, the control unit 211 calculates (determines) the movement information in the xy plane corresponding to the movement other than the hand rotation, among the movement information of the controller 120, as shift information (Δxs, Δys, 0).

[0050] The control unit 211 calculates shift information (Δxs, Δys, 0) based on the relationship between the position of the controller 120, the position of the thumb, and the position of the virtual dial 300 at the time of step S504, in addition to the coordinates (x(t), y(t), z(t)) of the current position of the controller 120. Specifically, the control unit 211 calculates shift information (Δxs, Δys, 0) in the x-axis direction and the y-axis direction using the distance L and distance Lt calculated in step S504, in addition to the coordinates (x(t), y(t), z(t)) of the current position of the controller 120 (see FIG. 7B).

[0051] First, the control unit 211 determines a point P at a position that is a distance Lt away on the xy plane from the coordinates (xt(t), yt(t), zt(t)) of the current position of the thumb and a distance L away on the xy plane from the coordinates (x(t), y(t), z(t)). Here, the point P is determined so that a line segment connecting the coordinates (xc(t), yc(t), zc(t)) of the point P and the coordinates (xt(t), yt(t), zt(t)) of the position of the thumb and a line segment connecting the coordinates of the point P and the coordinates (x(t), y(t), z(t)) of the current position of the controller 120 form an angle θt.

[0052] Then, the control unit 211 calculates the difference (the difference in the x-axis direction and the y-axis direction) between the coordinates (xc(t), yc(t), zc(t)) of the point P and the coordinates (xc(0), yc(0), zc(0)) of the center position of the virtual dial 300, as in the following formula 5. The control unit 211 sets (determines) this calculation result as the shift information (Δxs, Δys, 0).

number

[0053] In step S508, the control unit 211 corrects (resets) the coordinates of the center position of the virtual dial 300 and the coordinates of the initial position of the controller 120. Specifically, the control unit 211 adds shift information (Δxs, Δys, 0) to the coordinates (xc(0), yc(0), zc(0)) of the center position of the virtual dial 300 and the coordinates (x(0), y(0), z(0)) of the initial position of the controller 120, as shown in Equation 6. Hereinafter, the corrected coordinates of the center position of the virtual dial 300 are represented as coordinates (xc(t), yc(t), zc(t)), and the corrected coordinates of the initial position of the controller 120 are represented as coordinates (x0(t), y0(t), z0(t)). The corrected coordinates of the center position of the virtual dial 300 are the same as the coordinates of the point P described above. According to this, when the user operates the virtual operation interface, the position of the finger may deviate from the CG position of the virtual operation interface, but even in this case, the user can continue to operate appropriately.

number

[0054] In step S509, the control unit 211 determines (detects) the rotation of the hand in the xy plane among the movements of the user's hand. Specifically, the control unit 211 determines the motion angle θ(t) representing the rotation angle of the hand. For example, when the process of step S508 is performed, the motion angle θ(t) can be determined as shown in the following formula 7. In formula 7, the coordinates (x(t), y(t), z(t)) of the current position of the controller 120, the coordinates (xc(t), yc(t), zc(t)) of the center position of the virtual dial 300, and the coordinates (x0(t), y0(t), z0(t)) of the initial position of the controller 120 are used.

number

[0055] When the process of step S508 has not been performed, the control unit 211 uses the coordinates (xc(0), yc(0), zc(0)) of the center position of the virtual dial 300 instead of the coordinates (xc(t), yc(t), zc(t)). In this case, the control unit 211 uses the coordinates (x(0), y(0), z(0)) of the initial position of the controller 120 instead of the coordinates (x0(t), y0(t), z0(t)).

[0056] In step S510, the control unit 211 operates the virtual dial 300 by the operation angle θ(t). For example, the control unit 211 rotates the virtual dial 300 by the operation angle θ(t). As a result, even if the virtual dial 300 is held in a different way, the same result can be obtained as long as the angle of rotation of the hand is the same.

[0057] In step S511, the control unit 211 determines whether or not the finger has been removed from the virtual dial 300. Specifically, the control unit 211 determines whether or not the finger position is included in the operation acceptance range 600 by the same method as in step S502. If it is determined that the finger position is not included in the operation acceptance range 600 (the finger has been removed from the virtual dial 300), the process proceeds to step S512. If it is determined that the finger position is included in the operation acceptance range 600 (the finger has not been removed from the virtual dial 300), the process returns to step S505.

[0058] In step S512, the control unit 211 ends the display of the virtual dial 300. This ends the volume adjustment.

[0059] (Virtual slider operation) An example of the operation of the virtual slider 301 will be described with reference to the flowchart of Fig. 9. Note that, below, an example of the virtual slider 301 for adjusting the volume shown in Fig. 3 will be described. Note that the process of the flowchart of Fig. 9 starts when the volume adjustment item is selected in the menu.

[0060] 8A, the control unit 211 displays a virtual slider 301 as a virtual operation interface on the image display unit 203. That is, the control unit 211 displays on the image display unit 203 a composite image representing a virtual space in which the virtual slider 301 is arranged in the real space.

[0061] In step S902, the control unit 211 determines the operation acceptance range 600 of the virtual slider 301. As in the example of the virtual dial 300, the control unit 211 may control the size of the operation acceptance range 600 depending on whether or not there is another virtual operation interface close to the virtual slider 301. In other words, the control unit 211 may control the operation acceptance range 600 of each of the multiple virtual operation interfaces depending on the distance between the multiple virtual operation interfaces (between the multiple virtual interfaces) in the virtual space.

[0062] In step S903, similarly to step S503, the control unit 211 determines whether or not the user's finger (hand) has touched the virtual slider 301 based on the captured image. Whether or not the finger has touched the virtual slider 301 can be determined by the same method as the method for determining whether or not the finger has touched the virtual dial 300. If it is determined that the finger has touched the virtual slider 301, the process proceeds to step S904. If it is determined that the finger has not touched the virtual slider 301, the process of step S903 is repeated.

[0063] In step S904, the control unit 211 determines the coordinates (x(0), y(0), z(0)) of the initial position of the controller 120 (the finger wearing the controller 120) based on the captured image, as shown in FIG. 8A.

[0064] In step S905, the control unit 211 determines the current position of the controller 120 (the finger wearing the controller 120). The control unit 211 determines the coordinates (x(t), y(t), z(t)) of the current position of the controller 120 based on the acceleration information (ax(t), ay(t), az(t)) acquired by the detection unit 222 (see FIG. 8B). When the imaging unit 202 is capturing an image of a finger, the control unit 211 may determine the coordinates (x(t), y(t), z(t)) of the current position of the controller 120 based on the coordinates (h, v) of the position of the finger obtained from the captured image in addition to the acceleration information.

[0065] First, the control unit 211 controls the detection unit 222 to determine (detect) the movement of the hand. Specifically, the detection unit 222 calculates movement information (Δx(t), Δy(t), Δz(t)) from the coordinates (x(0), y(0), z(0)) of the initial position as information representing the movement of the hand. In order to calculate the movement information (Δx(t), Δy(t), Δz(t)), the detection unit 222 integrates the detected acceleration twice. For example, as shown in the following formula 8, the detection unit 222 calculates the movement information Δx in the x-axis direction from time t=0 to time t=T using the acceleration ax(t) in the x-axis direction. Note that the calculation shown in formula 8 is performed for each of the x-axis direction, the y-axis direction, and the z-axis direction.

number

[0066] Then, the control unit 211 determines the value obtained by adding the movement information (Δx(t), Δy(t), Δz(t)) to the coordinates of the initial position (x(0), y(0), z(0)) as the coordinates (x(t), y(t), z(t)) of the current position of the controller 120.

[0067] In step S906, the control unit 211 determines a motion distance L(t) that represents the amount of hand movement in the xy plane, as shown in the following formula 9. In formula 9, the current position of the controller 120 is The coordinates of the current position (x(t), y(t), z(t)) and the initial position of the controller 120 (x(0), y(0), z(0)) are used.

number

[0068] In step S907, the control unit 211 operates the virtual slider 301 by the operating distance L(t). For example, the control unit 211 moves the bar 310 by the operating distance L(t) as shown in Figures 8A and 8B.

[0069] In step S908, the control unit 211 determines whether or not the finger has been released from the virtual slider 301. Specifically, the control unit 211 determines whether or not the position of the finger is included in the operation reception range 600. If it is determined that the position of the finger is not included in the operation reception range 600, the process proceeds to step S909. If it is determined that the position of the finger is included in the operation reception range 600, the process returns to step S905.

[0070] In step S909, the control unit 211 ends the display of the virtual slider 301.

[0071] As described above, the imaging unit 202 is always used only when detecting contact with the virtual operation interface, and in the subsequent processes, it is also possible to operate the virtual operation interface using only the detection unit 222 without using the imaging unit 202. Therefore, even if the user's hand cannot be detected from the captured image after detecting contact, it is possible to continue operating the virtual operation interface using the inertia information acquired from the detection unit 222. Note that the case where the user's hand cannot be detected from the captured image refers to a case where the user's hand goes out of the imaging range of the imaging unit 202 and cannot be detected, or a case where the user's fingers are hidden by the back of the hand in the captured image and cannot be detected.

[0072] It is also possible to recognize models of operating parts (dials, sliders, etc.) that do not communicate with the HMD, and overlay CG on the models to use as an operating interface. In this case, as shown in Figure 10, it becomes possible to use the operating interface as if it were an operating part that can be physically operated.

[0073] <Embodiment 2> Hereinafter, a method for operating the virtual operation interface according to the second embodiment will be described with reference to Fig. 11. Only the points different from the first embodiment will be described below.

[0074] In the second embodiment, the operational flow of the virtual dial 300 is different from that in the first embodiment, so the operational flow of the virtual dial 300 will be described with reference to the flowchart shown in FIG.

[0075] In steps S1101 to S1103, the same processes as those in steps S501 to S503 shown in FIG. 5 are performed.

[0076] In step S1104, the control unit 211 judges whether or not the size of the virtual dial 300 is equal to or larger than a predetermined size. If it is judged that the size of the virtual dial 300 is equal to or larger than the predetermined size, the process proceeds to step S1105. If it is judged that the size of the virtual dial 300 is smaller than the predetermined size, the process proceeds to step S504, and thereafter, the process of the flowchart shown in FIG. 5 is performed.

[0077] In step S1105, the control unit 211 determines whether the tip of the index finger is detected from the captured image. It is determined whether or not the tip of the index finger is detected. If it is determined that the tip of the index finger is detected, the process proceeds to step S1106. If it is determined that the tip of the index finger is not detected, the process proceeds to step S504, and thereafter, the process of the flowchart shown in Fig. 5 is performed. Note that it may be determined whether or not the controller 120 is detected from the captured image, instead of the tip of the index finger.

[0078] In step S1106, control unit 211 determines the initial position of controller 120 (index finger wearing controller 120). Specifically, control unit 211 determines the coordinates (x(0), y(0), z(0)) of the initial position of controller 120 based on the captured image acquired from imaging unit 202. Control unit 211 also determines the coordinates (xt(0), yt(0), zt(0)) of the initial position of the thumb based on the captured image.

[0079] Furthermore, similar to step S504, control unit 211 calculates distance L between the center position of virtual dial 300 on the xy plane and controller 120 using the above formula 1. Control unit 211 calculates distance Lt between the thumb and the center position of virtual dial 300 on the xy plane using the above formula 2. Then, control unit 211 calculates angle θt formed by a line segment connecting the center position of virtual dial 300 and the position of the thumb, and a line segment connecting the position of controller 120 and the center position of virtual dial 300 using the above formula 3.

[0080] In step S1107, the control unit 211 determines the coordinates (x(t), y(t), z(t)) of the current position of the controller 120 (the index finger wearing the controller 120). Here, the control unit 211 determines the coordinates (x(t), y(t), z(t)) of the current position of the controller 120 by converting the coordinates (h(t), v(t)) of the position of the controller 120 in the captured image acquired from the imaging unit 202.

[0081] In steps S1108 to S1113, the same processes as those in steps S507 to S512 shown in FIG. 5 are performed.

[0082] In this way, when the virtual dial 300 is large, the control unit 211 determines the position of the controller 120 (the finger wearing the controller 120) only from the captured image (the image capture result by the imaging unit 202). This can reduce the processing load. Note that when the virtual dial 300 is large, the hand movement becomes large, so there is a high possibility that the hand movement can be accurately determined only from the captured image. Therefore, the virtual operation interface can be appropriately operated without using inertia information of the controller 120.

[0083] <Embodiment 3> Hereinafter, a method for operating the virtual operation interface according to the third embodiment will be described with reference to Figures 12A, 12B, and 13. Only the points different from the first embodiment will be described below.

[0084] In the third embodiment, the operation flow of the virtual dial 300 is different from that of the first embodiment, so the operation flow of the virtual dial 300 will be described with reference to the flowchart shown in Fig. 13. As shown in Figs. 12A and 12B, the operation angle θ(t) is the same value as the rotation angle of the controller 120. Therefore, in the third embodiment, the control unit 211 calculates the operation angle θ(t) based on angular velocity information measured by the gyro sensor included in the detection unit 222.

[0085] In steps S1301 to S1304, the same processes as those in steps S501 to S504 shown in FIG. 5 are performed.

[0086] In step S1305, the control unit 211 determines (detects) the rotation of the user's hand in the xy plane among the movements of the hand. As shown in FIG. 12B, the motion angle θ(t) of the controller 120 (the finger wearing the controller 120) is calculated (detected). The detection unit 222 calculates the motion angle θ(t) using the acquired angular velocity information (rx, ry, rz). Specifically, the detection unit 222 calculates the rotation angle θz(t) representing the rotation of the hand by integrating the angular velocity only once. The control unit 211 calculates the rotation angle θz(t) in the z-axis direction from time t=0 to time t=T using the angular velocity rz(t) in the z-axis direction, for example, according to the following formula 10.

number

[0087] Thus, Equation 10 is performed for the z-axis direction. The motion angle θ(t), which is the rotation angle in the xy plane, is the rotation angle in the z-axis direction, so θ(t)=θz(t).

[0088] In step S1306, the control unit 211 operates the virtual dial 300 by an operation angle θ(t).

[0089] In step S1307, the control unit 211 determines whether or not the finger has been removed from the virtual dial 300, similarly to step S511. That is, the control unit 211 determines whether or not the position of the finger is included in the operation reception range 600. If it is determined that the position of the finger is not included in the operation reception range 600, the process proceeds to step S1308. If it is determined that the position of the finger is included in the operation reception range 600, the process returns to step S1305.

[0090] In step S1308, the control unit 211 ends the display of the virtual dial 300. This ends the volume adjustment.

[0091] According to the third embodiment, the number of processing steps for calculating the operation angle θ(t) can be reduced compared to the first embodiment. Therefore, the time from when the user moves his / her finger until the movement is reflected in the operation of the virtual dial 300 can be shortened.

[0092] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." For this reason, unless a contradiction occurs, the expression "equal to or greater than A" may be read as "A or greater (high; long; many)," or may be read as "greater than A (high; long; many)." On the other hand, the expression "equal to or less than A" may be read as "A or smaller (low; short; few)," or may be read as "smaller than A (low; short; few)." Furthermore, "bigger (higher; longer; more) than A" may be read as "A or greater," and "smaller (lower; shorter; fewer) than A" may be read as "A or less."

[0093] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0094] Each functional unit in each of the above embodiments (variations) may or may not be an individual piece of hardware. The functions of two or more functional units may be realized by a common piece of hardware. Each of the multiple functions of one functional unit may be realized by a separate piece of hardware. The functions of at least some of the functional units may be realized by a common hardware. Two or more functions of one functional unit may be realized by a common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, an FPGA, or a DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0095] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) A determination means for determining a movement of a hand based on inertial information of a controller held by or attached to a user's hand; a control means for controlling a display means so as to display a virtual space in which the virtual interface is arranged in a real space; an operating means for operating the virtual interface based on the movement of the hand determined by the determining means when a position of the virtual interface and a position of the hand have a specific relationship in the virtual space; An information processing system comprising: (Configuration 2) The specific relationship is a relationship in which the position of the hand is included in an operation acceptance range including the position of the virtual interface. 2. The information processing system according to configuration 1. (Configuration 3) when a plurality of virtual interfaces are arranged in the real space, the control means controls a size of the operation acceptance range based on a distance between the plurality of virtual interfaces. 3. The information processing system according to configuration 2. (Configuration 4) the control means determines whether or not a position of the virtual interface and a position of the hand are in the specific relationship based on a captured image of the user's hand; 4. The information processing system according to any one of configurations 1 to 3. (Configuration 5) the determining means determines a position of the hand at a specific time when the position of the virtual interface and the position of the hand have the specific relationship as an initial position, and determines a movement of the hand from the initial position based on the inertia information. 5. The information processing system according to configuration 4. (Configuration 6) The determination means determines the movement of the hand based on the inertia information, not based on a captured image of the user's hand. 6. The information processing system according to any one of configurations 1 to 5. (Configuration 7) The inertial information is acceleration information or angular velocity information. 7. The information processing system according to any one of configurations 1 to 6. (Configuration 8) the inertial information is the acceleration information, the determination means determines movement information of the controller representing the movement of the hand based on the acceleration information. 8. The information processing system according to configuration 7. (Configuration 9) the virtual interface is a dial operable in response to a rotation of the hand in a particular plane; the determining means determines a rotation of the hand from among the hand movements based on movement information of the controller; the operation means operates the dial based on the rotation of the hand determined by the determination means when the position of the dial and the position of the hand are in the specific relationship. 9. The information processing system according to configuration 8. (Configuration 10) The determination means is determining shift information, which is movement information corresponding to a hand movement in the specific plane other than a rotation of the hand, among movement information of the controller, based on a relationship between a position of the dial, a position of the controller, and a position of a specific finger of the user; The information processing system according to configuration 9, further comprising: determining rotation of the hand based on movement information of the controller and the shift information. (Configuration 11) the control means corrects a position of the dial based on movement information of the controller after the dial is operated based on the hand movement; 11. The information processing system according to configuration 9 or 10. (Configuration 12) When the size of the virtual interface is larger than a predetermined size, the determination means determines the movement of the hand based on a captured image of the user's hand. 12. The information processing system according to any one of configurations 1 to 11. (Configuration 13) The controller has at least one of an acceleration sensor and a gyro sensor as a sensor for acquiring the inertial information. 13. The information processing system according to any one of configurations 1 to 12. (method) a determining step of determining a movement of the hand based on inertial information of a controller held in or attached to a hand of a user; a control step of controlling a display means to display a virtual space in which the virtual interface is arranged in a real space; an operation step of operating the virtual interface based on the movement of the hand determined in the determination step when a position of the virtual interface and a position of the hand have a specific relationship in the virtual space; 13. An information processing method comprising: (program) A program for causing a computer to function as each of the means of the information processing system according to any one of configurations 1 to 13. [Explanation of symbols]

[0096] 1: Information processing system, 100: HMD, 110: image processing device, 120: controller, 203: image display unit, 211: control unit, 222: detection unit

Claims

1. a determination means for determining a movement of a hand based on inertial information of a controller held by or attached to a user's hand; a control means for controlling the display means so as to display a virtual space in which the virtual interface is arranged in the real space; an operating means for operating the virtual interface based on the hand movement determined by the determining means when a position of the virtual interface and a position of the hand have a specific relationship in the virtual space; and the specific relationship is a relationship in which the position of the hand is included in an operation acceptance range that includes the position of the virtual interface, An information processing system characterized in that, when multiple virtual interfaces are placed in the real space, the control means controls the size of the operation acceptance range based on the distance between the multiple virtual interfaces.

2. the control means determines whether or not the position of the virtual interface and the position of the hand are in the specific relationship based on a captured image of the user's hand.

2. The information processing system according to claim 1, wherein:

3. the determining means determines a movement of the hand from an initial position based on the inertia information, the position of the hand at a specific time when the position of the virtual interface and the position of the hand have the specific relationship.

3. The information processing system according to claim 2.

4. the determining means determines the movement of the user's hand based on the inertia information, not based on a captured image of the user's hand.

4. The information processing system according to claim 1, wherein the information processing system is a data processing system.

5. The inertial information is acceleration information or angular velocity information.

4. The information processing system according to claim 1, wherein the information processing system is a data processing system.

6. the inertial information is the acceleration information, the determination means determines movement information of the controller representing the movement of the hand based on the acceleration information.

6. The information processing system according to claim 5.

7. the virtual interface is a dial that can be operated in response to a rotation of the hand in a specific plane; the determining means determines a rotation of the hand from among the hand movements based on movement information of the controller; the operating means operates the dial based on the rotation of the hand determined by the determining means when the position of the dial and the position of the hand are in the specific relationship.

7. The information processing system according to claim 6.

8. The determination means determining shift information, which is movement information corresponding to a hand movement in the specific plane other than a hand rotation, from among movement information of the controller, based on a relationship between the position of the dial, the position of the controller, and the position of a specific finger of the user; determining a rotation of the hand based on the movement information and the shift information of the controller; 8. The information processing system according to claim 7,

9. the control means corrects the position of the dial based on movement information of the controller after the dial is operated based on the hand movement; 8. The information processing system according to claim 7,

10. A determination means for determining a movement of a hand based on inertial information of a controller held in or attached to a user's hand; a control means for controlling the display means so as to display a virtual space in which the virtual interface is arranged in the real space; an operating means for operating the virtual interface based on the hand movement determined by the determining means when a position of the virtual interface and a position of the hand have a specific relationship in the virtual space; and When the size of the virtual interface is larger than a predetermined size, the determination means determines the movement of the hand based on a captured image of the user's hand. An information processing system comprising:

11. the controller has at least one of an acceleration sensor and a gyro sensor as a sensor for acquiring the inertial information; 4. The information processing system according to claim 1, wherein the information processing system is a data processing system.

12. a determining step of determining a movement of the user's hand based on inertial information of a controller held in or attached to the user's hand; a control step of controlling a display means to display a virtual space in which the virtual interface is arranged in a real space; an operation step of operating the virtual interface based on the hand movement determined in the determination step when a position of the virtual interface and a position of the hand have a specific relationship in the virtual space; and The specific relationship is that the hand is within an operation acceptance range including the position of the virtual interface. It is a relationship that includes position, In the control step, when a plurality of virtual interfaces are arranged in the real space, the size of the operation acceptance range is controlled based on the distance between the plurality of virtual interfaces.

1. An information processing method comprising:

13. A determination step of determining a movement of a user's hand based on inertial information of a controller held in or attached to the user's hand; a control step of controlling a display means to display a virtual space in which the virtual interface is arranged in a real space; an operation step of operating the virtual interface based on the hand movement determined in the determination step when a position of the virtual interface and a position of the hand have a specific relationship in the virtual space; and In the determining step, if the size of the virtual interface is larger than a predetermined size, the movement of the hand is determined based on a captured image of the user's hand.

1. An information processing method comprising:

14. A program for causing a computer to function as each of the means of the information processing system according to claim 1 or 10.

Images